Take the 2-minute tour ×

I get this warning when running scrapy :

 [scrapy] WARNING: Disabled HttpCacheMiddleware: Unable to find scrapy.cfg file to infer project data dir
share|improve this question
add comment

1 Answer

Where is your scrapy.cfg file? Are you running from the command line? If so, you should be running from the same directory as scrapy.cfg.

What does your project directory look like. It shoud look something like:

├── dirbot
│   ├── __init__.py
│   ├── items.py
│   ├── pipelines.py
│   ├── settings.py
│   └── spiders
│       ├── dmoz.py
│       └── __init__.py
├── README.rst
└── scrapy.cfg


If you are using cron, try not to (meaning don't) use the root user. You can add the PATH info right into the crontab: How to get CRON to call in the correct paths

share|improve this answer
Project directory is the same as the one you described, the spider is called from a scheduler.php file in the same dir as scrapy.cfg –  Vanddel Apr 2 '13 at 12:20
How are you calling scrapy? Popen()? who is calling scheduler.php? Apache? It sounds like the user does not have this directory in their Python path. –  Steven Almeroth Apr 2 '13 at 12:45
a cron job calls scheduler.php which runs the spiders that should be running now. –  Vanddel Apr 2 '13 at 12:47
What is the cron user? He does not have your directory in his Python path. –  Steven Almeroth Apr 2 '13 at 12:51
The cron user is root, how do I add the directory to his python path? –  Vanddel Apr 2 '13 at 12:56
add comment

Your Answer


