mirror of
https://github.com/scrapy/scrapy.git
synced 2025-02-27 20:04:06 +00:00
30 lines
1.3 KiB
ReStructuredText
30 lines
1.3 KiB
ReStructuredText
=====================
|
|
Setting everything up
|
|
=====================
|
|
|
|
| In this tutorial, we'll teach you how to scrape http://www.dmoz.org, a websites directory.
|
|
| We'll assume that you've checked and fulfilled the requirements specified in the Download page, and that Scrapy is already installed in your system.
|
|
|
|
For starting a new project, enter the directory where you'd like your project to be located, and run::
|
|
|
|
scrapy-admin.py startproject dmoz
|
|
|
|
As long as Scrapy is well installed and the path is set, this should create a directory called "dmoz"
|
|
containing the following files:
|
|
|
|
* *scrapy-ctl.py* - the project's control script. It's used for running the different tasks (like "crawl" and "parse"). We'll talk more about this later.
|
|
* *scrapy_settings.py* - the project's settings file.
|
|
* *items.py* - were you define the different kinds of items you're going to scrape.
|
|
* *spiders* - directory where you'll later place your spiders.
|
|
* *templates* - directory containing some templates for newly created spiders, and where you can put your own.
|
|
|
|
| Ok, now that you have your project's structure defined, the last thing to do is to set your PYTHONPATH to your project's directory.
|
|
| You can do this by adding this to your .bashrc file:
|
|
|
|
::
|
|
|
|
export PYTHONPATH=/path/to/your/project
|
|
|
|
|
|
Now you can continue with the next part of the tutorial.
|