mirror of
https://github.com/scrapy/scrapy.git
synced 2025-02-26 06:04:08 +00:00
42dc34f924
Install twisted trunk because released twisted doesn't install properly, see https://twistedmatrix.com/trac/ticket/6539. Also, test-requirements are removed for 3.3 because mitmproxy doesn't install in Python 3.3, and mock is in Python 3.3 standard library. Note that trial scrapy is not doing what is expected because twisted doesn't have trial runner for 3.3 yet, so this command will most likely call "trial" outside tox virtualenv and will use incorrect interpreter.
======
Scrapy
======
.. image:: https://badge.fury.io/py/Scrapy.png
:target: http://badge.fury.io/py/Scrapy
.. image:: https://secure.travis-ci.org/scrapy/scrapy.png?branch=master
:target: http://travis-ci.org/scrapy/scrapy
Overview
========
Scrapy is a fast high-level screen scraping and web crawling framework, used to
crawl websites and extract structured data from their pages. It can be used for
a wide range of purposes, from data mining to monitoring and automated testing.
For more information including a list of features check the Scrapy homepage at:
http://scrapy.org
Requirements
============
* Python 2.7
* Works on Linux, Windows, Mac OSX, BSD
Install
=======
The quick way::
pip install scrapy
For more details see the install section in the documentation:
http://doc.scrapy.org/en/latest/intro/install.html
Releases
========
You can download the latest stable and development releases from:
http://scrapy.org/download/
Documentation
=============
Documentation is available online at http://doc.scrapy.org/ and in the ``docs``
directory.
Community (blog, twitter, mail list, IRC)
=========================================
See http://scrapy.org/community/
Contributing
============
See http://doc.scrapy.org/en/latest/contributing.html
Companies using Scrapy
======================
See http://scrapy.org/companies/
Commercial Support
==================
See http://scrapy.org/support/