scrapy

commands

fix small bug with parse command

2012-06-21 20:06:50 +02:00

contrib

domain and path support for request cookies

2012-06-25 20:17:59 +02:00

contrib_exp

Pass into the model only existing fields

2012-07-25 10:10:46 +08:00

core

Avoid logging useless error messages about ignored requests in robots.txt

2012-04-10 13:37:32 -03:00

http

be consistent removing BOM from decoded bodies. #123

2012-04-27 00:31:03 -03:00

selector

make libxml2 optional now that lxml is the default

2012-05-16 18:17:51 -03:00

settings

changed default user agent to Scrapy/0.15 (+http://scrapy.org ) and removed no longer needed BOT_VERSION setting

2012-03-23 13:45:21 -03:00

templates

update docstring in project template to avoid confusion with genspider command, which may be considered as an advanced feature. refs #107

2012-04-19 02:35:48 -03:00

tests

Simple test for default values of model in DjangoItem

2012-07-26 20:55:55 +08:00

utils

Small improvement on *MemoryQueue performance (mainly on push time)

2012-08-10 10:38:20 -03:00

xlib

fix urlparse monkeypatches for python 2.7.4. closes #144

2012-06-18 11:50:43 -03:00

__init__.py

make libxml2 optional now that lxml is the default

2012-05-16 18:17:51 -03:00

cmdline.py

improved command line tool usage help which explains that more commands are available when run from project directory. refs #107

2012-04-19 02:54:57 -03:00

command.py

Dropped support for Python 2.5. See: http://blog.scrapy.org/scrapy-dropping-support-for-python-25

2012-03-01 08:18:12 -02:00

conf.py

Removed hacky SCRAPY_SETTINGS_DISABLED environment variable

2010-09-22 16:08:18 -03:00

crawler.py

added from_crawler class method to SpiderManager

2011-08-16 11:16:35 -03:00

dupefilter.py

pass close reason to close() method of new DupeFilter

2011-08-24 11:26:35 -03:00

exceptions.py

replaced DeprecationWarning by a new ScrapyDeprecationWarning category, since the default DeprecationWarning is silenced on Python 2.7+

2011-08-08 10:39:53 -03:00

extension.py

Made ExtensionManager a subclass of MiddlewareManager

2010-08-22 05:33:08 -03:00

interfaces.py

removed class method from_settings from ISpiderManager interface

2011-08-26 09:24:01 -03:00

item.py

Changed default representation of items to pretty-printed dicts. This improves

2011-06-03 01:13:01 -03:00

link.py

warn if Link objects are instantiated with unicode urls

2012-05-16 13:12:25 -03:00

linkextractor.py

added more extensions to ignore on link extractors

2012-08-07 14:23:50 -03:00

log.py

removed backwards compatibility code for old scrapy versions

2012-03-06 05:42:09 -02:00

logformatter.py

improved download errors propagation to the spiders, and removed no longer needed code to simplify

2011-07-13 14:10:05 -03:00

mail.py

removed (somewhat hacky) MAIL_DEBUG setting

2010-08-22 22:42:00 -03:00

middleware.py

Another step towards singleton removal: deprecated crawler singleton import (from scrapy.project import crawler) by a new class method that extensions can implement to receive the crawler

2011-08-08 11:42:44 -03:00

mime.types

moved module scrapy.core.downloader.responsetypes to scrapy.responsetypes

2011-08-07 02:49:57 -03:00

project.py

Another step towards singleton removal: deprecated crawler singleton import (from scrapy.project import crawler) by a new class method that extensions can implement to receive the crawler

2011-08-08 11:42:44 -03:00

resolver.py

removed wrong blocking api usage (socket.gethostbyname()) from downloader when using CONCURRENT_REQUESTS_PER_IP

2011-08-03 23:55:59 -03:00

responsetypes.py

Dropped support for Python 2.5. See: http://blog.scrapy.org/scrapy-dropping-support-for-python-25

2012-03-01 08:18:12 -02:00

shell.py

scrapy shell: start shell in main thread and crawler in secondary thread, instead of the other way around. fixes #100

2012-05-22 19:15:54 -03:00

signals.py

Merged item passed and item scraped concepts, as they have often proved

2011-06-03 01:13:00 -03:00

spider.py

change start_request to return a generator instead of a list, in order

2012-03-05 12:49:17 -02:00

spidermanager.py

improve handling of KeyError exception when creating spiders in spider manager. closes issue 49

2011-11-14 17:00:25 -02:00

squeue.py

Some changes to persistent scheduler after some initial usage feedback:

2011-09-23 13:03:07 -03:00

stats.py

Splitted stats collector classes from stats collection facility (#204 )

2010-08-22 01:24:07 -03:00

statscol.py

call stats collector engine_stopped() after the engine is closed (to make sure all data from extensions has been collected), and added that method to documented api

2011-06-06 03:12:40 -03:00

telnet.py

fixed links to doc. closes #150

2012-06-24 01:00:33 -03:00

webservice.py

fixed bug in json-rpc webservice reported in https://groups.google.com/d/topic/scrapy-users/qgVBmFybNAQ/discussion . also removed no longer supported 'run' command from extras/scrapy-ws.py

2012-05-03 12:05:40 -03:00