1
0
mirror of https://github.com/scrapy/scrapy.git synced 2025-02-25 06:44:23 +00:00

20 Commits

Author SHA1 Message Date
Pablo Hoffman
bf467fc37a Check 'dont_merge_cookies' membership in request.meta, instead of getting its value 2010-09-10 15:29:15 -03:00
Pablo Hoffman
7d14a52234 Reference dont_merge_cookies in list of special Request.meta keys 2010-09-09 21:54:26 -03:00
Pablo Hoffman
7f21a6384f Documented handle_httpstatus_list request.meta key 2010-09-09 21:50:40 -03:00
Pablo Hoffman
f1c943543a Added dont_retry request.meta key to make RetryMiddleware ignore requests. Closes #234 2010-09-09 21:43:44 -03:00
Pablo Hoffman
9f01e3e79e Added dont_redirect request.meta key to make RedirectMiddleware ignore requests. Closes #233 2010-09-09 21:37:35 -03:00
Pablo Hoffman
7da79b90fe Make url/body attributes of Request/Response objects read-only - use replace() to change them. Deprecation warning left for backwards compatibilty. 2010-09-08 00:15:11 -03:00
Pablo Hoffman
c1aab2f58e Copy callback/errback attributes when copying Requests 2010-09-08 00:15:09 -03:00
Pablo Hoffman
9aefa242d5 Applied documentation patch provided by Lucian Ursu (closes #207) 2010-08-21 01:26:35 -03:00
Pablo Hoffman
73305b1eb3 Added support for Requests without callbacks (#166) - the Spider.parse() method
is used in those cases.

Also removed Request.deferred attribute.
2010-06-08 18:18:02 -03:00
Daniel Grana
c0d45846b8 Automated merge with ssh://hg.scrapy.org/scrapy-0.8 2010-04-26 22:29:45 -03:00
Steven Almeroth
5d03405cac FormRequest.from_response doc fix. closes #155
--HG--
extra : rebase_source : d54979f6a15e5e997072dcbbc6d43b426189312b
2010-04-26 22:28:07 -03:00
Rolando Espinoza La fuente
db5c3df679 SEP12 implementation
* Rename BaseSpider.domain_name to BaseSpider.name

    This patch implements the domain_name to name change in BaseSpider class and
    change all spider instantiations to use the new attribute.

  * Add allowed_domains to spider

    This patch implements the merging of spider.domain_name and
    spider.extra_domain_names in spider.allowed_domains for offsite checking
    purposes.

    Note that spider.domain_name is not touched by this patch, only not used.

  * Remove spider.domain_name references from scrapy.stats

    * Rename domain_stats to spider_stats in MemoryStatsCollector
    * Use ``spider`` instead of ``domain`` in SimpledbStatsCollector
    * Rename domain_stats_history table to spider_data_history and rename domain
    field to spider in MysqlStatsCollector

  * Refactor genspider command

    The new signature for genspider is: genspider [options] <domain_name>.

    Genspider uses domain_name for spider name and for the module name.

  * Remove spider.domain_name references

  * Update crawl command signature <spider|url>

  * docs: updated references to domain_name

  * examples/experimental: use spider.name

  * genspider: require <name> <domain>

  * spidermanager: renamed crawl_domain to crawl_spider_name

  * spiderctl: updated references of *domain* to spider

  * added backward compatiblity with legacy spider's attributes
    'domain_name' and 'extra_domain_names'
2010-04-01 18:27:22 -03:00
Pablo Hoffman
1330697c3d Some improvements to Response encoding support:
* added encoding aliases, configurable through a new ENCODING_ALIASES setting
* Response.encoding now returns the real encoding detected for the body
* simplified TextResponse API by removing body_encoding() and
  headers_encoding() methods
* Response.encoding now tries to infer the encoding from the body always (it
  was done before only on HtmlResponse and TextResponse)
* removed scrapy.utils.encoding.add_encoding_alias() function
* updated implementation of scrapy.utils.response function to reflect these API
  changes
* updated documentation to reflect API changes
2010-03-25 15:47:10 -03:00
Pablo Hoffman
180c091fb2 Fixed encoding issue (reported in #135) when the encoding declared in the HTTP header is unknown. This is the patch proposed by Rolando, with an update to the Request/Response documentation. 2010-02-24 14:01:29 -02:00
Pablo Hoffman
904cde6513 added clarification about new dont_click argument of FormRequest.from_response() method 2009-10-29 13:47:10 -02:00
Ismael Carnales
a244d23b89 added dont_click attr to FormRequest 2009-10-29 13:18:13 -02:00
Pablo Hoffman
720bc166cf updated new clickdata argument doc 2009-10-20 17:21:56 -02:00
Daniel Grana
6abb3c17ee Improve FormRequest.from_response method to pass click data arguments to ClientForm library 2009-10-20 15:51:41 -02:00
Pablo Hoffman
31693eb90f dropped "cache" attribute of Request and Response objects 2009-08-24 10:34:05 -03:00
Ismael Carnales
33089d287d merged topics and reference doc 2009-08-18 14:05:15 -03:00