BackTrackZ/scrapy
BackTrackZ/scrapy
1
0
Fork 0
mirror of https://github.com/scrapy/scrapy.git synced 2025-02-24 22:43:57 +00:00
Code Releases Wiki Activity
3,769 Commits 39 Branches 119 Tags
Commit Graph

45 Commits

Author SHA1 Message Date
Pablo Hoffman
e42e3743fe quick documentation for #475 2013-12-24 12:19:15 -02:00
Mikhail Korobov
086b8a20d4 typo fix in TextResponse docs 2013-10-17 04:50:30 +06:00
Dragon Dave
a3b711bdea Move callback blob; mention errback 2013-10-15 12:19:42 +01:00
scraperdragon
0ba0d85685 Parameters to Request() in wrong order 
Implied that callback wasn't the first optional unnamed parameter.
 2013-10-15 11:50:43 +01:00
Pablo Hoffman
37c24e01d7 document bindaddress request meta 2013-10-02 17:13:17 -03:00
Pablo Hoffman
a9c3519897 updated required twisted version to 10.0 2013-10-01 14:07:38 -03:00
cacovsky
71b320914a Update request-response.rst 
Fix small doc typo (too many backticks)
 2013-09-18 11:45:25 -03:00
Mikhail Korobov
1a1c93fafe tiny FormRequest doc fix 2013-07-15 15:47:34 +06:00
Steven Almeroth
70179c7c0c doc: remove trailing spaces 2013-03-21 13:57:39 -06:00
Steven Almeroth
0d7747d353 doc: Response.replace() cannot take meta argument 
>>> response.replace(meta={'foo':1})
Traceback (most recent call last):
  File "<input>", line 1, in <module>
  File "/srv/scrapy/scrapy-fork/scrapy/scrapy/http/response/text.py", line 45, in replace
    return Response.replace(self, *args, **kwargs)
  File "/srv/scrapy/scrapy-fork/scrapy/scrapy/http/response/__init__.py", line 77, in replace
    return cls(*args, **kwargs)
  File "/srv/scrapy/scrapy-fork/scrapy/scrapy/http/response/text.py", line 22, in __init__
    super(TextResponse, self).__init__(*args, **kwargs)
TypeError: __init__() got an unexpected keyword argument 'meta'
 2013-03-21 13:49:55 -06:00
Chris Tilden
aae6aed4fb fixes spelling errors in documentation 2013-01-22 14:52:18 -08:00
Daniel Graña
897195186a document new FormRequest parameter named formxpath that matches forms using xpath 2013-01-08 18:36:20 -02:00
Pablo Hoffman
39274a2457 doc: removed obsolete references to ClientForm 2012-11-23 19:06:47 -02:00
Pablo Hoffman
babfc6e79b Updated documentation after singleton removal changes. 
Also removed some unused code and made some minor additional
refactoring.
 2012-08-28 18:35:57 -03:00
Alexandru Cepoi
177c81745d domain and path support for request cookies 2012-06-25 20:17:59 +02:00
Jason Yeo
da826aa13d fixed minor mistake in Request objects documentation 2012-03-21 10:25:41 +08:00
Pablo Hoffman
26c8004125 added documentation for the new cookiejar Request.meta key 2012-02-27 19:58:58 -02:00
Pablo Hoffman
41fd3c4f6c doc: removed duplicated callback argument from Request.replace() 2011-12-23 15:55:46 -02:00
Pablo Hoffman
6a31ab667d minor fix to doc 2011-09-01 15:08:23 -03:00
Pablo Hoffman
d98b058c21 no longer recommend using labmda's in the doc, as they're not friendly with scheduler persistence 2011-09-01 15:06:49 -03:00
Pablo Hoffman
5c6b0631e2 minor doc fix 2011-08-19 11:42:03 -03:00
Pablo Hoffman
9d97e73a24 fixed priority handling on the new scheduler so that it's backwards compatible (ie. bigger priorities are higher). also fixed a few documentation bugs related to requests priority 2011-08-19 08:26:41 -03:00
Pablo Hoffman
3ee2c94e93 Improved cookies middleware by making COOKIES_DEBUG nicer and documenting it 2011-04-06 14:54:48 -03:00
Pablo Hoffman
91a7c25797 * Made Response.meta attribute map to Request.meta attribute. Closes #290 
* Record redirected URLs in redirect middleware. Closes #291
 2010-11-18 12:51:54 -02:00
Shuaib
9288f622f9 Added formname parameter for FormRequest.from_response 2010-09-20 08:33:24 -03:00
Pablo Hoffman
bf467fc37a Check 'dont_merge_cookies' membership in request.meta, instead of getting its value 2010-09-10 15:29:15 -03:00
Pablo Hoffman
7d14a52234 Reference dont_merge_cookies in list of special Request.meta keys 2010-09-09 21:54:26 -03:00
Pablo Hoffman
7f21a6384f Documented handle_httpstatus_list request.meta key 2010-09-09 21:50:40 -03:00
Pablo Hoffman
f1c943543a Added dont_retry request.meta key to make RetryMiddleware ignore requests. Closes #234 2010-09-09 21:43:44 -03:00
Pablo Hoffman
9f01e3e79e Added dont_redirect request.meta key to make RedirectMiddleware ignore requests. Closes #233 2010-09-09 21:37:35 -03:00
Pablo Hoffman
7da79b90fe Make url/body attributes of Request/Response objects read-only - use replace() to change them. Deprecation warning left for backwards compatibilty. 2010-09-08 00:15:11 -03:00
Pablo Hoffman
c1aab2f58e Copy callback/errback attributes when copying Requests 2010-09-08 00:15:09 -03:00
Pablo Hoffman
9aefa242d5 Applied documentation patch provided by Lucian Ursu (closes #207) 2010-08-21 01:26:35 -03:00
Pablo Hoffman
73305b1eb3 Added support for Requests without callbacks (#166) - the Spider.parse() method 
is used in those cases.

Also removed Request.deferred attribute.
 2010-06-08 18:18:02 -03:00
Daniel Grana
c0d45846b8 Automated merge with ssh://hg.scrapy.org/scrapy-0.8 2010-04-26 22:29:45 -03:00
Steven Almeroth
5d03405cac FormRequest.from_response doc fix. closes #155 
--HG--
extra : rebase_source : d54979f6a15e5e997072dcbbc6d43b426189312b
 2010-04-26 22:28:07 -03:00
Rolando Espinoza La fuente
db5c3df679 SEP12 implementation 
* Rename BaseSpider.domain_name to BaseSpider.name

    This patch implements the domain_name to name change in BaseSpider class and
    change all spider instantiations to use the new attribute.

  * Add allowed_domains to spider

    This patch implements the merging of spider.domain_name and
    spider.extra_domain_names in spider.allowed_domains for offsite checking
    purposes.

    Note that spider.domain_name is not touched by this patch, only not used.

  * Remove spider.domain_name references from scrapy.stats

    * Rename domain_stats to spider_stats in MemoryStatsCollector
    * Use ``spider`` instead of ``domain`` in SimpledbStatsCollector
    * Rename domain_stats_history table to spider_data_history and rename domain
    field to spider in MysqlStatsCollector

  * Refactor genspider command

    The new signature for genspider is: genspider [options] <domain_name>.

    Genspider uses domain_name for spider name and for the module name.

  * Remove spider.domain_name references

  * Update crawl command signature <spider|url>

  * docs: updated references to domain_name

  * examples/experimental: use spider.name

  * genspider: require <name> <domain>

  * spidermanager: renamed crawl_domain to crawl_spider_name

  * spiderctl: updated references of *domain* to spider

  * added backward compatiblity with legacy spider's attributes
    'domain_name' and 'extra_domain_names'
 2010-04-01 18:27:22 -03:00
Pablo Hoffman
1330697c3d Some improvements to Response encoding support: 
* added encoding aliases, configurable through a new ENCODING_ALIASES setting
* Response.encoding now returns the real encoding detected for the body
* simplified TextResponse API by removing body_encoding() and
  headers_encoding() methods
* Response.encoding now tries to infer the encoding from the body always (it
  was done before only on HtmlResponse and TextResponse)
* removed scrapy.utils.encoding.add_encoding_alias() function
* updated implementation of scrapy.utils.response function to reflect these API
  changes
* updated documentation to reflect API changes
 2010-03-25 15:47:10 -03:00
Pablo Hoffman
180c091fb2 Fixed encoding issue (reported in #135) when the encoding declared in the HTTP header is unknown. This is the patch proposed by Rolando, with an update to the Request/Response documentation. 2010-02-24 14:01:29 -02:00
Pablo Hoffman
904cde6513 added clarification about new dont_click argument of FormRequest.from_response() method 2009-10-29 13:47:10 -02:00
Ismael Carnales
a244d23b89 added dont_click attr to FormRequest 2009-10-29 13:18:13 -02:00
Pablo Hoffman
720bc166cf updated new clickdata argument doc 2009-10-20 17:21:56 -02:00
Daniel Grana
6abb3c17ee Improve FormRequest.from_response method to pass click data arguments to ClientForm library 2009-10-20 15:51:41 -02:00
Pablo Hoffman
31693eb90f dropped "cache" attribute of Request and Response objects 2009-08-24 10:34:05 -03:00
Ismael Carnales
33089d287d merged topics and reference doc 2009-08-18 14:05:15 -03:00