BackTrackZ/scrapy
BackTrackZ/scrapy
1
0
Fork 0
mirror of https://github.com/scrapy/scrapy.git synced 2025-03-01 17:08:21 +00:00
Code Releases Wiki Activity
5,153 Commits 39 Branches 119 Tags
Commit Graph

5153 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
HalfCrazy
b21a28cc9a Afterwords->Afterwards 2014-11-10 06:28:09 +05:00
Daniel Graña
2c67bd6c57 pywin32 is required by Twisted. closes #937 
see:
* http://twistedmatrix.com/trac/ticket/6032
* https://tahoe-lafs.org/trac/tahoe-lafs/ticket/2028
 2014-11-05 23:05:54 -02:00
Daniel Graña
6cb8995731 Update install.rst 
fixes #937
 2014-11-05 22:50:52 -02:00
Pablo Hoffman
efe589c643 Merge pull request #882 from ahlen/feature/csvfeed-quotechar 
[MRG+1] Allow to specify the quotechar in CSVFeedSpider
 2014-11-04 11:32:59 -02:00
Lazar-T
38dcf50cd6 comma instead of fullstop 2014-10-25 09:19:50 +06:00
Pablo Hoffman
675fd5ba04 Merge pull request #898 from scrapy/download-timeout 
[MRG] DOC document download_timeout
 2014-10-24 16:52:42 -02:00
Pablo Hoffman
5de6a11fda Merge pull request #925 from Digenis/master 
a leftover for.15 compatibility
 2014-10-22 18:23:19 -02:00
Nikolaos-Digenis Karagiannis
2227805619 Compatibility with .15 leftover 2014-10-22 21:48:08 +03:00
Pablo Hoffman
0dce283459 Merge pull request #893 from kmike/less-ads 
[MRG] DOC simplify extension docs
 2014-10-21 17:13:59 -02:00
Daniel Graña
44cbbecb44 Merge pull request #914 from brunsgaard/master 
Deleted bin folder from root, fixes #913
 2014-10-09 14:58:36 -02:00
Pablo Hoffman
aa61f615d8 Merge pull request #795 from chekunkov/spider_error_processing_referer 
Add referer to "Spider error processing" log message
 2014-10-07 20:57:48 -02:00
Jonas Brunsgaard
db2474f7e7 Deleted bin folder from root, fixes #913 2014-10-07 13:54:04 +02:00
Mikhail Korobov
7d68b084a4 DOC document download_timeout Request.meta key and download_timeout spider attribute. 2014-10-07 04:23:11 +06:00
Pablo Hoffman
9af61d5df6 Merge pull request #895 from scrapy/we-are-past-0.15 
[MRG] drop support for CONCURRENT_REQUESTS_PER_SPIDER
 2014-10-03 17:14:00 -03:00
Pablo Hoffman
5f1bbe2dd3 Merge pull request #911 from nyov/nyov/dev 
Drop old engine code
 2014-10-03 17:13:17 -03:00
nyov
7db6bbce27 Drop old engine code 
* remove Downloader import unused since 1fba64
  * remove CONCURRENT_SPIDERS deprecation warning from a1dbc6 (2011)
 2014-10-03 19:07:51 +00:00
Mikhail Korobov
14957ed71e Merge pull request #909 from VKen/master 
updated deprecated cgi.parse_qsl to use six's parse_qsl
 2014-10-03 16:30:39 +06:00
VKen
33a7c1d438 updated deprecated cgi.parse_qsl to use six's parse_qsl 2014-10-03 04:16:21 +08:00
Mikhail Korobov
ea3b372b4f DOC typo fix in leaks.rst 2014-10-02 15:20:13 +06:00
Pablo Hoffman
993b543e1b mark SEP-019 as Final 2014-10-02 01:17:31 -03:00
Pablo Hoffman
e7843d35de Merge pull request #894 from kmike/leaks-docs 
Leaks docs
 2014-10-02 01:14:54 -03:00
Pablo Hoffman
5835224eee Merge pull request #896 from scrapy/robotstxt-once 
[MRG] process robots.txt once
 2014-10-02 00:58:55 -03:00
Pablo Hoffman
9e2c60430d Merge pull request #902 from scrapy/load_object_full_traceback 
[MRG] scrapy.utils.misc.load_object should print full traceback
 2014-10-02 00:33:27 -03:00
Pablo Hoffman
0c23b1342b Merge pull request #904 from scrapy/from_crawler_docs 
[MRG] DOC document from_crawler method for item pipelines
 2014-10-02 00:09:07 -03:00
Mikhail Korobov
6fcf9dce50 DOC document from_crawler method for item pipelines; add an example. 2014-09-25 03:13:51 +06:00
Mikhail Korobov
5086262913 don't hide original exception in scrapy.utils.misc.load_object 2014-09-24 13:27:14 +06:00
Mikhail Korobov
36eec8f413 dont_obey_robotstxt meta key; don't process requests to /robots.txt 2014-09-23 00:10:43 +06:00
Mikhail Korobov
fe6f3efe95 RobotsTxtMiddleware: remove unused attribute 2014-09-22 22:56:54 +06:00
Mikhail Korobov
d11c8595e6 drop support for CONCURRENT_REQUESTS_PER_SPIDER 2014-09-22 04:29:22 +06:00
Mikhail Korobov
bdbca1e2d7 DOC request queue memory usage 2014-09-21 07:30:44 +06:00
Mikhail Korobov
bc0f481a73 DOC bring back notes about multiple spiders per process because it is now documented how to do that 2014-09-21 07:12:01 +06:00
Mikhail Korobov
a122fdbfea Update leaks.rst: there is now only a single spider in a process. 2014-09-21 06:54:00 +06:00
Mikhail Korobov
7be3479c20 CookieJar cleanup 2014-09-21 06:37:32 +06:00
Mikhail Korobov
49645d4bf9 TST small cleanup of a cookie test 2014-09-21 05:31:34 +06:00
Mikhail Korobov
c543fe6e4c Merge pull request #878 from andrewshir/master 
Fix bug for ".local" host name
 2014-09-21 05:24:54 +06:00
Mikhail Korobov
e435b3e3a3 DOC simplify extension docs 2014-09-21 00:19:24 +06:00
John-Scott Atlakson
a312ebfb43 Update request-response.rst 
Fixed minor typo
 2014-09-14 22:06:31 +06:00
andrewshir
e583c030db Test for local domains (without dots) added 2014-09-14 14:24:16 +06:00
Mikael Åhlén
22da1783bd added a test-case for wrong quotechar 2014-09-13 03:47:40 +02:00
Mikael Åhlén
47b6dff9f1 Allow to specify the quotechar in CSVFeedSpider 2014-09-13 02:14:57 +02:00
Daniel Graña
5bcabfe9c9 SPIDER_MODULES can be set as a csv string 2014-09-10 23:25:57 -03:00
Daniel Graña
c05e99a4f4 oops, restore Pillow from precise test requirements 2014-09-10 12:21:08 -03:00
Daniel Graña
a823207f18 Stop logobserver only when set 2014-09-10 12:09:07 -03:00
Daniel Graña
ec93c0fdcc Add the tests changes for previous commit 2014-09-10 12:05:18 -03:00
Daniel Graña
ce180227fa Twisted 11.1.0 (precise) can not deal with generators in DeferredList 
Also create a list of the crawlers before iterating them because crawlers are removed from the set once stopped
 2014-09-10 12:04:14 -03:00
Daniel Graña
99971dc8a8 Do not pop the crawler from the managed list 2014-09-09 20:59:07 +00:00
Daniel Graña
774aa9ee56 Merge branch 'crawler-api' 2014-09-09 17:23:11 -03:00
Daniel Graña
8ddf0811a8 Correctly detect when all managed crawlers are done in CrawlerRunner 2014-09-09 17:21:39 -03:00
Daniel Graña
68954fa503 Merge pull request #879 from Curita/fix-874-issue 
Fix #874 issue
 2014-09-08 14:32:06 -03:00
Julia Medina
51532af69a Erase unneeded flag in CrawlerProcess.start 2014-09-07 13:03:34 -03:00