Mikhail Korobov
fe6f3efe95
RobotsTxtMiddleware: remove unused attribute
2014-09-22 22:56:54 +06:00
Mikhail Korobov
7be3479c20
CookieJar cleanup
2014-09-21 06:37:32 +06:00
Mikhail Korobov
49645d4bf9
TST small cleanup of a cookie test
2014-09-21 05:31:34 +06:00
Mikhail Korobov
c543fe6e4c
Merge pull request #878 from andrewshir/master
...
Fix bug for ".local" host name
2014-09-21 05:24:54 +06:00
John-Scott Atlakson
a312ebfb43
Update request-response.rst
...
Fixed minor typo
2014-09-14 22:06:31 +06:00
andrewshir
e583c030db
Test for local domains (without dots) added
2014-09-14 14:24:16 +06:00
Daniel Graña
5bcabfe9c9
SPIDER_MODULES can be set as a csv string
2014-09-10 23:25:57 -03:00
Daniel Graña
c05e99a4f4
oops, restore Pillow from precise test requirements
2014-09-10 12:21:08 -03:00
Daniel Graña
a823207f18
Stop logobserver only when set
2014-09-10 12:09:07 -03:00
Daniel Graña
ec93c0fdcc
Add the tests changes for previous commit
2014-09-10 12:05:18 -03:00
Daniel Graña
ce180227fa
Twisted 11.1.0 (precise) can not deal with generators in DeferredList
...
Also create a list of the crawlers before iterating them because crawlers are removed from the set once stopped
2014-09-10 12:04:14 -03:00
Daniel Graña
99971dc8a8
Do not pop the crawler from the managed list
2014-09-09 20:59:07 +00:00
Daniel Graña
774aa9ee56
Merge branch 'crawler-api'
2014-09-09 17:23:11 -03:00
Daniel Graña
8ddf0811a8
Correctly detect when all managed crawlers are done in CrawlerRunner
2014-09-09 17:21:39 -03:00
Daniel Graña
68954fa503
Merge pull request #879 from Curita/fix-874-issue
...
Fix #874 issue
2014-09-08 14:32:06 -03:00
Julia Medina
51532af69a
Erase unneeded flag in CrawlerProcess.start
2014-09-07 13:03:34 -03:00
Julia Medina
d513b5a542
Run root logger in CrawlerProcess creation instead of in its start method
2014-09-07 13:02:39 -03:00
andrewshir
dfca7b3c80
Fix bug for ".local" host name
...
It's necessary to put new list member in squared brackets (i.e. create new list) to merge lists properly, otherwise we will get result list with character elements instead of string element.
2014-09-06 18:23:27 +06:00
Pablo Hoffman
acb0a61cf1
Merge pull request #871 from eltermann/master
...
Removed unused 'load=False' parameter from walk_modules()
2014-09-02 12:55:04 -03:00
eltermann
1dff1fbf75
Removed unused 'load=False' parameter from walk_modules()
2014-09-02 08:33:36 -03:00
Daniel Graña
5daa14770b
Merge branch 'Curita-per-spider-settings'
2014-09-01 21:58:13 -03:00
Julia Medina
c2592b39fd
Test verifying that CrawlerRunner populates spider class settings
2014-09-01 21:56:57 -03:00
Julia Medina
77bd26a66d
Non mutable default in Spider.custom_settings
2014-09-01 21:56:57 -03:00
Julia Medina
16e62e9c9b
Per-spider settings documentation
2014-09-01 21:56:57 -03:00
Julia Medina
9ef3972cfb
Per-spider settings tests
2014-09-01 21:56:57 -03:00
Julia Medina
4932ec43a7
Per-spider settings implementation
2014-09-01 21:56:57 -03:00
Daniel Graña
ccde3317d7
Merge pull request #816 from Curita/api-cleanup
...
GSoC API cleanup
2014-09-01 21:55:36 -03:00
Pablo Hoffman
620dbe7116
Merge pull request #865 from yakxxx/master
...
SgmlLinkExtractor - fix for parsing <area> tag with Unicode present
2014-08-29 16:18:44 -03:00
Mikhail Korobov
8aa731c87f
Merge pull request #866 from adamdonahue/patch-1
...
Fix typo
2014-08-29 06:59:47 +06:00
Adam Donahue
d92914d297
Fix typo
2014-08-28 20:30:50 -04:00
yakxxx
e4689556f0
SgmlLinkExtractor - fix for parsing <area> tag with Unicode present
2014-08-28 18:55:58 +02:00
Mikhail Korobov
774ab74ad2
Merge pull request #864 from younghz/master
...
Duplicate comma in request-response.rst
2014-08-28 18:52:51 +06:00
Uyounghz
d49766a6ac
Duplicate comma in request-response.rst
2014-08-28 19:58:58 +08:00
Mikhail Korobov
c4f9e9d8e7
Merge pull request #856 from eliasdorneles/fix-embed-ipython-shell
...
fix IPython shell scope issue by using IPython.embed()
2014-08-20 00:16:33 +06:00
Pablo Hoffman
8360380db0
removed scrapy-ws.py, moved to scrapy-jsonrpc package
2014-08-19 10:30:25 -03:00
Elias Dorneles
6f50cf55a4
fix IPython shell scope issue and load IPython user config
2014-08-18 18:41:05 -03:00
Daniel Graña
841dd5f1f5
Update webservice.rst
2014-08-18 17:48:01 -03:00
Daniel Graña
d684ecad7b
Merge pull request #846 from rocioar/master
...
fix dont_merge_cookies bad behaviour when set to false on meta
2014-08-18 13:54:11 -03:00
Daniel Graña
94d00b2a26
Merge branch 'jsonrpc-split'
2014-08-15 23:28:26 -03:00
Daniel Graña
a9292cfab7
jsonrpc webservice moved to https://github.com/scrapy/scrapy-jsonrpc repository
2014-08-15 23:28:13 -03:00
Rocio Aramberri
51b0bd281d
fix dont settings on meta behaviour, add docs and tests
2014-08-15 13:47:42 -07:00
Julia Medina
6339864f95
Minor refactor in the docs and functions used in the shell command
2014-08-14 12:32:37 -03:00
Julia Medina
70f2010db1
Change error type when updating frozen settings
2014-08-14 11:59:25 -03:00
Julia Medina
3547ca6e61
Add example on running spiders outside projects
2014-08-14 11:50:33 -03:00
Julia Medina
419026615f
Deprecate Crawler.spiders attribute
2014-08-14 09:19:41 -03:00
Julia Medina
c90977ca98
Drop support for scrapy.project.crawler (And scrapy.stats consequently)
2014-08-12 14:02:56 -03:00
Julia Medina
9cbbfd8b04
Adjust spiders' utils to new SpiderManager API
2014-08-12 14:02:56 -03:00
Julia Medina
900a487682
Support multiple simultaneous LogObservers listening different crawlers
2014-08-12 14:02:56 -03:00
Julia Medina
89df18bd2b
Fix usage of crawler_process in ScrapyCommands
2014-08-12 14:02:56 -03:00
Julia Medina
d0edad4b0b
Drop support for ScrapyCommand.crawler property
2014-08-12 14:02:56 -03:00
