Pablo Hoffman
7458092eef
added spider contracts to release notes and warn that its API is still subject to change
2012-09-29 03:06:30 -03:00
Pablo Hoffman
34f1477317
make tests code python 2.6 compatible
2012-09-28 18:55:12 -03:00
Pablo Hoffman
c380910b40
Merge pull request #167 from alexcepoi/sep-017
...
Spider contracts (SEP-017)
2012-09-28 13:57:07 -07:00
Pablo Hoffman
a5f8943d3e
added change introduced in previous commit to release notes
2012-09-26 13:17:01 -03:00
Pablo Hoffman
4ca35dba87
Merge pull request #175 from euphoris/master
...
add options -o and -t to the runspider command
2012-09-26 09:15:40 -07:00
Jae-Myoung Yu
e94bd8187c
add options -o and -t to the runspider command
...
copy the codes from commands/crawl.py to commands/runspider.py
2012-09-25 09:38:00 +09:00
Pablo Hoffman
a6eacf2c9b
Merge pull request #172 from artem-dev/new_git_version
...
nicer deploy versions for git repos if tags are available
2012-09-21 07:39:47 -07:00
Artem Bogomyagkov
7685cb14a4
nicer deploy versions for git repos if tags are available
2012-09-21 12:34:45 +03:00
Alex Cepoi
73e6bc1b10
remove unused import
2012-09-21 00:54:11 +02:00
Alex Cepoi
11d29c7005
SEP-017 contracts: add tests and minor improvements
2012-09-21 00:12:46 +02:00
Pablo Hoffman
b46b5a6ef0
Documented AutoThrottle extension and added to extensions available by
...
default. Also deprecated concurrency and delay settings, in favour of
using the standard Scrapy ones.
2012-09-20 18:52:57 -03:00
Pablo Hoffman
c1cbc5de3f
renamed CONTRIBUTING to CONTRIBUTING.md so that links are rendered as links in github
2012-09-19 13:58:58 -03:00
Pablo Hoffman
f3516aed69
added CONTRIBUTING file, for github: https://github.com/blog/1184-contributing-guidelines
2012-09-19 13:54:16 -03:00
Pablo Hoffman
e8eacd6f9e
minor code simplification
2012-09-19 13:28:34 -03:00
Pablo Hoffman
7afedb89e9
backwards compatibility support for scrapy.conf.settings singleton for when scrapy.conf is imported *before* calling scrapy.cmdline.execute(), such as in custom bot runners
2012-09-19 12:44:59 -03:00
Pablo Hoffman
768a483953
merge adding missing changes from previous commit
2012-09-19 03:19:03 -03:00
Pablo Hoffman
c7f8219901
- removed scrapy.conf singleton from scrapy.log, scrapy.responsetypes,
...
scrapy.http.response.text, scrapy.selector
- fixed bug with scrapy.conf.settings backwards compatibility support
- added facility to notify (and provide some guidelines) about deprecated/obsolete settings
2012-09-19 03:03:34 -03:00
Pablo Hoffman
aadf7ff143
- removed scrapy.conf singleton from scrapy.log, scrapy.responsetypes,
...
scrapy.http.response.text, scrapy.selector
- fixed bug with scrapy.conf.settings backwards compatibility support
- added facility to notify (and provide some guidelines) about deprecated/obsolete settings
2012-09-19 02:56:38 -03:00
Pablo Hoffman
391cc06063
Merge pull request #174 from stav/master
...
Selector documentation typo fixes
2012-09-18 11:19:23 -07:00
stav
303e13f616
selector documentation typos
2012-09-18 12:56:52 -05:00
Pablo Hoffman
3d736e657f
fixed typo in doc
2012-09-18 10:51:01 -03:00
Pablo Hoffman
eed6eb49da
make DBM the new default storage backend for HTTP cache middleware, simplified DBM storage backend code to avoid dealing with many spiders at once (not needed), and update httpcache stats names (hit -> hits, miss -> misses)
2012-09-17 10:11:07 -03:00
Pablo Hoffman
8f2dda12cc
removed another instance of scrapy.conf.settings singleton, this time from scrapy.utils.trackref. From now on, trackrefs functionality will be always enabled as it imposes a very minimal performance overhead
2012-09-16 21:21:44 -03:00
Pablo Hoffman
cd823018aa
removed another instance of scrapy.conf.settings singleton, this time from scrapy.cmdline (which manages scrapy command line tool), by moving the get_project_settings() function to scrapy.utils.project
2012-09-16 20:53:52 -03:00
Pablo Hoffman
9685c24059
restored stats_spider_{opened,closing,closed} signals in backwards compatibility mode
2012-09-14 12:38:00 -03:00
Pablo Hoffman
81ed2d2d0b
major Stats Collection refactoring: removed separation of global/per-spider stats, removed stats-related signals (stats_spider_opened, etc). Stats are much simpler now, backwards compatibility is kept on the Stats Collector API.
2012-09-14 12:31:33 -03:00
Pablo Hoffman
8b48420559
better names for scheduler stats
2012-09-13 20:07:11 -03:00
Pablo Hoffman
263e302b6a
added more scheduler stats
2012-09-13 19:46:45 -03:00
Pablo Hoffman
d6867f7984
removed sphinx warnings about duplicate reference names 'this page'
2012-09-13 15:26:45 -03:00
Pablo Hoffman
a874964ad4
renamed 'XPath Selectors' title to just 'Selectors'
2012-09-13 15:24:44 -03:00
Pablo Hoffman
acb8895e1a
changed note in scrapyd doc to use sphinx notes
2012-09-13 15:22:59 -03:00
Pablo Hoffman
26f1d5cb48
Merge pull request #171 from artem-dev/scrapyd_job_times
...
added start and stop times for scrapyd list jobs web service
2012-09-13 11:16:01 -07:00
Artem Bogomyagkov
4d8f253912
commited doc file missed from prev commit
2012-09-12 11:12:59 +03:00
Artem Bogomyagkov
5dde26d3c8
simplified code for finished jobs times, updated docs for scrapyd
2012-09-12 11:10:49 +03:00
Pablo Hoffman
7ace23c385
removed another instance of scrapy.conf singleton, this time from test_cmdline
2012-09-11 16:37:55 -03:00
Pablo Hoffman
8642645109
removed another instance of scrapy.conf singleton
2012-09-11 16:34:58 -03:00
Pablo Hoffman
7ef593c5c2
refactored MailSender to get rid of scrapy.conf singleton, also removed ill-designed scrapy.mail.mail_sent signal
2012-09-11 16:27:19 -03:00
Artem Bogomyagkov
a42a8cafef
added start and stop times for scrapyd list jobs web service
2012-09-11 21:06:36 +03:00
Alex Cepoi
bf8dc61fb7
SEP-017 contracts: pretty-printing and docs
2012-09-10 23:17:27 +02:00
Pablo Hoffman
1e2efe5664
scrapy deploy: retry setup.py bdist_egg on EINTR. closes #9
2012-09-07 15:40:26 -03:00
Pablo Hoffman
3c05fbb6a9
removed redundant comment
2012-09-05 15:54:23 -03:00
Pablo Hoffman
44fe4c6b78
set missing 'settings' attribute in FeedExports
2012-09-05 15:54:13 -03:00
Pablo Hoffman
0135a5364e
removed debugging print
2012-09-05 15:53:30 -03:00
Pablo Hoffman
7bf0af7a58
fixed some bugs in get_func_args() and added more test cases
2012-09-04 19:01:20 -03:00
Pablo Hoffman
251d9a9033
backwards compatibility support for FeedExport's that don't receive a settings object in their constructor
2012-09-04 18:13:27 -03:00
Pablo Hoffman
caa64908d2
added tests for get_func_args() and support for more cases
2012-09-04 18:07:44 -03:00
Pablo Hoffman
fff2871828
added doc section (and FAQ) about spider arguments
2012-09-04 14:49:30 -03:00
Pablo Hoffman
241abcb8aa
scrapyd: log errors in API calls to the scrapyd log
2012-09-04 14:22:21 -03:00
Pablo Hoffman
f4a17ec272
removed references to Scrapy Snippets site
2012-09-03 22:19:15 -03:00
Pablo Hoffman
b901e64044
replaced memory usage acounting with (more portable) resource module, removed scrapy.utils.memory module. closes #161
2012-09-03 19:28:16 -03:00
