Pablo Hoffman
|
eed6eb49da
|
make DBM the new default storage backend for HTTP cache middleware, simplified DBM storage backend code to avoid dealing with many spiders at once (not needed), and update httpcache stats names (hit -> hits, miss -> misses)
|
2012-09-17 10:11:07 -03:00 |
|
Pablo Hoffman
|
8f2dda12cc
|
removed another instance of scrapy.conf.settings singleton, this time from scrapy.utils.trackref. From now on, trackrefs functionality will be always enabled as it imposes a very minimal performance overhead
|
2012-09-16 21:21:44 -03:00 |
|
Pablo Hoffman
|
cd823018aa
|
removed another instance of scrapy.conf.settings singleton, this time from scrapy.cmdline (which manages scrapy command line tool), by moving the get_project_settings() function to scrapy.utils.project
|
2012-09-16 20:53:52 -03:00 |
|
Pablo Hoffman
|
9685c24059
|
restored stats_spider_{opened,closing,closed} signals in backwards compatibility mode
|
2012-09-14 12:38:00 -03:00 |
|
Pablo Hoffman
|
81ed2d2d0b
|
major Stats Collection refactoring: removed separation of global/per-spider stats, removed stats-related signals (stats_spider_opened, etc). Stats are much simpler now, backwards compatibility is kept on the Stats Collector API.
|
2012-09-14 12:31:33 -03:00 |
|
Pablo Hoffman
|
8b48420559
|
better names for scheduler stats
|
2012-09-13 20:07:11 -03:00 |
|
Pablo Hoffman
|
263e302b6a
|
added more scheduler stats
|
2012-09-13 19:46:45 -03:00 |
|
Pablo Hoffman
|
d6867f7984
|
removed sphinx warnings about duplicate reference names 'this page'
|
2012-09-13 15:26:45 -03:00 |
|
Pablo Hoffman
|
a874964ad4
|
renamed 'XPath Selectors' title to just 'Selectors'
|
2012-09-13 15:24:44 -03:00 |
|
Pablo Hoffman
|
acb8895e1a
|
changed note in scrapyd doc to use sphinx notes
|
2012-09-13 15:22:59 -03:00 |
|
Pablo Hoffman
|
26f1d5cb48
|
Merge pull request #171 from artem-dev/scrapyd_job_times
added start and stop times for scrapyd list jobs web service
|
2012-09-13 11:16:01 -07:00 |
|
Artem Bogomyagkov
|
4d8f253912
|
commited doc file missed from prev commit
|
2012-09-12 11:12:59 +03:00 |
|
Artem Bogomyagkov
|
5dde26d3c8
|
simplified code for finished jobs times, updated docs for scrapyd
|
2012-09-12 11:10:49 +03:00 |
|
Pablo Hoffman
|
7ace23c385
|
removed another instance of scrapy.conf singleton, this time from test_cmdline
|
2012-09-11 16:37:55 -03:00 |
|
Pablo Hoffman
|
8642645109
|
removed another instance of scrapy.conf singleton
|
2012-09-11 16:34:58 -03:00 |
|
Pablo Hoffman
|
7ef593c5c2
|
refactored MailSender to get rid of scrapy.conf singleton, also removed ill-designed scrapy.mail.mail_sent signal
|
2012-09-11 16:27:19 -03:00 |
|
Artem Bogomyagkov
|
a42a8cafef
|
added start and stop times for scrapyd list jobs web service
|
2012-09-11 21:06:36 +03:00 |
|
Pablo Hoffman
|
1e2efe5664
|
scrapy deploy: retry setup.py bdist_egg on EINTR. closes #9
|
2012-09-07 15:40:26 -03:00 |
|
Pablo Hoffman
|
3c05fbb6a9
|
removed redundant comment
|
2012-09-05 15:54:23 -03:00 |
|
Pablo Hoffman
|
44fe4c6b78
|
set missing 'settings' attribute in FeedExports
|
2012-09-05 15:54:13 -03:00 |
|
Pablo Hoffman
|
0135a5364e
|
removed debugging print
|
2012-09-05 15:53:30 -03:00 |
|
Pablo Hoffman
|
7bf0af7a58
|
fixed some bugs in get_func_args() and added more test cases
|
2012-09-04 19:01:20 -03:00 |
|
Pablo Hoffman
|
251d9a9033
|
backwards compatibility support for FeedExport's that don't receive a settings object in their constructor
|
2012-09-04 18:13:27 -03:00 |
|
Pablo Hoffman
|
caa64908d2
|
added tests for get_func_args() and support for more cases
|
2012-09-04 18:07:44 -03:00 |
|
Pablo Hoffman
|
fff2871828
|
added doc section (and FAQ) about spider arguments
|
2012-09-04 14:49:30 -03:00 |
|
Pablo Hoffman
|
241abcb8aa
|
scrapyd: log errors in API calls to the scrapyd log
|
2012-09-04 14:22:21 -03:00 |
|
Pablo Hoffman
|
f4a17ec272
|
removed references to Scrapy Snippets site
|
2012-09-03 22:19:15 -03:00 |
|
Pablo Hoffman
|
b901e64044
|
replaced memory usage acounting with (more portable) resource module, removed scrapy.utils.memory module. closes #161
|
2012-09-03 19:28:16 -03:00 |
|
Pablo Hoffman
|
e2f9daac67
|
fixed formatting in scrapyd release notes
|
2012-09-03 16:58:58 -03:00 |
|
Pablo Hoffman
|
dd1398b280
|
removed another instance of scrapy.conf.settings singleton, this time from download handlers
|
2012-09-03 16:44:18 -03:00 |
|
Daniel Graña
|
3b768ae989
|
oops. remove santas debug line
|
2012-08-31 20:10:32 -03:00 |
|
Pablo Hoffman
|
c6435c5aa7
|
restored scrapyd log message
|
2012-08-31 18:49:20 -03:00 |
|
Pablo Hoffman
|
7c8af83138
|
added issue role to documentation
|
2012-08-31 18:39:30 -03:00 |
|
Pablo Hoffman
|
3891c20840
|
Merge pull request #164 from dangra/lazy-log-formatting
format log lines lazily in case they are dropped by loglevels
|
2012-08-31 14:33:41 -07:00 |
|
Daniel Graña
|
a2d22307ad
|
update news.rst
|
2012-08-31 17:38:40 -03:00 |
|
Daniel Graña
|
dcef7b03c1
|
format log lines lazily in case they are dropped by loglevels
|
2012-08-31 17:14:35 -03:00 |
|
Pablo Hoffman
|
be206ca5ab
|
added process_start_requests method to spider middlewares
|
2012-08-31 16:41:50 -03:00 |
|
Pablo Hoffman
|
b920674666
|
corrected minor issue with doc references
|
2012-08-31 16:41:49 -03:00 |
|
Pablo Hoffman
|
a2d5a304b1
|
Merge pull request #168 from alexcepoi/parse-fixes
parse command: fix crash in case of bad callback
|
2012-08-31 09:12:19 -07:00 |
|
Alex Cepoi
|
0350bed8fd
|
parse command: fix crash in case of bad callback
|
2012-08-31 16:43:04 +02:00 |
|
Pablo Hoffman
|
4ec99117d3
|
fixed minor doc typo
|
2012-08-30 11:56:30 -03:00 |
|
Pablo Hoffman
|
94b40162a9
|
fixed tests to work on windows
|
2012-08-30 11:24:29 -03:00 |
|
Pablo Hoffman
|
3904f5f55a
|
more reliable way to set subprocess PYTHONPATH in tests
|
2012-08-30 10:09:20 -03:00 |
|
Pablo Hoffman
|
4a5f70278f
|
minor tidy up to installation guide windows notes
|
2012-08-29 15:44:24 -03:00 |
|
Pablo Hoffman
|
098d892c03
|
simplified installation guide to only mention pip/easy_install mechanism, and provide hints for Windows users
|
2012-08-29 15:37:05 -03:00 |
|
Pablo Hoffman
|
6217f108cc
|
avoid random "Interrupted system call" errors
|
2012-08-29 11:44:00 -03:00 |
|
Pablo Hoffman
|
8aa46a4b5d
|
removed another usage of scrapy.conf.settings singleton
|
2012-08-29 11:43:01 -03:00 |
|
Pablo Hoffman
|
280d9d4094
|
added file missed in previous commit
|
2012-08-29 11:26:52 -03:00 |
|
Pablo Hoffman
|
70f8e517a1
|
promoted DjangoItem to main contrib
|
2012-08-29 11:23:11 -03:00 |
|
Pablo Hoffman
|
52151e8703
|
fixed bug in FeedExports extension, introduced in previous commit
|
2012-08-28 19:32:43 -03:00 |
|