Pablo Hoffman
26f1d5cb48
Merge pull request #171 from artem-dev/scrapyd_job_times
...
added start and stop times for scrapyd list jobs web service
2012-09-13 11:16:01 -07:00
Artem Bogomyagkov
4d8f253912
commited doc file missed from prev commit
2012-09-12 11:12:59 +03:00
Artem Bogomyagkov
5dde26d3c8
simplified code for finished jobs times, updated docs for scrapyd
2012-09-12 11:10:49 +03:00
Pablo Hoffman
7ace23c385
removed another instance of scrapy.conf singleton, this time from test_cmdline
2012-09-11 16:37:55 -03:00
Pablo Hoffman
8642645109
removed another instance of scrapy.conf singleton
2012-09-11 16:34:58 -03:00
Pablo Hoffman
7ef593c5c2
refactored MailSender to get rid of scrapy.conf singleton, also removed ill-designed scrapy.mail.mail_sent signal
2012-09-11 16:27:19 -03:00
Artem Bogomyagkov
a42a8cafef
added start and stop times for scrapyd list jobs web service
2012-09-11 21:06:36 +03:00
Alex Cepoi
bf8dc61fb7
SEP-017 contracts: pretty-printing and docs
2012-09-10 23:17:27 +02:00
Pablo Hoffman
1e2efe5664
scrapy deploy: retry setup.py bdist_egg on EINTR. closes #9
2012-09-07 15:40:26 -03:00
Pablo Hoffman
3c05fbb6a9
removed redundant comment
2012-09-05 15:54:23 -03:00
Pablo Hoffman
44fe4c6b78
set missing 'settings' attribute in FeedExports
2012-09-05 15:54:13 -03:00
Pablo Hoffman
0135a5364e
removed debugging print
2012-09-05 15:53:30 -03:00
Pablo Hoffman
7bf0af7a58
fixed some bugs in get_func_args() and added more test cases
2012-09-04 19:01:20 -03:00
Pablo Hoffman
251d9a9033
backwards compatibility support for FeedExport's that don't receive a settings object in their constructor
2012-09-04 18:13:27 -03:00
Pablo Hoffman
caa64908d2
added tests for get_func_args() and support for more cases
2012-09-04 18:07:44 -03:00
Pablo Hoffman
fff2871828
added doc section (and FAQ) about spider arguments
2012-09-04 14:49:30 -03:00
Pablo Hoffman
241abcb8aa
scrapyd: log errors in API calls to the scrapyd log
2012-09-04 14:22:21 -03:00
Pablo Hoffman
f4a17ec272
removed references to Scrapy Snippets site
2012-09-03 22:19:15 -03:00
Pablo Hoffman
b901e64044
replaced memory usage acounting with (more portable) resource module, removed scrapy.utils.memory module. closes #161
2012-09-03 19:28:16 -03:00
Pablo Hoffman
e2f9daac67
fixed formatting in scrapyd release notes
2012-09-03 16:58:58 -03:00
Pablo Hoffman
dd1398b280
removed another instance of scrapy.conf.settings singleton, this time from download handlers
2012-09-03 16:44:18 -03:00
Daniel Graña
3b768ae989
oops. remove santas debug line
2012-08-31 20:10:32 -03:00
Pablo Hoffman
c6435c5aa7
restored scrapyd log message
2012-08-31 18:49:20 -03:00
Pablo Hoffman
7c8af83138
added issue role to documentation
2012-08-31 18:39:30 -03:00
Pablo Hoffman
3891c20840
Merge pull request #164 from dangra/lazy-log-formatting
...
format log lines lazily in case they are dropped by loglevels
2012-08-31 14:33:41 -07:00
Daniel Graña
a2d22307ad
update news.rst
2012-08-31 17:38:40 -03:00
Daniel Graña
dcef7b03c1
format log lines lazily in case they are dropped by loglevels
2012-08-31 17:14:35 -03:00
Pablo Hoffman
be206ca5ab
added process_start_requests method to spider middlewares
2012-08-31 16:41:50 -03:00
Pablo Hoffman
b920674666
corrected minor issue with doc references
2012-08-31 16:41:49 -03:00
Pablo Hoffman
a2d5a304b1
Merge pull request #168 from alexcepoi/parse-fixes
...
parse command: fix crash in case of bad callback
2012-08-31 09:12:19 -07:00
Alex Cepoi
0350bed8fd
parse command: fix crash in case of bad callback
2012-08-31 16:43:04 +02:00
Pablo Hoffman
4ec99117d3
fixed minor doc typo
2012-08-30 11:56:30 -03:00
Pablo Hoffman
94b40162a9
fixed tests to work on windows
2012-08-30 11:24:29 -03:00
Pablo Hoffman
3904f5f55a
more reliable way to set subprocess PYTHONPATH in tests
2012-08-30 10:09:20 -03:00
Pablo Hoffman
4a5f70278f
minor tidy up to installation guide windows notes
2012-08-29 15:44:24 -03:00
Pablo Hoffman
098d892c03
simplified installation guide to only mention pip/easy_install mechanism, and provide hints for Windows users
2012-08-29 15:37:05 -03:00
Alex Cepoi
6f1a5d8db6
SEP-017 contracts: various minor changes
2012-08-29 18:31:10 +02:00
Pablo Hoffman
6217f108cc
avoid random "Interrupted system call" errors
2012-08-29 11:44:00 -03:00
Pablo Hoffman
8aa46a4b5d
removed another usage of scrapy.conf.settings singleton
2012-08-29 11:43:01 -03:00
Pablo Hoffman
280d9d4094
added file missed in previous commit
2012-08-29 11:26:52 -03:00
Pablo Hoffman
70f8e517a1
promoted DjangoItem to main contrib
2012-08-29 11:23:11 -03:00
Pablo Hoffman
52151e8703
fixed bug in FeedExports extension, introduced in previous commit
2012-08-28 19:32:43 -03:00
Pablo Hoffman
da234af456
Merge branch 'singleton_removal'
2012-08-28 18:42:33 -03:00
Pablo Hoffman
babfc6e79b
Updated documentation after singleton removal changes.
...
Also removed some unused code and made some minor additional
refactoring.
2012-08-28 18:35:57 -03:00
Alex Cepoi
901987154e
SEP-017 contracts
...
* load contracts from settings
* refactored contracts manager
* fixed callback bug, which caused responses to be evaluated with a
wrong callback sometimes
* "returns" contract
2012-08-24 16:42:13 +02:00
Pablo Hoffman
c95f717ef2
fixed typo in tests
2012-08-22 13:50:23 -03:00
Pablo Hoffman
5fa5c4545b
use ScrapyJSONEncoder JsonItemExporter & JsonLinesItemExporter, to support nested items properly
2012-08-22 13:46:50 -03:00
Pablo Hoffman
1e12c92b8f
Removed signals/stats singletons
...
This change removes singletons for stats collection and signal
dispatching facilities, by making them a member of the Crawler class.
Here are some examples to illustrates the old and new API:
Signals - before:
from scrapy import signals
from scrapy.xlib.pydispatch import dispatcher
dispatcher.connect(self.spider_opened, signals.spider_opened)
Signals - now:
from scrapy import signals
crawler.signals.connect(self.spider.opened, signals.spider_opened)
Stats collection - before:
from scrapy.stats import stats
stats.inc_value('foo')
Stats collection - now:
crawler.stats.inc_value('foo')
Backwards compatibility was retained as much as possible and the old API
has been properly flagged with deprecation warnings.
2012-08-21 18:32:30 -03:00
Pablo Hoffman
36f47a4aec
Removed per-spider settings concept, and scrapy.conf.settings singleton from many extensions and middlewares. There are some still remaining, that will be removed in future commits
2012-08-21 17:27:45 -03:00
Alex Cepoi
99b76eaa2c
SEP-017 contracts: first draft
2012-08-21 02:47:35 +02:00
