BackTrackZ/scrapy
BackTrackZ/scrapy
1
0
Fork 0
mirror of https://github.com/scrapy/scrapy.git synced 2025-02-25 11:24:24 +00:00
Code Releases Wiki Activity
2,532 Commits 39 Branches 119 Tags
Commit Graph

2532 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Pablo Hoffman
65fc2fbd1f Set CONCURRENT_SPIDERS=1 in Scrapyd to force one spider per process 2011-02-04 13:30:01 -02:00
Pablo Hoffman
b1c89508f5 fixed wrong changes commited in previous changeset 2011-01-28 16:22:39 -02:00
Pablo Hoffman
4361150494 added missing scrapyd/default_scrapyd.conf file to MANIFEST.in 2011-01-28 16:21:00 -02:00
Pablo Hoffman
09f084c220 simplified scrapy shell code after recent changes. refs #306 2011-01-13 13:11:39 -02:00
Pablo Hoffman
0aac226b42 Fixed bug in Scrapy shell's fetch() which wasn't updating local variables properly. Closes #306 2011-01-13 13:08:11 -02:00
LucianU
0c5f605b0e The xmlfeed.tmpl file didn't use the naming convention specific of the XMLFeedSpider. Namely, it used parse_item (which has been deprecated) instead of parse_node and it didn't show the iterator and itertag attributes. 2011-01-11 17:23:46 -02:00
Pablo Hoffman
7dc521c56e make scrapyd package depend on specific scrapy version 2011-01-11 17:03:51 -02:00
Pablo Hoffman
048044c1f8 A couple of changes to fix #303: 
* improved detection of inside-project environments
* make list command faster (by only instantiating the spider manger)
* print a warning when extensions (middlewares, etc) are disabled with a message on NotConfigured exception
* assert that scrapy configuration hasn't been loaded in scrapyd.runner
* simplified IgnoreRequest exception, to avoid loading settings when importing scrapy.exceptions
* added test to make sure certain modules don't cause scrapy.conf module to be
  loaded, to ensure the scrapyd runner bootstraping performs properly
 2011-01-05 15:59:43 -02:00
Pablo Hoffman
48b30ba939 fixed compatibility with python 2.5 and removed unused code 2011-01-05 12:03:54 -02:00
Pablo Hoffman
ac8166251e make scrapy*-0.12 packages conflict with scrapy*-0.11 packages 2011-01-04 13:56:28 -02:00
Pablo Hoffman
b56e933be9 bumped version to 0.12 in documentation 2011-01-02 17:28:33 -02:00
Pablo Hoffman
5879389ad0 Bumped version to 0.12 2011-01-02 16:16:40 -02:00
Vikas Dhiman
fe218abbfd scrapy.utils.memory.get_vmvalue_from_procfs() 
causes test case failure on SunOS 5.10 i86pc. Modified it to support SunOS 5.10
 2011-01-02 15:46:11 -02:00
Shane Evans
aebe5d5073 make inside_project work with SCRAPY_SETTINGS_MODULE. Closes #300 2010-12-28 15:34:39 -02:00
Pablo Hoffman
3d8b368fc6 scrapyd: use runner from config (if not specified) on get_spider_list() 2010-12-28 11:16:58 -02:00
Pablo Hoffman
fa644f7a5e Some simplifications to Scrapyd architecture and internals: 
- launcher no longer knows about egg storage
- removed get_spider_list_from_eggifile() file and replaced by simpler
  get_spider_list() which doesn't receive en egg file as argument
- changed "egg runner" name to just "runner" to reflect the fact that it
  doesn't necesarilly run eggs (though it does in the default case)

--HG--
rename : scrapyd/eggrunner.py => scrapyd/runner.py
 2010-12-27 16:22:32 -02:00
Pablo Hoffman
9cd649b3a0 scrapyd: populate SCRAPY_SPIDER and SCRAPY_JOB environment variables 2010-12-26 19:32:56 -02:00
Pablo Hoffman
1c8d74eb5b scrapyd: populate SCRAPY_SLOT environment variable with the scrapyd slot number 2010-12-24 12:47:59 -02:00
Martin Olveyra
d9a3df45c6 Remove deprecated match common prefix feature from IBL code 2010-12-23 14:40:22 -02:00
Pablo Hoffman
633ebc4c43 minor indentation improvement 2010-12-23 13:04:49 -02:00
Pablo Hoffman
db07a9a938 Added notice to documentation, pointing dev to stable versions and viceversa 2010-12-23 13:03:40 -02:00
Pablo Hoffman
544308d6d0 updated ubuntu repos doc, in preparation for the 0.11 release 2010-12-21 11:02:56 -02:00
Pablo Hoffman
fff22c269a Added test to make sure custom mime.types shipped with Scrapy is loaded, and made Scrapy more egg-friendly by using pkutil.get_data() 2010-12-20 15:16:21 -02:00
Pablo Hoffman
87aa63da2b moved coverage report script to extras/ 2010-12-15 12:08:59 -02:00
Pablo Hoffman
432251bedd Added rules to Makefile.buildbot for generating coverage reports 2010-12-15 11:13:45 -02:00
Pablo Hoffman
63c97f65f4 fixed csviter bug when called with a Response, not TextResponse 2010-12-14 18:21:07 -02:00
Pablo Hoffman
55ec7a0554 deploy command: cleanup temporary files before doing a new build 2010-12-14 17:20:31 -02:00
Pablo Hoffman
3df9070ffd fixed bin\runtests.bat windows script 2010-12-14 12:20:38 -02:00
Pablo Hoffman
002abf204f Updated item_passed signal to send passed item in 'item' argument, instead of 'output' argument, keeping backwards compatibility for the 'output' argument. Closes #273 2010-12-13 14:05:47 -02:00
Pablo Hoffman
f984d438a0 updated docs to use scrapy version on aptitude install lines 2010-12-13 14:02:42 -02:00
Pablo Hoffman
60bdb7dc0a Removed some deprecated code and modules 2010-12-13 10:34:07 -02:00
Pablo Hoffman
97a790d659 improved mechanism for tagging version with hg revision 2010-12-13 10:21:39 -02:00
Pablo Hoffman
119fd20e91 Added verbose option to 'version' command. Closes #298 2010-12-13 00:32:44 -02:00
Pablo Hoffman
6f2cea4775 added deprecation messages to queue and runserver commands 2010-12-12 20:04:35 -02:00
Pablo Hoffman
b19ff21acd scrapyd: added support for deferred spider queues 2010-12-10 15:55:40 -02:00
Martin Olveyra
02ccca01eb use safe_url_string in canonicalize_url, to avoid to convert safe characters into percent representation. Lead to errors with many sites (RFC3986). closes #297 2010-12-08 16:28:38 -02:00
Pablo Hoffman
6a1b69c93f renamed command 'scrapyd' to 'server', and deprecated 'runserver' and 'queue' commands 
--HG--
rename : scrapy/commands/scrapyd.py => scrapy/commands/server.py
 2010-11-30 20:23:27 -02:00
Pablo Hoffman
831dc818d6 scrapyd: added more information webui homepage 2010-11-30 18:43:59 -02:00
Pablo Hoffman
a3d30c35fe scrapyd: log url where web console can be accesed 2010-11-30 17:58:34 -02:00
Pablo Hoffman
823fd9822c scrapyd: fixed bug discovering the current project scrapy.cfg file 2010-11-30 16:19:17 -02:00
Pablo Hoffman
7b84591ea9 added command for starting a scrapyd server for the current project 2010-11-30 15:52:15 -02:00
Pablo Hoffman
5a46ce47ee scrapyd: add extra_sources consturctor argument, and also read scrapyd configuratoin from current project's scrapy.cfg file 2010-11-30 15:47:05 -02:00
Pablo Hoffman
c02d6db6a3 scrapyd: force application to receive config as argument 2010-11-30 15:46:24 -02:00
Pablo Hoffman
85890a5092 scrapyd: log process logfile when process starts/finishes 2010-11-30 15:45:42 -02:00
Pablo Hoffman
5c4f562ec4 scrapyd: changed keys used in poller message to _project, _spider, _job, and added link to log file in web ui 2010-11-30 13:03:20 -02:00
Pablo Hoffman
df54ed0041 Some Scrapyd enhancements: 
* added minimal web ui
* return unique id per job (spider scheduled)
* store one log per spider run (job) and rotate them, keeping the last N logs (where N is configurable through settings)
 2010-11-30 02:26:31 -02:00
Pablo Hoffman
46e5d694e6 Scrapyd: return project and version in addversion.json 2010-11-29 17:22:28 -02:00
Pablo Hoffman
bbffa59497 Some changes to Scrapyd: 
* Always start one process per spider
* Added max_proc_per_cpu option (defaults to 4)
* Return the number of spiders (instead of a list of them) in schedule.json
 2010-11-29 17:19:05 -02:00
Pablo Hoffman
42e8346d06 fixed failing test on win32 2010-11-29 10:27:12 -02:00
Pablo Hoffman
3cda681755 utils.testproc: make spawned process use the original CWD, instead of the temporary one created by twisted trial 2010-11-29 09:56:14 -02:00