BackTrackZ/scrapy
BackTrackZ/scrapy
1
0
Fork 0
mirror of https://github.com/scrapy/scrapy.git synced 2025-02-22 17:44:00 +00:00
Code Releases Wiki Activity
2,508 Commits 39 Branches 119 Tags
Commit Graph

2508 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Pablo Hoffman
432251bedd Added rules to Makefile.buildbot for generating coverage reports 2010-12-15 11:13:45 -02:00
Pablo Hoffman
63c97f65f4 fixed csviter bug when called with a Response, not TextResponse 2010-12-14 18:21:07 -02:00
Pablo Hoffman
55ec7a0554 deploy command: cleanup temporary files before doing a new build 2010-12-14 17:20:31 -02:00
Pablo Hoffman
3df9070ffd fixed bin\runtests.bat windows script 2010-12-14 12:20:38 -02:00
Pablo Hoffman
002abf204f Updated item_passed signal to send passed item in 'item' argument, instead of 'output' argument, keeping backwards compatibility for the 'output' argument. Closes #273 2010-12-13 14:05:47 -02:00
Pablo Hoffman
f984d438a0 updated docs to use scrapy version on aptitude install lines 2010-12-13 14:02:42 -02:00
Pablo Hoffman
60bdb7dc0a Removed some deprecated code and modules 2010-12-13 10:34:07 -02:00
Pablo Hoffman
97a790d659 improved mechanism for tagging version with hg revision 2010-12-13 10:21:39 -02:00
Pablo Hoffman
119fd20e91 Added verbose option to 'version' command. Closes #298 2010-12-13 00:32:44 -02:00
Pablo Hoffman
6f2cea4775 added deprecation messages to queue and runserver commands 2010-12-12 20:04:35 -02:00
Pablo Hoffman
b19ff21acd scrapyd: added support for deferred spider queues 2010-12-10 15:55:40 -02:00
Martin Olveyra
02ccca01eb use safe_url_string in canonicalize_url, to avoid to convert safe characters into percent representation. Lead to errors with many sites (RFC3986). closes #297 2010-12-08 16:28:38 -02:00
Pablo Hoffman
6a1b69c93f renamed command 'scrapyd' to 'server', and deprecated 'runserver' and 'queue' commands 
--HG--
rename : scrapy/commands/scrapyd.py => scrapy/commands/server.py
 2010-11-30 20:23:27 -02:00
Pablo Hoffman
831dc818d6 scrapyd: added more information webui homepage 2010-11-30 18:43:59 -02:00
Pablo Hoffman
a3d30c35fe scrapyd: log url where web console can be accesed 2010-11-30 17:58:34 -02:00
Pablo Hoffman
823fd9822c scrapyd: fixed bug discovering the current project scrapy.cfg file 2010-11-30 16:19:17 -02:00
Pablo Hoffman
7b84591ea9 added command for starting a scrapyd server for the current project 2010-11-30 15:52:15 -02:00
Pablo Hoffman
5a46ce47ee scrapyd: add extra_sources consturctor argument, and also read scrapyd configuratoin from current project's scrapy.cfg file 2010-11-30 15:47:05 -02:00
Pablo Hoffman
c02d6db6a3 scrapyd: force application to receive config as argument 2010-11-30 15:46:24 -02:00
Pablo Hoffman
85890a5092 scrapyd: log process logfile when process starts/finishes 2010-11-30 15:45:42 -02:00
Pablo Hoffman
5c4f562ec4 scrapyd: changed keys used in poller message to _project, _spider, _job, and added link to log file in web ui 2010-11-30 13:03:20 -02:00
Pablo Hoffman
df54ed0041 Some Scrapyd enhancements: 
* added minimal web ui
* return unique id per job (spider scheduled)
* store one log per spider run (job) and rotate them, keeping the last N logs (where N is configurable through settings)
 2010-11-30 02:26:31 -02:00
Pablo Hoffman
46e5d694e6 Scrapyd: return project and version in addversion.json 2010-11-29 17:22:28 -02:00
Pablo Hoffman
bbffa59497 Some changes to Scrapyd: 
* Always start one process per spider
* Added max_proc_per_cpu option (defaults to 4)
* Return the number of spiders (instead of a list of them) in schedule.json
 2010-11-29 17:19:05 -02:00
Pablo Hoffman
42e8346d06 fixed failing test on win32 2010-11-29 10:27:12 -02:00
Pablo Hoffman
3cda681755 utils.testproc: make spawned process use the original CWD, instead of the temporary one created by twisted trial 2010-11-29 09:56:14 -02:00
Pablo Hoffman
1d726063d6 * Added tests for shell/fetch/version commands (closes #255) 
* Fixed bug causing Scrapy shell to fail if started without any argument (closes #294)
 2010-11-28 18:14:45 -02:00
Pablo Hoffman
6f82ea19de Fixed bug in addversion.json with old Twisted versions. Closes #293 2010-11-25 12:12:42 -02:00
Pablo Hoffman
2557777c39 Updated doc referring to HTTP cache middleware 2010-11-24 13:27:44 -02:00
Pablo Hoffman
d59ef48231 Fixed SgmlLinkExtractor bug which failed to recognize <base> tags when using restrict_xpaths 2010-11-23 17:28:29 -02:00
Pablo Hoffman
426b6fa100 docs/intro/install.rst: added -U flag to easy_install command 2010-11-22 13:50:19 -02:00
Pablo Hoffman
91e6753035 scrapy.bat: minor fix to support spaces in python installation dir (windows) 2010-11-22 00:39:45 -02:00
Pablo Hoffman
91a7c25797 * Made Response.meta attribute map to Request.meta attribute. Closes #290 
* Record redirected URLs in redirect middleware. Closes #291
 2010-11-18 12:51:54 -02:00
Pablo Hoffman
ac007802d6 Simplified installation guide, including lxml as alternative dependency to libxml2. Closes #280 2010-11-17 21:32:23 -02:00
Pablo Hoffman
2897061b98 Make scrapy conflict with previous versionsof the debian package 2010-11-17 17:30:22 -02:00
Pablo Hoffman
3926ca45b3 debian/control: Added scrapy/scrapyd to Provides 2010-11-17 17:08:21 -02:00
Pablo Hoffman
a034d078c8 Changed Debian packaging to use the scrapy version in the package name, so we can have multiple Scrapy versions in the same apt repo 
--HG--
rename : debian/scrapy.1 => debian/scrapy-files/scrapy.1
rename : debian/000-default => debian/scrapyd-files/000-default
rename : debian/scrapyd.upstart => debian/scrapyd.scrapyd.upstart
rename : debian/scrapy.1 => extras/scrapy.1
 2010-11-17 17:03:00 -02:00
Pablo Hoffman
67adb2a05f Always use micro versions in Scrapy from now on 2010-11-17 00:09:14 -02:00
Pablo Hoffman
5a5364d0c1 Updated documentation to point out that simplejson is now required if using Python 2.5, and to recommended switching to Python 2.6 2010-11-16 03:31:04 -02:00
Pablo Hoffman
7c712eeda1 Removed scrapy.xlib.simplejson module. Scrapy now requires simplejson if running on Python 2.5. Closes #289 2010-11-16 03:11:12 -02:00
Pablo Hoffman
28cf8625b6 Automated merge with http://hg.scrapy.org/scrapy-0.10 2010-11-16 02:37:19 -02:00
Pablo Hoffman
5ded8251d2 Fixed bug with deferred spider queues. Closes #288 2010-11-16 02:36:18 -02:00
Pablo Hoffman
b3c96c698d Fixed bug with deploy command if ~/.netrc doesn't exist. Closes #286 2010-11-13 16:38:30 -02:00
Pablo Hoffman
e1f419e9e9 canonicalize_url(): ignore case in domain names 2010-11-12 16:47:36 -02:00
Martin Olveyra
b4cc2d91f4 Allow to reapply a labelled region so to allow to use ignored regions inside repeated variants 2010-11-12 13:30:39 -02:00
Pablo Hoffman
08bbbc2f82 shell: properly refresh all vars when fetching a new request 2010-11-11 18:05:36 -02:00
Pablo Hoffman
5c18f02ade Only instantiate XPath selectors if the response is of the proper type. Closes #285 2010-11-11 18:00:02 -02:00
Pablo Hoffman
4e800e90d3 Automated merge with http://hg.scrapy.org/scrapy-0.10 2010-11-10 13:36:05 -02:00
Pablo Hoffman
7698ff2a7b Disabled help() in telnet console. Closes #284 2010-11-10 13:35:36 -02:00
Pablo Hoffman
d988ca1ec2 Some changes to scrapy deploy command: 
* changed deploy section names to [deploy:target]
* project is now passed through a -p|--project option
* version can now be set in the target configuration
* switched meaning of -l and -L options

* updated documentation accordingly
 2010-11-08 17:01:06 -02:00