BackTrackZ/scrapy
BackTrackZ/scrapy
1
0
Fork 0
mirror of https://github.com/scrapy/scrapy.git synced 2025-03-01 17:47:56 +00:00
Code Releases Wiki Activity
2,244 Commits 39 Branches 119 Tags
Commit Graph

2244 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Pablo Hoffman
6525d3fe8c bumped version to 0.9 final 0.9 2010-06-28 00:54:43 -03:00
Pablo Hoffman
66053291d6 Automated merge with http://hg.scrapy.org/scrapy-0.9 2010-06-27 19:57:54 -03:00
Pablo Hoffman
861c04fb7e made encoding explicit in test_get_meta_refresh, to avoid depending on 
unreliable UnicodeDammit criteria.
 2010-06-27 19:55:39 -03:00
Pablo Hoffman
696bb63ac5 Automated merge with http://hg.scrapy.org/scrapy-0.9 2010-06-27 19:32:47 -03:00
Pablo Hoffman
22555df56a response_httprepr: fixed error with unknown response codes (closes #169) 2010-06-27 19:32:26 -03:00
Pablo Hoffman
5408202697 Automated merge with http://hg.scrapy.org/scrapy-0.9 2010-06-27 19:23:41 -03:00
Ismael Carnales
2571e1b7aa docs: Some DjangoItem docs improvements, closes #134. Thanks tn! 2010-06-27 09:09:54 -03:00
Daniel Grana
7b80cce148 Automated merge with ssh://hg.scrapy.org/scrapy 2010-06-25 13:06:14 -03:00
Daniel Grana
9148321765 do not redirect when there is a commented meta refresh header. closes #170 2010-06-25 12:57:15 -03:00
Pablo Hoffman
a61df93a22 Automated merge with http://hg.scrapy.org/scrapy-0.9 2010-06-22 14:00:51 -03:00
Pablo Hoffman
3bbad36998 Raise when trying to set an item field value using setattr api, and added tests. 2010-06-22 14:00:31 -03:00
Pablo Hoffman
504f90c6f7 Automated merge with http://hg.scrapy.org/scrapy-0.9 2010-06-22 13:39:29 -03:00
Pablo Hoffman
baa523055f removed nltk dependency from IBL code 2010-06-22 13:38:32 -03:00
Pablo Hoffman
e665e5abb7 bumped version to 0.10-dev 2010-06-14 22:00:54 -03:00
Pablo Hoffman
8815de94ff Added tag 0.9-rc1 for changeset 8b9c31e18c08 2010-06-14 18:37:55 -03:00
Pablo Hoffman
61e374eb56 bumped version to 0.9-rc1 0.9-rc1 2010-06-14 18:37:52 -03:00
Pablo Hoffman
115e9f2162 Added FAQ entry about running Scrapy deployment. 2010-06-14 18:21:12 -03:00
Daniel Grana
f3d2ee41cc mediapipeline: bugfix error raised when media requests has not callbacks, remove item_media_{downloaded,failed} hooks in favour or request.{errback,calback}, and add tests 
--HG--
extra : rebase_source : 72172406ab4ffc748e1648b46fe976e403b87c29
 2010-06-14 12:34:52 -03:00
Pablo Hoffman
0731c42e24 added "hg purge" to make tarball 2010-06-14 09:58:17 -03:00
Pablo Hoffman
4262a0af8c moved sign_release.sh code to Makefile 2010-06-14 09:49:07 -03:00
Pablo Hoffman
d8ac4857a5 a couple of fixes to make tests pass on win32 2010-06-14 09:11:35 -03:00
Pablo Hoffman
2d3d5b6aba use mercurial revision to construct version, when building a non-final version 2010-06-14 08:59:20 -03:00
Pablo Hoffman
68e8e6ac11 removed unused code 2010-06-14 08:28:48 -03:00
Pablo Hoffman
ede1df4b4f updated copyright year, and indentation space 2010-06-14 07:16:51 -03:00
Pablo Hoffman
247fc26598 moved scrapy.tac to extras/ 
--HG--
rename : bin/scrapy.tac => extras/scrapy.tac
 2010-06-13 23:09:08 -03:00
Pablo Hoffman
09182efaff added scrapy-sqs.py to deployed scripts 2010-06-13 19:17:17 -03:00
Pablo Hoffman
37f71a9957 upstart script: exec twistd and use pidfile 2010-06-13 18:59:52 -03:00
Pablo Hoffman
91e1e0aff3 fixed bug and updated old code in googledir example project 2010-06-13 17:31:33 -03:00
Pablo Hoffman
bd16d1cd48 Added SMTP-AUTH support to scrapy.mail (closes #149) 2010-06-13 17:14:46 -03:00
Pablo Hoffman
495f23dea2 utils.serialize: added support for encoding Deferreds, and to refer spiders by name using 'spider::name' 2010-06-11 18:16:09 -03:00
Pablo Hoffman
1b083911e6 scrapy-ws.py: added stop command 2010-06-11 18:14:01 -03:00
Pablo Hoffman
ed5d7561f9 Added SQS Execution Queue, and example script to add spiders to the queue 2010-06-11 17:22:14 -03:00
olveyra
efe9811d92 Populate annotation metadata with data not used by IBL extractor. 2010-06-11 13:09:56 -03:00
Pablo Hoffman
ea8b5ddfd5 debian package: fix dh_auto_build confusing with Makefile, added scrapy-ws.py to deployed scripts 2010-06-11 12:48:35 -03:00
Pablo Hoffman
03912a6504 Added Ping Yin to AUTHORS 2010-06-11 11:33:02 -03:00
Pablo Hoffman
d13b50a234 Added sources and Makefile for building Debian package 2010-06-11 01:18:16 -03:00
Pablo Hoffman
d76276408e scrapy.service: fixed minor logging bug on win32 platform with different line endings 2010-06-10 14:50:06 -03:00
Pablo Hoffman
a8b80f3e2f scrapy.service: added support for logging stdout/stderr tails of finished processes 2010-06-10 14:08:54 -03:00
Pablo Hoffman
a33e8b507f scrapy.service: fixed bug with process respawning 2010-06-10 13:39:45 -03:00
Pablo Hoffman
075b59f4af some improvements and fixes to scrapy.service 2010-06-10 11:51:46 -03:00
Pablo Hoffman
6a33d6c4d0 * Added Scrapy Web Service with documentation and tests. 
* Marked Web Console as deprecated.
* Removed Web Console documentation to discourage its use.
 2010-06-09 13:46:22 -03:00
Pablo Hoffman
2499dfee5e removed obsolete test 2010-06-09 13:06:05 -03:00
Daniel Grana
62f5c61a9d fix broken request tests. refs #166 2010-06-09 00:44:18 -03:00
Pablo Hoffman
73305b1eb3 Added support for Requests without callbacks (#166) - the Spider.parse() method 
is used in those cases.

Also removed Request.deferred attribute.
 2010-06-08 18:18:02 -03:00
Pablo Hoffman
76ed9d442b Relocated some modules: 
* scrapy.spider.middelware moved to scrapy.core.spidermw
* scrapy.core.scheduler.schedulers to scrapy.core.scheduler
* scrapy.core.scheduler.middleware to scrapy.core.schedulermw

Also removed dir: scrapy/core/scheduler/

--HG--
rename : scrapy/core/scheduler/schedulers.py => scrapy/core/scheduler.py
rename : scrapy/core/scheduler/middleware.py => scrapy/core/schedulermw.py
rename : scrapy/spider/middleware.py => scrapy/core/spidermw.py
 2010-06-07 15:11:25 -03:00
Pablo Hoffman
72df5cb7ef removed unused code 2010-06-03 01:07:40 -03:00
Pablo Hoffman
38b5793152 Some changes to telnet console: 
* moved module from scrapy.management.telnet to scrapy.telnet (to minimize
  nested modules)
* added signal for updating telnet console variables (fixes #165)

--HG--
rename : scrapy/management/telnet.py => scrapy/telnet.py
 2010-06-02 17:49:18 -03:00
Pablo Hoffman
4595c92cc2 Core logic improvement: wait for Downloader and Scraper to close the spiders before going on and finish closing them 2010-06-01 13:49:01 -03:00
Pablo Hoffman
9523cab25c Fixed bug that was causing the engine to notify the manager of spider closes too early 2010-06-01 11:07:04 -03:00
Ping Yin
fcdc4ee7d9 downloadermiddleware/redirect: always do "HEAD" if origin request method is HEAD 
Signed-off-by: Ping Yin <pkufranky@gmail.com>
 2010-05-04 16:11:45 +08:00