BackTrackZ/scrapy
BackTrackZ/scrapy
1
0
Fork 0
mirror of https://github.com/scrapy/scrapy.git synced 2025-02-27 13:05:38 +00:00
Code Releases Wiki Activity
3,409 Commits 39 Branches 119 Tags
Commit Graph

3409 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Pablo Hoffman
2df010a972 Merge pull request #241 from zuhao/master 
Fix url_has_any_extension bug
 2013-02-11 05:29:14 -08:00
Zuhao Wan
27ca25472f Fix url_has_any_extension bug 2013-02-11 17:19:31 +08:00
Jordi Llonch
6d4b764f29 spider.log to pass keyword arguments into twisted log.msg 2013-02-11 05:00:06 +11:00
Daniel Graña
c0a7040f16 add PHONYs so build does not match build/ path 2013-02-08 15:07:00 -02:00
Daniel Graña
910effd145 get scrapy version from package data 2013-02-06 11:44:26 -02:00
Daniel Graña
79cb0318d0 update travis-ci conf 2013-02-06 10:36:08 -02:00
Daniel Graña
5db45b3825 remove scrapyd, it was migrated to its own repository 2013-02-06 05:24:07 +00:00
Daniel Graña
33ca295129 Merge pull request #235 from whodatninja/patch-1 
Fix typo labeling attrs type bool instead of list
 2013-02-05 17:27:10 -08:00
whodatninja
8e3b5baac5 Fix typo labeling attrs type bool instead of list 2013-02-05 15:10:41 -05:00
Daniel Graña
f293d08e7a Merge pull request #234 from darkrho/httpcache-dbm-module-fullpath 
allow to use full path in HTTPCACHE_DBM_MODULE setting
 2013-02-01 09:01:27 -08:00
Rolando Espinoza La fuente
d00e6cf1d7 tests: dummydbm.error should be KeyError. 2013-02-01 12:52:59 -04:00
Rolando Espinoza La fuente
6ed44a3a67 httpcache: added tests for custom dbm module. 
And removed the break in the line to improve readability as there
is no strict 80-chars line width convention.
 2013-02-01 12:41:52 -04:00
Rolando Espinoza La fuente
a2602df90b httpcache: allow to import submodules within packages as HTTPCACHE_DBM_MODULE setting. 2013-01-31 23:49:36 -04:00
Daniel Graña
e5edb8ec3e Merge branch 'dangra/issue-24' 2013-01-30 16:55:53 -02:00
Daniel Graña
3af240f5f9 Merge branch 'dangra/issue-12' 2013-01-30 16:55:16 -02:00
Daniel Graña
0d3e4b4c43 do not unquote slash and question mark in url paths. fix #24 2013-01-30 11:22:03 -02:00
Daniel Graña
872a22df68 pep8ize sgml link extractors 2013-01-29 18:08:32 -02:00
Daniel Graña
cc69b3aa4c Fix #199 encoding error concatenating link text 2013-01-29 17:56:34 -02:00
Daniel Graña
95fde0a498 register namespaces when XMLFeedSpider uses iternodes mode. fixes #12 2013-01-29 12:00:13 -02:00
Daniel Graña
8e77f27897 Find form nodes in invalid html5 documents 
lxml fails to parse invalid html5 documents
This error was reported in scrapy/loginform#3
 2013-01-24 17:47:53 -02:00
Daniel Graña
ff04480675 fix exporting nested items as xml. fixes #66 2013-01-24 14:57:53 -02:00
Pablo Hoffman
71d7df9273 simplify InitSpider implementation, fixing one bug (closes #228) and adding support generators return from start_requests() (which the previous version didn't) 2013-01-23 16:28:41 -02:00
Daniel Graña
3cf7f4975b Add 0.16.4 to release notes 
Conflicts:
	docs/news.rst
 2013-01-23 11:29:38 -02:00
Daniel Graña
c40f947dc1 Merge pull request #229 from christilden/master 
fixes spelling errors in documentation
 2013-01-22 20:27:40 -08:00
Chris Tilden
aae6aed4fb fixes spelling errors in documentation 2013-01-22 14:52:18 -08:00
Pablo Hoffman
27583922a7 remove unused imports 2013-01-21 14:01:12 -02:00
Pablo Hoffman
65258e3621 Merge pull request #227 from tonal/correct-old-cache 
Correct init bag for load FilesystemCacheStorage from old location
 2013-01-21 04:57:19 -08:00
Alexandr N Zamaraev (aka tonal)
2c51266a40 Correct init bag for load old scrapy.contrib.httpcache.FilesystemCacheStorage 2013-01-21 13:54:36 +07:00
Pablo Hoffman
6ab8afb992 improve documentation about removing namespaces 2013-01-18 12:35:30 -02:00
Pablo Hoffman
1ba04b1fc3 added remove_namespaces() method to XmlXPathSelector objects 2013-01-18 12:20:03 -02:00
Pablo Hoffman
b7eeeff410 get rid of assertDictEqual (since it's python 2.7+ only) 2013-01-17 13:18:23 -02:00
Pablo Hoffman
c31441a273 revert default HTTP cache policy to dummy (instead of RFC2616) 2013-01-17 13:08:29 -02:00
Daniel Graña
897195186a document new FormRequest parameter named formxpath that matches forms using xpath 2013-01-08 18:36:20 -02:00
Daniel Graña
7527ef97ba Merge pull request #185 from notsobad/master 
Added xpath support in FormRequest.from_response
 2013-01-08 12:34:06 -08:00
Daniel Graña
75563b3f00 Add list of supported and missing RFC2616 caching features 2013-01-08 18:16:44 -02:00
Daniel Graña
3cbc4d0b94 django is an optional_features, its imports must not fail 2013-01-08 17:56:46 -02:00
Daniel Graña
d8a760bf57 Merge branch 'http-cache-middleware' 
Conflicts:
	scrapy/contrib/downloadermiddleware/httpcache.py
	scrapy/contrib/httpcache.py
	scrapy/tests/test_downloadermiddleware_httpcache.py
 2013-01-08 17:34:48 -02:00
Daniel Graña
864a7aef87 More httpcache updates 
* Change default cache policy to RFC2616
* Update HttpCacheMiddleware documentation
* Move policies to scrapy.contrib.httpcache
* remove a lint error for .has_key() usage in DBM storage backend
 2013-01-08 17:26:32 -02:00
Daniel Graña
487299e068 TakeFirst doc says it returns first non-null/non-empty value, zero is a valid value. closes #59 2013-01-08 15:47:33 -02:00
Daniel Graña
672d09ea2e add meta-refresh changes to release notes 2013-01-08 12:30:36 -02:00
Daniel Graña
9527c5819a pep8ize settings 2013-01-08 11:48:36 -02:00
Daniel Graña
defc4f89b5 update metarefresh settings 2013-01-08 11:41:19 -02:00
Daniel Graña
6a2b23883a Add MetaRefreshMiddleware docs 2013-01-08 11:25:38 -02:00
Daniel Graña
076ba40404 update DOWNLOADER_MIDDLEWARES_BASE setting documentation 2013-01-08 10:50:27 -02:00
Daniel Graña
71db7f1b25 Split redirection into status and metarefresh middlewares, also changes httpcompression priority. closes #78 2013-01-08 09:59:38 -02:00
Rolando Espinoza La fuente
fe5d0ce2e0 tests: added downloader middleware manager integration tests for gzipped redirection. 2013-01-08 09:59:38 -02:00
Pablo Hoffman
227a1d666b add doc about disabling an extension. refs #132 2013-01-07 13:16:19 -02:00
Pedro Faustino
5d3a4d755f Update downloader middleware documentation 2013-01-06 18:53:14 +00:00
Pedro Faustino
59dc71f394 Merge branch 'http-cache-middleware', remote-tracking branch 'dangra/http-cache-middleware' into http-cache-middleware 2013-01-06 17:53:25 +00:00
Pablo Hoffman
7f990a4af2 Merge pull request #221 from emschorsch/patch-1 
Proposed Changes to DjangoItem documentation
 2013-01-04 14:17:54 -08:00