BackTrackZ/scrapy
BackTrackZ/scrapy
1
0
Fork 0
mirror of https://github.com/scrapy/scrapy.git synced 2025-02-26 00:03:57 +00:00
Code Releases Wiki Activity
1,091 Commits 39 Branches 119 Tags
Commit Graph

1091 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Pablo Hoffman
b553bb3dff removed TODO comment 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401091
 2009-04-27 12:20:24 +00:00
Daniel Grana
8eaa9219f6 test: fix webclient test 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401090
 2009-04-27 06:52:38 +00:00
Daniel Grana
e8ecc00cfb core: clean HTTP handling and add unittests 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401089
 2009-04-27 06:48:24 +00:00
Daniel Grana
0842fafd63 useragent: add unittests to mw and prepare to remove UA from downloader handlers 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401088
 2009-04-27 06:47:26 +00:00
Daniel Grana
a86a89f41a cookies: mv cookies module under http module 
--HG--
rename : scrapy/trunk/scrapy/utils/cookies.py => scrapy/trunk/scrapy/http/cookies.py
rename : scrapy/trunk/scrapy/tests/test_utils_cookies.py => scrapy/trunk/scrapy/tests/test_http_cookies.py
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401087
 2009-04-27 06:46:46 +00:00
Daniel Grana
68f54a1a63 cookies: add test to check that wrapped cookiejar thread based lock is overrided 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401086
 2009-04-27 06:45:49 +00:00
Daniel Grana
45f55c0df0 cookies: check mw outputs. refs #73 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401085
 2009-04-23 13:27:04 +00:00
Daniel Grana
315a24add4 cookies: fix typo in cookiejar lock attribute name. refs #73 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401084
 2009-04-22 23:05:15 +00:00
Pablo Hoffman
bec0b12270 changed settings doc according to reflect recent cookies refactoring 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401083
 2009-04-22 20:03:56 +00:00
Daniel Grana
1b2c9e49d8 cookies: fix test import. refs #73 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401082
 2009-04-22 17:52:43 +00:00
Daniel Grana
1ecf874a75 cookies: final new cookie middleware integration into core and enabled by default. refs #73 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401081
 2009-04-22 17:21:46 +00:00
Daniel Grana
3de3c2a331 cookies: fix merging user cookies 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401080
 2009-04-21 17:13:12 +00:00
Daniel Grana
4a6f7af739 cookies: now really add unittests 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401079
 2009-04-21 13:33:42 +00:00
Daniel Grana
1ce1481eab cookies: override cookielib cookiejar thread based lock by dummy lock 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401078
 2009-04-21 13:32:19 +00:00
Daniel Grana
391dde86ee cookies: rename cookies dict as jars 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401077
 2009-04-21 13:31:58 +00:00
Daniel Grana
48b436e81b cookies: add unittest and bugfix 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401076
 2009-04-21 13:31:26 +00:00
Daniel Grana
799301b865 http: bugfix appendlist method not setting headers if first time 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401075
 2009-04-21 13:30:55 +00:00
Pablo Hoffman
3170dd9fdb redirect mw: remove body and content-type/content-length headers on 302 redirects, and added tests. also renamed some tests to more meaningful names 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401074
 2009-04-21 01:38:11 +00:00
Daniel Grana
5861b345c7 cookies: add cookiejar to wrap cookielib and remove cookielib from mw 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401073
 2009-04-20 23:59:24 +00:00
Daniel Grana
0f22dfecb8 cluster: bugfix in remove and schedule actions 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401072
 2009-04-20 12:22:05 +00:00
Pablo Hoffman
f6e83be60d docs/request-response.rst: added sphinx :param: tags to improve documentation structure and readability. also documented default behaviour for FormRequest changes commited in r1069 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401071
 2009-04-20 02:31:22 +00:00
Pablo Hoffman
558a1a5033 FormRequest.from_response: fixed bug with setting method, override fields with those included in formdata, for those which already existed in the response <form> 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401070
 2009-04-20 02:29:19 +00:00
Daniel Grana
be5106d730 core: strip not working close delay support from core to (working) extension 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401069
 2009-04-17 20:43:50 +00:00
Daniel Grana
4d16571988 core: try to process requests after getting a DontCloseDomain to avoid delay on processing reinjected requests 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401068
 2009-04-17 19:13:38 +00:00
Pablo Hoffman
5f5962e6a4 improved SCHEDULER_ORDER setting doc 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401067
 2009-04-17 13:06:31 +00:00
Daniel Grana
512b75942c cookies: add TODO comments to experimental cookie mw 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401066
 2009-04-16 19:14:32 +00:00
Daniel Grana
60b330257a cookies: add new cookie middleware to experimental 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401065
 2009-04-16 19:05:27 +00:00
Daniel Grana
d6c52d51ed cluster: promote new code as replacement of old pbcluster 
--HG--
rename : scrapy/trunk/scrapy/contrib/pbcluster/crawler/__init__.py => scrapy/trunk/scrapy/contrib/cluster/crawler/__init__.py
rename : scrapy/trunk/scrapy/contrib_exp/cluster/crawler/manager.py => scrapy/trunk/scrapy/contrib/cluster/crawler/manager.py
rename : scrapy/trunk/scrapy/contrib_exp/cluster/hooks/__init__.py => scrapy/trunk/scrapy/contrib/cluster/hooks/__init__.py
rename : scrapy/trunk/scrapy/contrib_exp/cluster/hooks/svn.py => scrapy/trunk/scrapy/contrib/cluster/hooks/svn.py
rename : scrapy/trunk/scrapy/contrib/pbcluster/master/__init__.py => scrapy/trunk/scrapy/contrib/cluster/master/__init__.py
rename : scrapy/trunk/scrapy/contrib_exp/cluster/master/manager.py => scrapy/trunk/scrapy/contrib/cluster/master/manager.py
rename : scrapy/trunk/scrapy/contrib_exp/cluster/master/web.py => scrapy/trunk/scrapy/contrib/cluster/master/web.py
rename : scrapy/trunk/scrapy/contrib/pbcluster/master/ws_api.txt => scrapy/trunk/scrapy/contrib/cluster/master/ws_api.txt
rename : scrapy/trunk/scrapy/contrib/pbcluster/tools/scrapy-cluster-ctl.py => scrapy/trunk/scrapy/contrib/cluster/tools/scrapy-cluster-ctl.py
rename : scrapy/trunk/scrapy/contrib_exp/cluster/tools/test-worker.py => scrapy/trunk/scrapy/contrib/cluster/tools/test-worker.py
rename : scrapy/trunk/scrapy/contrib/pbcluster/worker/__init__.py => scrapy/trunk/scrapy/contrib/cluster/worker/__init__.py
rename : scrapy/trunk/scrapy/contrib_exp/cluster/worker/manager.py => scrapy/trunk/scrapy/contrib/cluster/worker/manager.py
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401064
 2009-04-16 18:58:56 +00:00
Daniel Grana
821f6be3ce http: fix xmlrpc request cloning 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401063
 2009-04-16 18:23:12 +00:00
Pablo Hoffman
bbd41bda33 more cleanup of default project settings templates and added a notice as suggested by Mark Ellul 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401062
 2009-04-16 15:32:06 +00:00
Pablo Hoffman
3fce2496c7 modified page <title> 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401061
 2009-04-15 01:31:04 +00:00
Daniel Grana
d18b78b8fe site: download section is linking to very old scrapy zip tarball 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401060
 2009-04-14 11:20:52 +00:00
Daniel Grana
fae3d7a8dc core: adapt redirection and media scheduler priorities 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401059
 2009-04-13 20:27:35 +00:00
Daniel Grana
b96a0c2756 core: change default priority to 0 to use balanced priorityqueue, and increase priority of redirected requests so memory doesnt hog because of redirection requests waiting for others requests to finish before they got a chance to be downloaded 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401058
 2009-04-13 16:42:26 +00:00
Pablo Hoffman
29b3746def updated some homepage texts 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401057
 2009-04-12 10:21:00 +00:00
Pablo Hoffman
743936062d removed unused code from Makefile 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401056
 2009-04-12 10:20:46 +00:00
Pablo Hoffman
74f39480c6 added a couple of unittests for from_response errors 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401055
 2009-04-12 09:16:31 +00:00
Pablo Hoffman
54ad49f765 doc: fixed a couple of broken links 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401054
 2009-04-12 09:05:00 +00:00
Pablo Hoffman
50efaab447 - added from_response() class method to FormRequest to support pre-populating 
HTML forms with fields taken from <form> elements contained in responses.
  implemented using the ClientForm library

- added ClientForm to Scrapy bundled libraries (scrapy.xlib)

- added unittests for new from_response() method

- documented new from_response() method, added a user login example to
  illustrate it, and a new faq entry

- improved overall quality of request/response doc

--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401053
 2009-04-12 08:31:55 +00:00
Pablo Hoffman
5378a35197 some more tuning to installation guide 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401052
 2009-04-11 18:34:44 +00:00
Pablo Hoffman
b5bbf827b7 added tips about using Firefox addons to inspect the live browser DOM 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401051
 2009-04-11 06:44:09 +00:00
Pablo Hoffman
9c005f8e75 skip image pipeline tests if Image module is not available 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401050
 2009-04-11 05:05:58 +00:00
Pablo Hoffman
3945c36f30 added scrapy.tests.run module to support running tests with 'python -m scrapy.tests.run' in Python 2.6, since Python 2.6 doesn't allow running packages with -m 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401049
 2009-04-11 05:01:37 +00:00
Pablo Hoffman
7e854af69b skip serialization tests if simplejson module is not available 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401048
 2009-04-11 04:56:39 +00:00
Pablo Hoffman
2799ffacff doc: updated installation guide with setup.py install mechanism 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401047
 2009-04-10 11:01:56 +00:00
Pablo Hoffman
eb0826e813 removed old .attribute() api from project spider templates until adaptors are stable 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401046
 2009-04-10 10:13:17 +00:00
Pablo Hoffman
4a2bca83ce removed templates from googledir example (to keep it DRY) as they made no sense there 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401045
 2009-04-10 10:09:51 +00:00
Pablo Hoffman
d503735bf0 removed old cluster from extensions loaded by default 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401044
 2009-04-10 09:59:17 +00:00
Pablo Hoffman
2bc0cdbaab added warning about twisted bug 3707 and installing pywin32 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401043
 2009-04-10 09:51:50 +00:00
Pablo Hoffman
8e7d5f69ec - fixed setup.py script (closes #80) 
- added .tmpl extension to project template files to prevent distutils from crashing when trying to compile those files
- cleaned up some garbage from settings templates

--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401042
 2009-04-10 08:06:44 +00:00