BackTrackZ/scrapy
BackTrackZ/scrapy
1
0
Fork 0
mirror of https://github.com/scrapy/scrapy.git synced 2025-02-26 11:23:57 +00:00
Code Releases Wiki Activity
1,077 Commits 39 Branches 119 Tags
Commit Graph

1077 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Daniel Grana
391dde86ee cookies: rename cookies dict as jars 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401077
 2009-04-21 13:31:58 +00:00
Daniel Grana
48b436e81b cookies: add unittest and bugfix 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401076
 2009-04-21 13:31:26 +00:00
Daniel Grana
799301b865 http: bugfix appendlist method not setting headers if first time 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401075
 2009-04-21 13:30:55 +00:00
Pablo Hoffman
3170dd9fdb redirect mw: remove body and content-type/content-length headers on 302 redirects, and added tests. also renamed some tests to more meaningful names 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401074
 2009-04-21 01:38:11 +00:00
Daniel Grana
5861b345c7 cookies: add cookiejar to wrap cookielib and remove cookielib from mw 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401073
 2009-04-20 23:59:24 +00:00
Daniel Grana
0f22dfecb8 cluster: bugfix in remove and schedule actions 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401072
 2009-04-20 12:22:05 +00:00
Pablo Hoffman
f6e83be60d docs/request-response.rst: added sphinx :param: tags to improve documentation structure and readability. also documented default behaviour for FormRequest changes commited in r1069 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401071
 2009-04-20 02:31:22 +00:00
Pablo Hoffman
558a1a5033 FormRequest.from_response: fixed bug with setting method, override fields with those included in formdata, for those which already existed in the response <form> 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401070
 2009-04-20 02:29:19 +00:00
Daniel Grana
be5106d730 core: strip not working close delay support from core to (working) extension 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401069
 2009-04-17 20:43:50 +00:00
Daniel Grana
4d16571988 core: try to process requests after getting a DontCloseDomain to avoid delay on processing reinjected requests 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401068
 2009-04-17 19:13:38 +00:00
Pablo Hoffman
5f5962e6a4 improved SCHEDULER_ORDER setting doc 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401067
 2009-04-17 13:06:31 +00:00
Daniel Grana
512b75942c cookies: add TODO comments to experimental cookie mw 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401066
 2009-04-16 19:14:32 +00:00
Daniel Grana
60b330257a cookies: add new cookie middleware to experimental 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401065
 2009-04-16 19:05:27 +00:00
Daniel Grana
d6c52d51ed cluster: promote new code as replacement of old pbcluster 
--HG--
rename : scrapy/trunk/scrapy/contrib/pbcluster/crawler/__init__.py => scrapy/trunk/scrapy/contrib/cluster/crawler/__init__.py
rename : scrapy/trunk/scrapy/contrib_exp/cluster/crawler/manager.py => scrapy/trunk/scrapy/contrib/cluster/crawler/manager.py
rename : scrapy/trunk/scrapy/contrib_exp/cluster/hooks/__init__.py => scrapy/trunk/scrapy/contrib/cluster/hooks/__init__.py
rename : scrapy/trunk/scrapy/contrib_exp/cluster/hooks/svn.py => scrapy/trunk/scrapy/contrib/cluster/hooks/svn.py
rename : scrapy/trunk/scrapy/contrib/pbcluster/master/__init__.py => scrapy/trunk/scrapy/contrib/cluster/master/__init__.py
rename : scrapy/trunk/scrapy/contrib_exp/cluster/master/manager.py => scrapy/trunk/scrapy/contrib/cluster/master/manager.py
rename : scrapy/trunk/scrapy/contrib_exp/cluster/master/web.py => scrapy/trunk/scrapy/contrib/cluster/master/web.py
rename : scrapy/trunk/scrapy/contrib/pbcluster/master/ws_api.txt => scrapy/trunk/scrapy/contrib/cluster/master/ws_api.txt
rename : scrapy/trunk/scrapy/contrib/pbcluster/tools/scrapy-cluster-ctl.py => scrapy/trunk/scrapy/contrib/cluster/tools/scrapy-cluster-ctl.py
rename : scrapy/trunk/scrapy/contrib_exp/cluster/tools/test-worker.py => scrapy/trunk/scrapy/contrib/cluster/tools/test-worker.py
rename : scrapy/trunk/scrapy/contrib/pbcluster/worker/__init__.py => scrapy/trunk/scrapy/contrib/cluster/worker/__init__.py
rename : scrapy/trunk/scrapy/contrib_exp/cluster/worker/manager.py => scrapy/trunk/scrapy/contrib/cluster/worker/manager.py
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401064
 2009-04-16 18:58:56 +00:00
Daniel Grana
821f6be3ce http: fix xmlrpc request cloning 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401063
 2009-04-16 18:23:12 +00:00
Pablo Hoffman
bbd41bda33 more cleanup of default project settings templates and added a notice as suggested by Mark Ellul 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401062
 2009-04-16 15:32:06 +00:00
Pablo Hoffman
3fce2496c7 modified page <title> 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401061
 2009-04-15 01:31:04 +00:00
Daniel Grana
d18b78b8fe site: download section is linking to very old scrapy zip tarball 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401060
 2009-04-14 11:20:52 +00:00
Daniel Grana
fae3d7a8dc core: adapt redirection and media scheduler priorities 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401059
 2009-04-13 20:27:35 +00:00
Daniel Grana
b96a0c2756 core: change default priority to 0 to use balanced priorityqueue, and increase priority of redirected requests so memory doesnt hog because of redirection requests waiting for others requests to finish before they got a chance to be downloaded 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401058
 2009-04-13 16:42:26 +00:00
Pablo Hoffman
29b3746def updated some homepage texts 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401057
 2009-04-12 10:21:00 +00:00
Pablo Hoffman
743936062d removed unused code from Makefile 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401056
 2009-04-12 10:20:46 +00:00
Pablo Hoffman
74f39480c6 added a couple of unittests for from_response errors 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401055
 2009-04-12 09:16:31 +00:00
Pablo Hoffman
54ad49f765 doc: fixed a couple of broken links 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401054
 2009-04-12 09:05:00 +00:00
Pablo Hoffman
50efaab447 - added from_response() class method to FormRequest to support pre-populating 
HTML forms with fields taken from <form> elements contained in responses.
  implemented using the ClientForm library

- added ClientForm to Scrapy bundled libraries (scrapy.xlib)

- added unittests for new from_response() method

- documented new from_response() method, added a user login example to
  illustrate it, and a new faq entry

- improved overall quality of request/response doc

--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401053
 2009-04-12 08:31:55 +00:00
Pablo Hoffman
5378a35197 some more tuning to installation guide 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401052
 2009-04-11 18:34:44 +00:00
Pablo Hoffman
b5bbf827b7 added tips about using Firefox addons to inspect the live browser DOM 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401051
 2009-04-11 06:44:09 +00:00
Pablo Hoffman
9c005f8e75 skip image pipeline tests if Image module is not available 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401050
 2009-04-11 05:05:58 +00:00
Pablo Hoffman
3945c36f30 added scrapy.tests.run module to support running tests with 'python -m scrapy.tests.run' in Python 2.6, since Python 2.6 doesn't allow running packages with -m 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401049
 2009-04-11 05:01:37 +00:00
Pablo Hoffman
7e854af69b skip serialization tests if simplejson module is not available 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401048
 2009-04-11 04:56:39 +00:00
Pablo Hoffman
2799ffacff doc: updated installation guide with setup.py install mechanism 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401047
 2009-04-10 11:01:56 +00:00
Pablo Hoffman
eb0826e813 removed old .attribute() api from project spider templates until adaptors are stable 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401046
 2009-04-10 10:13:17 +00:00
Pablo Hoffman
4a2bca83ce removed templates from googledir example (to keep it DRY) as they made no sense there 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401045
 2009-04-10 10:09:51 +00:00
Pablo Hoffman
d503735bf0 removed old cluster from extensions loaded by default 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401044
 2009-04-10 09:59:17 +00:00
Pablo Hoffman
2bc0cdbaab added warning about twisted bug 3707 and installing pywin32 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401043
 2009-04-10 09:51:50 +00:00
Pablo Hoffman
8e7d5f69ec - fixed setup.py script (closes #80) 
- added .tmpl extension to project template files to prevent distutils from crashing when trying to compile those files
- cleaned up some garbage from settings templates

--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401042
 2009-04-10 08:06:44 +00:00
Pablo Hoffman
324ec07678 moved examples/contrib_exp to examples/experimental 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401041
 2009-04-10 08:03:26 +00:00
Pablo Hoffman
317221d21c removed old .attribute() API from doc - it will be restored in the future when adaptors code get stable 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401040
 2009-04-10 08:01:25 +00:00
Pablo Hoffman
46d9e681a9 minor updates to a couple of settings 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401039
 2009-04-10 07:33:46 +00:00
Pablo Hoffman
7c66f1739d Several documentation changes: 
- merged (and updated) new tutorial from proposed doc
- striped old tutorial and created new firebug topic
- added topic about useful third firefox add-ons
- rearranged main documentation index
- several assorted documentation fixes

--HG--
rename : scrapy/trunk/docs/proposed/tutorial.rst => scrapy/trunk/docs/intro/tutorial.rst
rename : scrapy/trunk/docs/intro/tutorial/scrot1.png => scrapy/trunk/docs/topics/_images/firebug1.png
rename : scrapy/trunk/docs/intro/tutorial/scrot2.png => scrapy/trunk/docs/topics/_images/firebug2.png
rename : scrapy/trunk/docs/intro/tutorial/scrot3.png => scrapy/trunk/docs/topics/_images/firebug3.png
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401038
 2009-04-10 05:35:53 +00:00
Pablo Hoffman
0fb3ee16d8 more improvements to scrapy shell: added Request object, and support for modifying it and re-fetching it by issuing an empty 'get' command 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401037
 2009-04-03 04:13:21 +00:00
Pablo Hoffman
ef9f4961a1 scrapy.xpath: added docstring pointing to the doc 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401036
 2009-04-03 04:05:57 +00:00
Pablo Hoffman
b7bd00b336 minor changes to some module descriptions 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401035
 2009-04-03 03:20:45 +00:00
Pablo Hoffman
7e7823654c some code refactoring for the scrapy shell command 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401034
 2009-04-03 03:13:22 +00:00
Pablo Hoffman
b521ca4d36 massive improvements to xpath selectors doc. refs #25 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401033
 2009-04-03 01:33:52 +00:00
Pablo Hoffman
d732a79ead added documentation for scrapy shell. closes #78 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401032
 2009-04-03 01:19:36 +00:00
Pablo Hoffman
931c29eabc updated Scrapy architecture doc by adding reference to Scheduler middlewares. refs #31 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401031
 2009-04-03 00:43:09 +00:00
Pablo Hoffman
8757359775 doc: added note about logging from spiders 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401030
 2009-04-02 22:43:52 +00:00
Daniel Grana
1683c4e971 tests: comment failed test until fix 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401029
 2009-04-01 14:29:56 +00:00
artem
6ac767f84b tests: added url_query_parameter() fail case 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401028
 2009-04-01 13:40:30 +00:00