BackTrackZ/scrapy
BackTrackZ/scrapy
1
0
Fork 0
mirror of https://github.com/scrapy/scrapy.git synced 2025-02-26 01:23:56 +00:00
Code Releases Wiki Activity
1,061 Commits 39 Branches 119 Tags
Commit Graph

1061 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Pablo Hoffman
3fce2496c7 modified page <title> 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401061
 2009-04-15 01:31:04 +00:00
Daniel Grana
d18b78b8fe site: download section is linking to very old scrapy zip tarball 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401060
 2009-04-14 11:20:52 +00:00
Daniel Grana
fae3d7a8dc core: adapt redirection and media scheduler priorities 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401059
 2009-04-13 20:27:35 +00:00
Daniel Grana
b96a0c2756 core: change default priority to 0 to use balanced priorityqueue, and increase priority of redirected requests so memory doesnt hog because of redirection requests waiting for others requests to finish before they got a chance to be downloaded 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401058
 2009-04-13 16:42:26 +00:00
Pablo Hoffman
29b3746def updated some homepage texts 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401057
 2009-04-12 10:21:00 +00:00
Pablo Hoffman
743936062d removed unused code from Makefile 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401056
 2009-04-12 10:20:46 +00:00
Pablo Hoffman
74f39480c6 added a couple of unittests for from_response errors 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401055
 2009-04-12 09:16:31 +00:00
Pablo Hoffman
54ad49f765 doc: fixed a couple of broken links 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401054
 2009-04-12 09:05:00 +00:00
Pablo Hoffman
50efaab447 - added from_response() class method to FormRequest to support pre-populating 
HTML forms with fields taken from <form> elements contained in responses.
  implemented using the ClientForm library

- added ClientForm to Scrapy bundled libraries (scrapy.xlib)

- added unittests for new from_response() method

- documented new from_response() method, added a user login example to
  illustrate it, and a new faq entry

- improved overall quality of request/response doc

--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401053
 2009-04-12 08:31:55 +00:00
Pablo Hoffman
5378a35197 some more tuning to installation guide 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401052
 2009-04-11 18:34:44 +00:00
Pablo Hoffman
b5bbf827b7 added tips about using Firefox addons to inspect the live browser DOM 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401051
 2009-04-11 06:44:09 +00:00
Pablo Hoffman
9c005f8e75 skip image pipeline tests if Image module is not available 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401050
 2009-04-11 05:05:58 +00:00
Pablo Hoffman
3945c36f30 added scrapy.tests.run module to support running tests with 'python -m scrapy.tests.run' in Python 2.6, since Python 2.6 doesn't allow running packages with -m 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401049
 2009-04-11 05:01:37 +00:00
Pablo Hoffman
7e854af69b skip serialization tests if simplejson module is not available 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401048
 2009-04-11 04:56:39 +00:00
Pablo Hoffman
2799ffacff doc: updated installation guide with setup.py install mechanism 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401047
 2009-04-10 11:01:56 +00:00
Pablo Hoffman
eb0826e813 removed old .attribute() api from project spider templates until adaptors are stable 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401046
 2009-04-10 10:13:17 +00:00
Pablo Hoffman
4a2bca83ce removed templates from googledir example (to keep it DRY) as they made no sense there 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401045
 2009-04-10 10:09:51 +00:00
Pablo Hoffman
d503735bf0 removed old cluster from extensions loaded by default 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401044
 2009-04-10 09:59:17 +00:00
Pablo Hoffman
2bc0cdbaab added warning about twisted bug 3707 and installing pywin32 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401043
 2009-04-10 09:51:50 +00:00
Pablo Hoffman
8e7d5f69ec - fixed setup.py script (closes #80) 
- added .tmpl extension to project template files to prevent distutils from crashing when trying to compile those files
- cleaned up some garbage from settings templates

--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401042
 2009-04-10 08:06:44 +00:00
Pablo Hoffman
324ec07678 moved examples/contrib_exp to examples/experimental 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401041
 2009-04-10 08:03:26 +00:00
Pablo Hoffman
317221d21c removed old .attribute() API from doc - it will be restored in the future when adaptors code get stable 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401040
 2009-04-10 08:01:25 +00:00
Pablo Hoffman
46d9e681a9 minor updates to a couple of settings 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401039
 2009-04-10 07:33:46 +00:00
Pablo Hoffman
7c66f1739d Several documentation changes: 
- merged (and updated) new tutorial from proposed doc
- striped old tutorial and created new firebug topic
- added topic about useful third firefox add-ons
- rearranged main documentation index
- several assorted documentation fixes

--HG--
rename : scrapy/trunk/docs/proposed/tutorial.rst => scrapy/trunk/docs/intro/tutorial.rst
rename : scrapy/trunk/docs/intro/tutorial/scrot1.png => scrapy/trunk/docs/topics/_images/firebug1.png
rename : scrapy/trunk/docs/intro/tutorial/scrot2.png => scrapy/trunk/docs/topics/_images/firebug2.png
rename : scrapy/trunk/docs/intro/tutorial/scrot3.png => scrapy/trunk/docs/topics/_images/firebug3.png
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401038
 2009-04-10 05:35:53 +00:00
Pablo Hoffman
0fb3ee16d8 more improvements to scrapy shell: added Request object, and support for modifying it and re-fetching it by issuing an empty 'get' command 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401037
 2009-04-03 04:13:21 +00:00
Pablo Hoffman
ef9f4961a1 scrapy.xpath: added docstring pointing to the doc 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401036
 2009-04-03 04:05:57 +00:00
Pablo Hoffman
b7bd00b336 minor changes to some module descriptions 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401035
 2009-04-03 03:20:45 +00:00
Pablo Hoffman
7e7823654c some code refactoring for the scrapy shell command 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401034
 2009-04-03 03:13:22 +00:00
Pablo Hoffman
b521ca4d36 massive improvements to xpath selectors doc. refs #25 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401033
 2009-04-03 01:33:52 +00:00
Pablo Hoffman
d732a79ead added documentation for scrapy shell. closes #78 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401032
 2009-04-03 01:19:36 +00:00
Pablo Hoffman
931c29eabc updated Scrapy architecture doc by adding reference to Scheduler middlewares. refs #31 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401031
 2009-04-03 00:43:09 +00:00
Pablo Hoffman
8757359775 doc: added note about logging from spiders 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401030
 2009-04-02 22:43:52 +00:00
Daniel Grana
1683c4e971 tests: comment failed test until fix 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401029
 2009-04-01 14:29:56 +00:00
artem
6ac767f84b tests: added url_query_parameter() fail case 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401028
 2009-04-01 13:40:30 +00:00
Daniel Grana
128de5a49b cluster: add missing import 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401027
 2009-03-31 04:00:58 +00:00
Daniel Grana
e15c928ed2 cluster: add log gzipping support 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401026
 2009-03-31 03:50:13 +00:00
Daniel Grana
c2852b13ea imagespipeline: restore original log message for uptodate image 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401025
 2009-03-30 10:41:55 +00:00
Daniel Grana
96ee661f2c pipeline: add missing future import for with statement at images pipeline 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401024
 2009-03-27 06:59:38 +00:00
Daniel Grana
186d0e93b0 Revert "added rule shorthand, for creating CrawlSpider rules" 
This reverts commit r863.

This is not the place for this shorthand, and it hasn't the blessing of
BDFL. I look forward for the return of this shorthand.

--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401023
 2009-03-27 06:46:06 +00:00
Daniel Grana
7227b65de1 http: fix copy and failing appendlist method of Headers, also add missing tests 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401022
 2009-03-27 06:42:24 +00:00
Daniel Grana
1d0e2d1202 linkextractors: add arg_to_iter support to RegexLinkExtractor 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401021
 2009-03-27 06:05:54 +00:00
Daniel Grana
4fa7e6cac4 tests: fix images tests 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401020
 2009-03-27 05:42:47 +00:00
Daniel Grana
5b7cfb3184 pipeline: refactor image pipelines moving common functionality to BaseImagesPipeline 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401019
 2009-03-27 04:41:11 +00:00
Daniel Grana
70a686c02b http: add errback to Request constructor 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401018
 2009-03-25 13:15:55 +00:00
Daniel Grana
828c3e0988 core: remove deferred_degenerator, instead use a cooperative map 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401017
 2009-03-25 13:15:24 +00:00
Daniel Grana
9d59bb3153 utils: remove unused function arg_to_list 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401016
 2009-03-25 13:14:50 +00:00
Daniel Grana
de0d57cd9b log: add an twisted.python.log.err handler wrapper just like log.msg one 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401015
 2009-03-25 13:14:26 +00:00
Pablo Hoffman
7c4c17088a added copy tests for Headers which need to pass but are currently failing 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401014
 2009-03-24 21:21:39 +00:00
Pablo Hoffman
d8c0440700 fixed bug in previous commit 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401013
 2009-03-24 21:03:38 +00:00
Pablo Hoffman
0c9631594d - made Request.copy() use Request.replace() 
- added callback, dont_filter, encoding to Request copy/replace
- fixed Request.replace() and Response.replace() which weren't working properly with empty arguments
- added new test cases
- added doc about copying requests and deferreds

--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401012
 2009-03-24 20:02:42 +00:00