BackTrackZ/scrapy
BackTrackZ/scrapy
1
0
Fork 0
mirror of https://github.com/scrapy/scrapy.git synced 2025-02-26 18:44:12 +00:00
Code Releases Wiki Activity
893 Commits 39 Branches 119 Tags
Commit Graph

893 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Daniel Grana
7c056c620e tutorial: fix outofdate and broken tutorial after adaptors were moved to experimental 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40843
 2009-02-12 03:58:54 +00:00
Daniel Grana
f43c58da1f adds a reference about empty request body and Content-Length header fix, closes #60. 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40842
 2009-02-10 18:27:59 +00:00
Daniel Grana
8fdf06ed08 Avoid sending Content-Length header when body is an empty string. 
Some sites can't handle "Content-Length: 0" header, but twisted
HTTPClientFactory adds Content-Length header unless body is None.

scrapy enforces request.body usage as string, using None is not
possible.

thanks Matt for report, See:
http://groups.google.com/group/scrapy-users/browse_thread/thread/380ffa111879989e?hl=en

--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40841
 2009-02-10 13:23:38 +00:00
Pablo Hoffman
f4224be411 - added get_meta_refresh to scrapy.utils.response 
- added tests for get_meta_refresh_url
- added RedirectMiddleware tests

--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40840
 2009-02-10 06:20:43 +00:00
Pablo Hoffman
d376b20996 minor fix to doctest 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40839
 2009-02-10 04:46:07 +00:00
Pablo Hoffman
90c1597cef fixed bug with items_to_csv and Excel. thanks Mat for the patch. See: 
http://groups.google.com/group/scrapy-developers/browse_thread/thread/9e2b80d226333011

--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40838
 2009-02-10 02:43:19 +00:00
Daniel Grana
1b2de80124 remove scrapy/dos external referenced from scrapy.org site 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40837
 2009-02-09 17:20:41 +00:00
Ismael Carnales
844b59b5b3 reduced introduction text in proposed doc 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40836
 2009-02-09 15:06:22 +00:00
Ismael Carnales
d98f35af94 added items section (ScrapedItems) to proposed doc 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40835
 2009-02-09 14:29:48 +00:00
Ismael Carnales
3a3723ba2e removed reference to guid in ScrapedItem 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40834
 2009-02-09 13:59:15 +00:00
Pablo Hoffman
38c62ca357 settings must be documented in alphabetical order 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40833
 2009-02-08 21:00:26 +00:00
Ismael Carnales
45340d3d87 better index for proposed documentation 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40832
 2009-02-06 20:19:17 +00:00
Ismael Carnales
5ca4728805 added complete spiders topic and reference (in one file) using an autodoc and manual doc mix 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40831
 2009-02-06 20:15:34 +00:00
Ismael Carnales
3fb945be7b fixed doc on BaseSpider 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40830
 2009-02-06 20:02:13 +00:00
Ismael Carnales
485b9c9ed5 updated docstrings of feedspiders for autodoc 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40829
 2009-02-06 20:01:27 +00:00
Ismael Carnales
9ba07653c6 added a proposed introduction (with all the main topics) to the proposed documentation, this would be the start for the new tutorial 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40828
 2009-02-06 16:20:49 +00:00
Ismael Carnales
be04a7f328 modify docs configuration for autodoc to work 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40827
 2009-02-05 15:39:52 +00:00
Ismael Carnales
96be669af2 fix tutorial1 errors 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40826
 2009-02-05 13:44:32 +00:00
Daniel Grana
fe2f018b1a FormRequest: urlencode multiples values of a single key using doseq 
this prevents urllib.urlencode from sending the repr of the value when
it founds a list or tuple.

--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40825
 2009-02-05 13:41:10 +00:00
Ismael Carnales
6650e56354 changed representation of the project tree in the tutorial 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40824
 2009-02-05 13:35:14 +00:00
Daniel Grana
8f349737f8 add missing dashes to docs 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40823
 2009-02-05 12:43:30 +00:00
Daniel Grana
bc2ea867ca Adds docs for PROJECT_NAME setting. refs #58 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40822
 2009-02-05 12:39:43 +00:00
Daniel Grana
bf87fb8455 add proper docstring to string_camelcase. refs #58 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40821
 2009-02-05 12:20:25 +00:00
Ismael Carnales
751a844e36 updated tutorial to reflect project's structure change 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40820
 2009-02-05 12:14:49 +00:00
Daniel Grana
f7dac0e449 Creates an usable project structure on startproject. closes #58 and closes #54 
--HG--
rename : scrapy/trunk/scrapy/conf/project_template/__init__.py => scrapy/trunk/scrapy/templates/project/module/__init__.py
rename : scrapy/trunk/scrapy/conf/project_template/items.py => scrapy/trunk/scrapy/templates/project/module/items.py
rename : scrapy/trunk/scrapy/conf/project_template/scrapy_settings.py => scrapy/trunk/scrapy/templates/project/module/settings.py
rename : scrapy/trunk/scrapy/conf/project_template/spiders/__init__.py => scrapy/trunk/scrapy/templates/project/module/spiders/__init__.py
rename : scrapy/trunk/scrapy/conf/project_template/templates/spider_basic.tmpl => scrapy/trunk/scrapy/templates/project/module/templates/spider_basic.tmpl
rename : scrapy/trunk/scrapy/conf/project_template/templates/spider_crawl.tmpl => scrapy/trunk/scrapy/templates/project/module/templates/spider_crawl.tmpl
rename : scrapy/trunk/scrapy/conf/project_template/templates/spider_csvfeed.tmpl => scrapy/trunk/scrapy/templates/project/module/templates/spider_csvfeed.tmpl
rename : scrapy/trunk/scrapy/conf/project_template/templates/spider_xmlfeed.tmpl => scrapy/trunk/scrapy/templates/project/module/templates/spider_xmlfeed.tmpl
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40819
 2009-02-05 12:12:15 +00:00
Ismael Carnales
bccef8a463 fixed basespider links in topics/spider 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40818
 2009-01-31 06:09:50 +00:00
Ismael Carnales
fa8ebd1147 corrected naming and references in ref/spiders doc 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40817
 2009-01-31 06:01:33 +00:00
Pablo Hoffman
6220609718 added bin/runtests.sh script to run tests 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40816
 2009-01-30 23:20:49 +00:00
Pablo Hoffman
20a61caa2c fixed some doc typos 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40815
 2009-01-30 23:20:21 +00:00
Pablo Hoffman
005a642240 added SETTINGS_DISABLED environment variable to turn off custom settings (and only use Scrapy defaults) 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40814
 2009-01-30 23:18:23 +00:00
Pablo Hoffman
16efbf87fc added some references to documentation and fixed some doc typos (thanks Patrick) 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40813
 2009-01-30 22:33:50 +00:00
Pablo Hoffman
c672223091 splitted spiders doc from link extractor docs, moved the corresponding parts to ref and topics 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40812
 2009-01-30 21:53:40 +00:00
Pablo Hoffman
483ef3ba7f changed typo in Items docs 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40811
 2009-01-30 21:52:41 +00:00
Ismael Carnales
a834eef6c9 added BaseSpider attributes and method references and metadata (from tutorial) 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40810
 2009-01-30 19:24:43 +00:00
Ismael Carnales
7e6c9c2e25 formatting changes and references to spiders added in the tutorial 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40809
 2009-01-30 19:14:16 +00:00
Ismael Carnales
2526e11be0 changed formatting a little bit and added class references to spiders topic 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40808
 2009-01-30 19:12:09 +00:00
Ismael Carnales
591a4a2bb5 added information on default item in new tutorial 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40807
 2009-01-30 14:45:52 +00:00
Ismael Carnales
f27d07c1a4 added warning of spider naming in new tutorial 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40806
 2009-01-30 14:26:40 +00:00
Ismael Carnales
3b012d3981 modified default project template structure in proposed tutorial 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40805
 2009-01-30 13:15:50 +00:00
elpolilla
648753190c Several corrections made to items and adaptors documentation 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40804
 2009-01-30 13:05:52 +00:00
Ismael Carnales
d9a90a1f3f added proposed documentation, moved tutorial there 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40803
 2009-01-30 11:29:55 +00:00
elpolilla
6cc806a843 Added adaptors diagram svg to the repo 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40802
 2009-01-30 11:19:20 +00:00
Pablo Hoffman
bcf6e5476b added one more case to responsetypes 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40801
 2009-01-30 11:14:08 +00:00
elpolilla
769519c2aa Added missing adaptors diagram to documentation 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40800
 2009-01-30 10:57:12 +00:00
Pablo Hoffman
fb13d3d83b doc: added status argument to domain_closed signal 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40799
 2009-01-29 20:18:53 +00:00
Daniel Grana
b3050ca0e4 pipeline: remove open_domain/close_domain hooks, use domain_open and domain_closed signals instead. docs updated. 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40798
 2009-01-29 19:18:03 +00:00
Pablo Hoffman
73c924643c bundled mime.types file with scrapy for platforms that have poor support for mime.types (windows), added from_args() convenience method to ResponseTypes 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40797
 2009-01-29 18:52:46 +00:00
Daniel Grana
f4e6ed3c2f run without RobotTxtMiddleware by default and update docs 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40796
 2009-01-29 18:22:59 +00:00
Ismael Carnales
e6375c6a5b added some info on items pipeline on the tutorial 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40795
 2009-01-29 17:57:21 +00:00
elpolilla
9634af9aa7 Updated adaptors and items documentation 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40794
 2009-01-29 17:46:32 +00:00