BackTrackZ/scrapy
BackTrackZ/scrapy
1
0
Fork 0
mirror of https://github.com/scrapy/scrapy.git synced 2025-02-26 20:44:04 +00:00
Code Releases Wiki Activity
929 Commits 39 Branches 119 Tags
Commit Graph

929 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Pablo Hoffman
32d842326a reverted r675 which broke precedence for environment variables 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40679
 2009-01-07 18:06:39 +00:00
Pablo Hoffman
393cf349a0 updated settings doc 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40678
 2009-01-07 18:04:40 +00:00
elpolilla
382621b814 Fixed bug in RegexLinkExtractor. Encoding was not being specified 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40677
 2009-01-07 17:39:09 +00:00
elpolilla
307f8b3321 . Modified loading of command-specific settings (were loaded as defaults, now as overrides) 
. Removed duplicated loading of extensions in shell command

--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40676
 2009-01-07 17:31:06 +00:00
elpolilla
8029275c75 Removed non-scrapy import in shell command 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40675
 2009-01-07 14:18:03 +00:00
Pablo Hoffman
9884c209c5 minor update to overview doc 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40674
 2009-01-07 14:02:28 +00:00
elpolilla
e4852bca78 Modified extract adaptor to make use of "adaptor_args" (as it should), and added test for AdaptorPipes 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40673
 2009-01-07 14:00:54 +00:00
Pablo Hoffman
41fbcbde48 minor corrections to overview doc 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40672
 2009-01-07 12:56:40 +00:00
elpolilla
cd148c946e Refactored extract adaptor 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40671
 2009-01-07 12:48:15 +00:00
elpolilla
66852b8291 Small bugfix in selectors constructor regarding strings and unicodes 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40670
 2009-01-07 12:09:47 +00:00
Pablo Hoffman
2c2dc61766 somes fixes and updates to scrapy documentation 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40669
 2009-01-07 03:59:39 +00:00
Pablo Hoffman
0319373c61 improved overview doc. closes #44 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40668
 2009-01-07 03:58:15 +00:00
elpolilla
a155ad26a9 Added documentation for Adaptors 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40667
 2009-01-07 03:18:18 +00:00
Pablo Hoffman
d1594df0cb made url_is_from_spider work for tuples in extra_domain_names 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40666
 2009-01-07 01:02:38 +00:00
Pablo Hoffman
a2d19760f3 doc: minor grammar correction 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40665
 2009-01-06 23:07:29 +00:00
Pablo Hoffman
abdf4dee50 added doc about scrapy architecture. closes #31 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40664
 2009-01-06 23:05:04 +00:00
olveyra
4286e5e47c fix to revision 660 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40663
 2009-01-06 19:43:58 +00:00
Pablo Hoffman
1d36c3f0dd updated robotstxt, spidermw and downloadermw docs 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40662
 2009-01-06 17:34:14 +00:00
Pablo Hoffman
e9d050b91d renamed spider middleware methods to more consistent ones: process_spider_input, process_spider_output, process_spider_exception 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40661
 2009-01-06 17:30:29 +00:00
Pablo Hoffman
cf34fd4c98 added spider middleware documentation. closes #28 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40660
 2009-01-06 16:53:49 +00:00
Pablo Hoffman
7f7196c150 removed unused import 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40659
 2009-01-06 16:44:06 +00:00
Pablo Hoffman
5e63690939 added DEFAULT_SPIDER to default_settings and fixed formatting in DEFAULT_SPIDER setting documentation 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40658
 2009-01-06 16:42:59 +00:00
elpolilla
c80fef7eb0 Added the example project from the tutorial 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40657
 2009-01-06 16:08:03 +00:00
Pablo Hoffman
9d6defa643 doc: fixed intro-install xref 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40656
 2009-01-06 14:34:26 +00:00
Pablo Hoffman
9da9041248 updated INSTALL file to point to docs/intro/install. also, utf-8 default encoding is no longer required 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40655
 2009-01-06 00:19:51 +00:00
Pablo Hoffman
3badafc8a7 updated installation doc 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40654
 2009-01-06 00:15:55 +00:00
Pablo Hoffman
c9d91106d8 updated documentation index 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40653
 2009-01-06 00:15:30 +00:00
Pablo Hoffman
4df3118c03 added note to scrapy version 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40652
 2009-01-05 23:49:06 +00:00
Pablo Hoffman
b3c7203ad9 updated some doc configuration - version, release and html_copy_source 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40651
 2009-01-05 23:29:25 +00:00
olveyra
9b857124c1 added DEFAULT_SPIDER setting help 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40650
 2009-01-05 18:11:48 +00:00
elpolilla
111ba5d583 Updated items documentation 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40649
 2009-01-05 16:55:20 +00:00
elpolilla
f1962c1c0a . Moved init and repr methods from RobustScrapedItem to ScrapedItem 
. Refactored ScrapedItem's attribute method, added docstring and tests

--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40648
 2009-01-05 16:54:54 +00:00
Pablo Hoffman
3c4012f924 updated scrapy version 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40647
 2009-01-05 03:37:06 +00:00
Pablo Hoffman
5c9c82d055 some improvements to selectors doc structure, added literalincludes for sample1.html (to avoid duplicating the content), renamed that file and moved to _static (so it appears on built doc), moved comments out of source code snippets and into documentation text, and splitted them. converted to required '>>>' console format, and added proper highlighting hints 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40646
 2009-01-05 02:49:23 +00:00
Pablo Hoffman
7ca989a781 disabled docs app (redirecting to doc.scrapy.org instead) 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40645
 2009-01-04 23:28:50 +00:00
Pablo Hoffman
e61b581890 fixed wrong xref name 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40644
 2009-01-04 19:03:15 +00:00
samus_
2ea631398e reconfigured scrapy Response in order to disallow direct use of ResponseBody class that should be private 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40643
 2009-01-04 18:23:12 +00:00
elpolilla
58e4cf74b6 Added Items documentation 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40642
 2009-01-04 15:37:43 +00:00
elpolilla
dc1a168d53 Fixed unquote adaptor's test 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40641
 2009-01-04 14:46:13 +00:00
elpolilla
604c86b779 Removed unappropiate unquote adaptor's default behaviour 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40640
 2009-01-04 14:25:15 +00:00
elpolilla
e4bfb27f83 Removed already-deprecated response decompression tool 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40639
 2009-01-04 10:50:14 +00:00
Pablo Hoffman
8d255f7bac updated some docs paths 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40638
 2009-01-04 04:54:06 +00:00
elpolilla
ea439e4b03 Updated Selectors documentation 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40637
 2009-01-04 01:15:08 +00:00
elpolilla
a5025899ab Updated Spiders documentation 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40636
 2009-01-04 00:36:16 +00:00
Pablo Hoffman
f8f0db8bd3 doc: several more improvements 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40635
 2009-01-03 09:14:52 +00:00
Pablo Hoffman
137429e518 doc: added README 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40634
 2009-01-03 07:54:33 +00:00
Pablo Hoffman
16f9f5a9ef doc: added topic about robots.txt, added ROBOTSTXT_OBEY setting, added missing REQUESTS_PER_DOMAIN setting 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40633
 2009-01-03 07:41:43 +00:00
Pablo Hoffman
b59acb18ff minor typo and grammar corrections to docs/faq 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40632
 2009-01-03 07:40:40 +00:00
Pablo Hoffman
4a3ba6957b updated default_settings with robots.txt related settings 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40631
 2009-01-03 07:36:35 +00:00
Pablo Hoffman
e437ff67ae finished working version of robots.txt downloader middleware, and renamed the module/class 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40630
 2009-01-03 07:35:30 +00:00