BackTrackZ/scrapy
BackTrackZ/scrapy
1
0
Fork 0
mirror of https://github.com/scrapy/scrapy.git synced 2025-02-26 17:24:38 +00:00
Code Releases Wiki Activity
993 Commits 39 Branches 119 Tags
Commit Graph

993 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Pablo Hoffman
4c54305bee added docstrings to unicode_to_str and str_to_unicode 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40693
 2009-01-09 10:35:14 +00:00
samus_
8e38f70a74 small improvement to Response.__init__ testcase 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40692
 2009-01-09 01:33:12 +00:00
samus_
d3ac5d5ab8 added test for Response.__init__ 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40691
 2009-01-09 00:45:12 +00:00
samus_
8240fc48ad refactored ResponseBody's encoding test 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40690
 2009-01-09 00:42:44 +00:00
samus_
1feace1bb0 a bit of performance 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40689
 2009-01-08 18:29:56 +00:00
samus_
d0e5245fba fixed bug 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40688
 2009-01-08 18:29:33 +00:00
samus_
c601bbb083 fixed bug in copy method of Response (tests coming soon) 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40687
 2009-01-08 16:08:04 +00:00
elpolilla
5cb9f344ac Changed method process_spider_output name to process_results in crawl and feed spiders 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40686
 2009-01-08 15:42:33 +00:00
Pablo Hoffman
6568dc4f35 added custom CSS for scrapy doc with minor modifications (colors only) 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40685
 2009-01-08 15:02:39 +00:00
Pablo Hoffman
6420c407d2 removed scrapyengine import from downoader code, minor improvements to docstrings 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40684
 2009-01-08 14:21:20 +00:00
elpolilla
f39b7b507a Disabled test until memory leak in libxml is fixed 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40683
 2009-01-08 13:46:14 +00:00
Pablo Hoffman
fc76af2eb4 reverted r680 until we there are tests and documentation available about the change 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40682
 2009-01-08 13:30:25 +00:00
elpolilla
4a9a48a629 . Added strip() to link texts in LinkExtractor in order to avoid extracting line breaks and such 
. Added test for restrict_xpaths in RegexLinkExtractor

--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40681
 2009-01-08 11:05:15 +00:00
elpolilla
8278c0cffc Changed process_results name in feed spiders to process_spider_output and moved its parameters to a more standard order 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40680
 2009-01-08 10:11:56 +00:00
Pablo Hoffman
32d842326a reverted r675 which broke precedence for environment variables 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40679
 2009-01-07 18:06:39 +00:00
Pablo Hoffman
393cf349a0 updated settings doc 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40678
 2009-01-07 18:04:40 +00:00
elpolilla
382621b814 Fixed bug in RegexLinkExtractor. Encoding was not being specified 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40677
 2009-01-07 17:39:09 +00:00
elpolilla
307f8b3321 . Modified loading of command-specific settings (were loaded as defaults, now as overrides) 
. Removed duplicated loading of extensions in shell command

--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40676
 2009-01-07 17:31:06 +00:00
elpolilla
8029275c75 Removed non-scrapy import in shell command 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40675
 2009-01-07 14:18:03 +00:00
Pablo Hoffman
9884c209c5 minor update to overview doc 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40674
 2009-01-07 14:02:28 +00:00
elpolilla
e4852bca78 Modified extract adaptor to make use of "adaptor_args" (as it should), and added test for AdaptorPipes 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40673
 2009-01-07 14:00:54 +00:00
Pablo Hoffman
41fbcbde48 minor corrections to overview doc 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40672
 2009-01-07 12:56:40 +00:00
elpolilla
cd148c946e Refactored extract adaptor 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40671
 2009-01-07 12:48:15 +00:00
elpolilla
66852b8291 Small bugfix in selectors constructor regarding strings and unicodes 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40670
 2009-01-07 12:09:47 +00:00
Pablo Hoffman
2c2dc61766 somes fixes and updates to scrapy documentation 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40669
 2009-01-07 03:59:39 +00:00
Pablo Hoffman
0319373c61 improved overview doc. closes #44 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40668
 2009-01-07 03:58:15 +00:00
elpolilla
a155ad26a9 Added documentation for Adaptors 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40667
 2009-01-07 03:18:18 +00:00
Pablo Hoffman
d1594df0cb made url_is_from_spider work for tuples in extra_domain_names 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40666
 2009-01-07 01:02:38 +00:00
Pablo Hoffman
a2d19760f3 doc: minor grammar correction 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40665
 2009-01-06 23:07:29 +00:00
Pablo Hoffman
abdf4dee50 added doc about scrapy architecture. closes #31 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40664
 2009-01-06 23:05:04 +00:00
olveyra
4286e5e47c fix to revision 660 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40663
 2009-01-06 19:43:58 +00:00
Pablo Hoffman
1d36c3f0dd updated robotstxt, spidermw and downloadermw docs 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40662
 2009-01-06 17:34:14 +00:00
Pablo Hoffman
e9d050b91d renamed spider middleware methods to more consistent ones: process_spider_input, process_spider_output, process_spider_exception 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40661
 2009-01-06 17:30:29 +00:00
Pablo Hoffman
cf34fd4c98 added spider middleware documentation. closes #28 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40660
 2009-01-06 16:53:49 +00:00
Pablo Hoffman
7f7196c150 removed unused import 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40659
 2009-01-06 16:44:06 +00:00
Pablo Hoffman
5e63690939 added DEFAULT_SPIDER to default_settings and fixed formatting in DEFAULT_SPIDER setting documentation 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40658
 2009-01-06 16:42:59 +00:00
elpolilla
c80fef7eb0 Added the example project from the tutorial 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40657
 2009-01-06 16:08:03 +00:00
Pablo Hoffman
9d6defa643 doc: fixed intro-install xref 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40656
 2009-01-06 14:34:26 +00:00
Pablo Hoffman
9da9041248 updated INSTALL file to point to docs/intro/install. also, utf-8 default encoding is no longer required 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40655
 2009-01-06 00:19:51 +00:00
Pablo Hoffman
3badafc8a7 updated installation doc 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40654
 2009-01-06 00:15:55 +00:00
Pablo Hoffman
c9d91106d8 updated documentation index 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40653
 2009-01-06 00:15:30 +00:00
Pablo Hoffman
4df3118c03 added note to scrapy version 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40652
 2009-01-05 23:49:06 +00:00
Pablo Hoffman
b3c7203ad9 updated some doc configuration - version, release and html_copy_source 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40651
 2009-01-05 23:29:25 +00:00
olveyra
9b857124c1 added DEFAULT_SPIDER setting help 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40650
 2009-01-05 18:11:48 +00:00
elpolilla
111ba5d583 Updated items documentation 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40649
 2009-01-05 16:55:20 +00:00
elpolilla
f1962c1c0a . Moved init and repr methods from RobustScrapedItem to ScrapedItem 
. Refactored ScrapedItem's attribute method, added docstring and tests

--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40648
 2009-01-05 16:54:54 +00:00
Pablo Hoffman
3c4012f924 updated scrapy version 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40647
 2009-01-05 03:37:06 +00:00
Pablo Hoffman
5c9c82d055 some improvements to selectors doc structure, added literalincludes for sample1.html (to avoid duplicating the content), renamed that file and moved to _static (so it appears on built doc), moved comments out of source code snippets and into documentation text, and splitted them. converted to required '>>>' console format, and added proper highlighting hints 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40646
 2009-01-05 02:49:23 +00:00
Pablo Hoffman
7ca989a781 disabled docs app (redirecting to doc.scrapy.org instead) 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40645
 2009-01-04 23:28:50 +00:00
Pablo Hoffman
e61b581890 fixed wrong xref name 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40644
 2009-01-04 19:03:15 +00:00