BackTrackZ/scrapy
BackTrackZ/scrapy
1
0
Fork 0
mirror of https://github.com/scrapy/scrapy.git synced 2025-02-26 23:04:29 +00:00
Code Releases Wiki Activity
710 Commits 39 Branches 119 Tags
Commit Graph

710 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Andres Moreira
eca60c7c4d Small change in canonicalize_url improved its performance a bit 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40710
 2009-01-12 17:18:54 +00:00
Pablo Hoffman
30e44c9a58 added settings: REQUEST_HEADER_ACCEPT, REQUEST_HEADER_ACCEPT_LANGUAGE. started built-in downloader middleware reference 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40709
 2009-01-12 00:53:37 +00:00
Pablo Hoffman
d0046196d8 ported MailSender class to use twisted non-blocking IO 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40708
 2009-01-11 23:04:50 +00:00
Pablo Hoffman
b0e37dc36a renamed StackTraceDebug extension to StackTraceDump 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40707
 2009-01-11 21:27:38 +00:00
Pablo Hoffman
73074721de improved settings doc 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40706
 2009-01-11 20:04:13 +00:00
Pablo Hoffman
dfdc04c28c some email doc improvments 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40705
 2009-01-11 19:49:11 +00:00
Pablo Hoffman
09459ed7f7 added logging doc 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40704
 2009-01-11 19:48:36 +00:00
Pablo Hoffman
9e93070382 moved email doc to reference (instead of topics) 
--HG--
rename : scrapy/trunk/docs/topics/email.rst => scrapy/trunk/docs/ref/email.rst
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40703
 2009-01-11 19:14:49 +00:00
Pablo Hoffman
5c15def3a5 added doc for scrapy.mail 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40702
 2009-01-11 19:11:17 +00:00
Pablo Hoffman
bf0050c321 added doc for extensions and web console (closes #29 and #33). also started stats doc 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40701
 2009-01-11 06:34:38 +00:00
Pablo Hoffman
300a0f4901 minor (and inoffensive) code improvements and fixes found while documenting 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40700
 2009-01-11 06:31:07 +00:00
Pablo Hoffman
7cfefc6e70 some minor doc improvements here and there 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40699
 2009-01-09 22:45:58 +00:00
Pablo Hoffman
45e812a8bd added misc section to doc 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40698
 2009-01-09 22:45:09 +00:00
Pablo Hoffman
0e6b518f35 added FAQ entry about Django 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40697
 2009-01-09 21:18:41 +00:00
elpolilla
71f7d62c68 Bugfix in AWSMiddleware regarding requests from local files 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40696
 2009-01-09 17:35:04 +00:00
elpolilla
5fa1009712 Improved adaptors documentation 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40695
 2009-01-09 13:45:11 +00:00
elpolilla
2cfd292f01 Added needed conversion from unicode to string before using twisted's logging system because it may trigger encoding issues 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40694
 2009-01-09 10:49:48 +00:00
Pablo Hoffman
4c54305bee added docstrings to unicode_to_str and str_to_unicode 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40693
 2009-01-09 10:35:14 +00:00
samus_
8e38f70a74 small improvement to Response.__init__ testcase 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40692
 2009-01-09 01:33:12 +00:00
samus_
d3ac5d5ab8 added test for Response.__init__ 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40691
 2009-01-09 00:45:12 +00:00
samus_
8240fc48ad refactored ResponseBody's encoding test 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40690
 2009-01-09 00:42:44 +00:00
samus_
1feace1bb0 a bit of performance 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40689
 2009-01-08 18:29:56 +00:00
samus_
d0e5245fba fixed bug 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40688
 2009-01-08 18:29:33 +00:00
samus_
c601bbb083 fixed bug in copy method of Response (tests coming soon) 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40687
 2009-01-08 16:08:04 +00:00
elpolilla
5cb9f344ac Changed method process_spider_output name to process_results in crawl and feed spiders 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40686
 2009-01-08 15:42:33 +00:00
Pablo Hoffman
6568dc4f35 added custom CSS for scrapy doc with minor modifications (colors only) 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40685
 2009-01-08 15:02:39 +00:00
Pablo Hoffman
6420c407d2 removed scrapyengine import from downoader code, minor improvements to docstrings 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40684
 2009-01-08 14:21:20 +00:00
elpolilla
f39b7b507a Disabled test until memory leak in libxml is fixed 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40683
 2009-01-08 13:46:14 +00:00
Pablo Hoffman
fc76af2eb4 reverted r680 until we there are tests and documentation available about the change 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40682
 2009-01-08 13:30:25 +00:00
elpolilla
4a9a48a629 . Added strip() to link texts in LinkExtractor in order to avoid extracting line breaks and such 
. Added test for restrict_xpaths in RegexLinkExtractor

--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40681
 2009-01-08 11:05:15 +00:00
elpolilla
8278c0cffc Changed process_results name in feed spiders to process_spider_output and moved its parameters to a more standard order 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40680
 2009-01-08 10:11:56 +00:00
Pablo Hoffman
32d842326a reverted r675 which broke precedence for environment variables 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40679
 2009-01-07 18:06:39 +00:00
Pablo Hoffman
393cf349a0 updated settings doc 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40678
 2009-01-07 18:04:40 +00:00
elpolilla
382621b814 Fixed bug in RegexLinkExtractor. Encoding was not being specified 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40677
 2009-01-07 17:39:09 +00:00
elpolilla
307f8b3321 . Modified loading of command-specific settings (were loaded as defaults, now as overrides) 
. Removed duplicated loading of extensions in shell command

--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40676
 2009-01-07 17:31:06 +00:00
elpolilla
8029275c75 Removed non-scrapy import in shell command 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40675
 2009-01-07 14:18:03 +00:00
Pablo Hoffman
9884c209c5 minor update to overview doc 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40674
 2009-01-07 14:02:28 +00:00
elpolilla
e4852bca78 Modified extract adaptor to make use of "adaptor_args" (as it should), and added test for AdaptorPipes 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40673
 2009-01-07 14:00:54 +00:00
Pablo Hoffman
41fbcbde48 minor corrections to overview doc 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40672
 2009-01-07 12:56:40 +00:00
elpolilla
cd148c946e Refactored extract adaptor 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40671
 2009-01-07 12:48:15 +00:00
elpolilla
66852b8291 Small bugfix in selectors constructor regarding strings and unicodes 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40670
 2009-01-07 12:09:47 +00:00
Pablo Hoffman
2c2dc61766 somes fixes and updates to scrapy documentation 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40669
 2009-01-07 03:59:39 +00:00
Pablo Hoffman
0319373c61 improved overview doc. closes #44 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40668
 2009-01-07 03:58:15 +00:00
elpolilla
a155ad26a9 Added documentation for Adaptors 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40667
 2009-01-07 03:18:18 +00:00
Pablo Hoffman
d1594df0cb made url_is_from_spider work for tuples in extra_domain_names 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40666
 2009-01-07 01:02:38 +00:00
Pablo Hoffman
a2d19760f3 doc: minor grammar correction 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40665
 2009-01-06 23:07:29 +00:00
Pablo Hoffman
abdf4dee50 added doc about scrapy architecture. closes #31 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40664
 2009-01-06 23:05:04 +00:00
olveyra
4286e5e47c fix to revision 660 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40663
 2009-01-06 19:43:58 +00:00
Pablo Hoffman
1d36c3f0dd updated robotstxt, spidermw and downloadermw docs 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40662
 2009-01-06 17:34:14 +00:00
Pablo Hoffman
e9d050b91d renamed spider middleware methods to more consistent ones: process_spider_input, process_spider_output, process_spider_exception 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40661
 2009-01-06 17:30:29 +00:00