BackTrackZ/scrapy
BackTrackZ/scrapy
1
0
Fork 0
mirror of https://github.com/scrapy/scrapy.git synced 2025-02-26 20:44:04 +00:00
Code Releases Wiki Activity
1,016 Commits 39 Branches 119 Tags
Commit Graph

1016 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Pablo Hoffman
468bfeb278 removed unused imports 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40716
 2009-01-13 10:10:19 +00:00
Pablo Hoffman
95d99d51b9 renamde old SchedulerStats web console module to ScheduleQueue and made it work with the new PriorityQueue 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40715
 2009-01-13 02:49:50 +00:00
Pablo Hoffman
deb960526b removed unused (Django) classes from scrapy.utils.datatypes: MergeDict, SortedDict, DotExpandedDict, FileDict. And also removed unused class gzStringIO 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40714
 2009-01-13 01:46:45 +00:00
Pablo Hoffman
ff637d9a0a added __len__ to PriorityQueue/Stack, and changed __iter__ implementation to return (item, priority) tuples, added more test cases 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40713
 2009-01-13 01:37:49 +00:00
Pablo Hoffman
2434000cda * ported PriorityQueue and PriorityStack to use heapq instead of queue.Queue + 
bisect which was up to 5x slower!
 * added test case for PriorityStack (only PriorityQueue had before)
 * changed Priority{Stack,Queue} API to just push(), pop(), and made them
   iterable

--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40712
 2009-01-13 01:14:40 +00:00
samus_
8b28d365b1 removed extra return 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40711
 2009-01-12 22:43:10 +00:00
Andres Moreira
eca60c7c4d Small change in canonicalize_url improved its performance a bit 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40710
 2009-01-12 17:18:54 +00:00
Pablo Hoffman
30e44c9a58 added settings: REQUEST_HEADER_ACCEPT, REQUEST_HEADER_ACCEPT_LANGUAGE. started built-in downloader middleware reference 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40709
 2009-01-12 00:53:37 +00:00
Pablo Hoffman
d0046196d8 ported MailSender class to use twisted non-blocking IO 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40708
 2009-01-11 23:04:50 +00:00
Pablo Hoffman
b0e37dc36a renamed StackTraceDebug extension to StackTraceDump 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40707
 2009-01-11 21:27:38 +00:00
Pablo Hoffman
73074721de improved settings doc 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40706
 2009-01-11 20:04:13 +00:00
Pablo Hoffman
dfdc04c28c some email doc improvments 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40705
 2009-01-11 19:49:11 +00:00
Pablo Hoffman
09459ed7f7 added logging doc 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40704
 2009-01-11 19:48:36 +00:00
Pablo Hoffman
9e93070382 moved email doc to reference (instead of topics) 
--HG--
rename : scrapy/trunk/docs/topics/email.rst => scrapy/trunk/docs/ref/email.rst
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40703
 2009-01-11 19:14:49 +00:00
Pablo Hoffman
5c15def3a5 added doc for scrapy.mail 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40702
 2009-01-11 19:11:17 +00:00
Pablo Hoffman
bf0050c321 added doc for extensions and web console (closes #29 and #33). also started stats doc 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40701
 2009-01-11 06:34:38 +00:00
Pablo Hoffman
300a0f4901 minor (and inoffensive) code improvements and fixes found while documenting 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40700
 2009-01-11 06:31:07 +00:00
Pablo Hoffman
7cfefc6e70 some minor doc improvements here and there 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40699
 2009-01-09 22:45:58 +00:00
Pablo Hoffman
45e812a8bd added misc section to doc 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40698
 2009-01-09 22:45:09 +00:00
Pablo Hoffman
0e6b518f35 added FAQ entry about Django 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40697
 2009-01-09 21:18:41 +00:00
elpolilla
71f7d62c68 Bugfix in AWSMiddleware regarding requests from local files 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40696
 2009-01-09 17:35:04 +00:00
elpolilla
5fa1009712 Improved adaptors documentation 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40695
 2009-01-09 13:45:11 +00:00
elpolilla
2cfd292f01 Added needed conversion from unicode to string before using twisted's logging system because it may trigger encoding issues 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40694
 2009-01-09 10:49:48 +00:00
Pablo Hoffman
4c54305bee added docstrings to unicode_to_str and str_to_unicode 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40693
 2009-01-09 10:35:14 +00:00
samus_
8e38f70a74 small improvement to Response.__init__ testcase 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40692
 2009-01-09 01:33:12 +00:00
samus_
d3ac5d5ab8 added test for Response.__init__ 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40691
 2009-01-09 00:45:12 +00:00
samus_
8240fc48ad refactored ResponseBody's encoding test 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40690
 2009-01-09 00:42:44 +00:00
samus_
1feace1bb0 a bit of performance 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40689
 2009-01-08 18:29:56 +00:00
samus_
d0e5245fba fixed bug 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40688
 2009-01-08 18:29:33 +00:00
samus_
c601bbb083 fixed bug in copy method of Response (tests coming soon) 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40687
 2009-01-08 16:08:04 +00:00
elpolilla
5cb9f344ac Changed method process_spider_output name to process_results in crawl and feed spiders 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40686
 2009-01-08 15:42:33 +00:00
Pablo Hoffman
6568dc4f35 added custom CSS for scrapy doc with minor modifications (colors only) 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40685
 2009-01-08 15:02:39 +00:00
Pablo Hoffman
6420c407d2 removed scrapyengine import from downoader code, minor improvements to docstrings 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40684
 2009-01-08 14:21:20 +00:00
elpolilla
f39b7b507a Disabled test until memory leak in libxml is fixed 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40683
 2009-01-08 13:46:14 +00:00
Pablo Hoffman
fc76af2eb4 reverted r680 until we there are tests and documentation available about the change 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40682
 2009-01-08 13:30:25 +00:00
elpolilla
4a9a48a629 . Added strip() to link texts in LinkExtractor in order to avoid extracting line breaks and such 
. Added test for restrict_xpaths in RegexLinkExtractor

--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40681
 2009-01-08 11:05:15 +00:00
elpolilla
8278c0cffc Changed process_results name in feed spiders to process_spider_output and moved its parameters to a more standard order 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40680
 2009-01-08 10:11:56 +00:00
Pablo Hoffman
32d842326a reverted r675 which broke precedence for environment variables 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40679
 2009-01-07 18:06:39 +00:00
Pablo Hoffman
393cf349a0 updated settings doc 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40678
 2009-01-07 18:04:40 +00:00
elpolilla
382621b814 Fixed bug in RegexLinkExtractor. Encoding was not being specified 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40677
 2009-01-07 17:39:09 +00:00
elpolilla
307f8b3321 . Modified loading of command-specific settings (were loaded as defaults, now as overrides) 
. Removed duplicated loading of extensions in shell command

--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40676
 2009-01-07 17:31:06 +00:00
elpolilla
8029275c75 Removed non-scrapy import in shell command 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40675
 2009-01-07 14:18:03 +00:00
Pablo Hoffman
9884c209c5 minor update to overview doc 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40674
 2009-01-07 14:02:28 +00:00
elpolilla
e4852bca78 Modified extract adaptor to make use of "adaptor_args" (as it should), and added test for AdaptorPipes 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40673
 2009-01-07 14:00:54 +00:00
Pablo Hoffman
41fbcbde48 minor corrections to overview doc 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40672
 2009-01-07 12:56:40 +00:00
elpolilla
cd148c946e Refactored extract adaptor 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40671
 2009-01-07 12:48:15 +00:00
elpolilla
66852b8291 Small bugfix in selectors constructor regarding strings and unicodes 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40670
 2009-01-07 12:09:47 +00:00
Pablo Hoffman
2c2dc61766 somes fixes and updates to scrapy documentation 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40669
 2009-01-07 03:59:39 +00:00
Pablo Hoffman
0319373c61 improved overview doc. closes #44 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40668
 2009-01-07 03:58:15 +00:00
elpolilla
a155ad26a9 Added documentation for Adaptors 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40667
 2009-01-07 03:18:18 +00:00