BackTrackZ/scrapy
BackTrackZ/scrapy
1
0
Fork 0
mirror of https://github.com/scrapy/scrapy.git synced 2025-02-27 06:23:41 +00:00
Code Releases Wiki Activity
585 Commits 39 Branches 119 Tags
Commit Graph

585 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Pablo Hoffman
1f247b1efc added settings documentation topic, and completed available settings reference. closes #30 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40585
 2008-12-30 13:28:36 +00:00
olveyra
af79bedf8f Added doc line in RegexLinkExtractor for the case when no allow/deny argument is given 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40584
 2008-12-30 13:04:06 +00:00
Ismael Carnales
8ef8766070 the badge is back to green ... hulk is angry 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40583
 2008-12-30 11:27:19 +00:00
Ismael Carnales
ad2c2368a7 using small gray badge 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40582
 2008-12-30 11:25:42 +00:00
Ismael Carnales
c149e2de2c removed menu from footer, added django badge 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40581
 2008-12-30 11:16:32 +00:00
Ismael Carnales
c9491b6041 replicating header menu in footer 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40580
 2008-12-30 10:56:15 +00:00
Ismael Carnales
81fecc0067 fixed settings indentation and added reference 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40579
 2008-12-30 10:45:19 +00:00
Ismael Carnales
36277f3a32 changed settings from description unit to crossreference 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40578
 2008-12-29 19:02:56 +00:00
Ismael Carnales
5a038e7d27 added docs breadrumb 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40577
 2008-12-29 18:46:43 +00:00
Ismael Carnales
e36f5624a7 fixed caps in news, reordered menu items 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40576
 2008-12-29 18:35:13 +00:00
Ismael Carnales
d619a11f88 make sidebar bigger 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40575
 2008-12-29 18:28:40 +00:00
Ismael Carnales
2f2c5cf19d removed django.contrib.comments requirement 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40574
 2008-12-29 18:27:16 +00:00
Ismael Carnales
3d85932ae0 moved blog to news 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40573
 2008-12-29 18:20:58 +00:00
Ismael Carnales
48682189b4 using a modified version of django simple blog 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40572
 2008-12-29 16:50:49 +00:00
Ismael Carnales
a5b609ac5f moved blog templates to backup folder 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40571
 2008-12-29 16:13:19 +00:00
elpolilla
25a38e53d5 Fixed minor encoding issues in adaptors 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40570
 2008-12-29 15:54:16 +00:00
elpolilla
bbd2a9d6f0 Fixed typo in test 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40569
 2008-12-29 15:17:34 +00:00
elpolilla
5eb7e5ba89 Fixed lots of encoding issues, and improved some adaptors tests 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40568
 2008-12-29 15:10:21 +00:00
Pablo Hoffman
fc3f66bd1c fixed grammar error 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40567
 2008-12-29 12:10:01 +00:00
Pablo Hoffman
b5a34dd21d started writing settings documentation 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40566
 2008-12-29 11:38:34 +00:00
Pablo Hoffman
a593af5f53 fixed cyclic import between scrapy.core.engine and scrapy.utils.db 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40565
 2008-12-28 08:45:27 +00:00
Pablo Hoffman
8a885e4dc8 moved scrapy-docs svn:external where it belongs 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40564
 2008-12-27 21:38:27 +00:00
Pablo Hoffman
06ad08c681 moved dia diagram to docs/media 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40563
 2008-12-27 21:32:18 +00:00
Pablo Hoffman
7e5b85ca9e moved scrapy docs from website source to scrapy source, since it makes more sense there 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40562
 2008-12-27 21:27:22 +00:00
Pablo Hoffman
97d4754200 moved docs to docs-old 
--HG--
rename : scrapy/trunk/docs/scrapy-architecture.dia => scrapy/trunk/docs-old/scrapy-architecture.dia
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40561
 2008-12-27 21:20:29 +00:00
Pablo Hoffman
913c34a828 added AUTHORS file 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40560
 2008-12-27 19:57:10 +00:00
Pablo Hoffman
77a99999f8 improved text 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40559
 2008-12-27 19:16:12 +00:00
Pablo Hoffman
a617c2cb36 some updates to home and download page 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40558
 2008-12-27 19:15:13 +00:00
Pablo Hoffman
ed5e25c913 removed tagline from scrapy logo 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40557
 2008-12-27 18:21:37 +00:00
Pablo Hoffman
f932a3d842 added spider_exceptions to scrapy stats 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40556
 2008-12-27 17:37:05 +00:00
Pablo Hoffman
6fa33f4463 fixed minor bug in scrapy manager 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40555
 2008-12-27 02:06:05 +00:00
Pablo Hoffman
e89ad0be8c fixed minor bug in scrapy manager 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40554
 2008-12-27 02:05:05 +00:00
Pablo Hoffman
9557e24203 added start_requests method to BaseSpider, made start_urls empty by default instead of a required attribute in every spider 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40553
 2008-12-27 01:07:29 +00:00
elpolilla
5b159d0f04 - Improved unquote_markup by using generators instead of lists 
- Added possibility of specifying headers in items_to_csv

--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40552
 2008-12-26 18:53:15 +00:00
elpolilla
f59f1c8bc0 Added items_to_csv function 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40551
 2008-12-26 16:10:03 +00:00
elpolilla
c7332cd372 Updated scrapy tutorial 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40550
 2008-12-26 14:03:45 +00:00
elpolilla
867ee20c66 Updated scrapy overview 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40549
 2008-12-26 12:21:53 +00:00
elpolilla
4ee3626763 Modified XMLFeedSpider in order to support parsing with HtmlXPathSelector 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40548
 2008-12-26 11:51:09 +00:00
elpolilla
b9360f659e Added tests for str_to_unicode and unicode_to_str 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40547
 2008-12-24 14:14:03 +00:00
elpolilla
86aa3bcb65 Added str_to_unicode and unicode_to_str functions, and used them in utils/markup 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40546
 2008-12-24 13:02:13 +00:00
elpolilla
95ed5d86a2 - Modified markup functions again; this time, in order to support utf-8 encoded strings (without interfering with unicode objects) 
- Reverted changes in tests

--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40545
 2008-12-24 00:00:14 +00:00
elpolilla
e2b0870957 Fixed markup tests. Those functions must always work with unicode objects. 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40544
 2008-12-23 23:27:53 +00:00
elpolilla
f4cecafc02 Removed some bogus decoding of strings in several markup-handling functions. Input must always be passed in a unicode object 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40543
 2008-12-23 23:12:56 +00:00
Daniel Grana
6d33e12c8b aws: sign requests if url hostname ends with s3.amazonaws.com common bucket suffix. This allows a spider to send requests to S3 with out need to sign it inside spider code 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40542
 2008-12-23 14:31:58 +00:00
elpolilla
0cd074c94c Some changes made to shell command: 
. Removed traces of the old decompressor tool
 . Made the responses go through the downloader middlewares after being downloaded
 . Added the command its own settings file

--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40541
 2008-12-23 14:21:46 +00:00
elpolilla
bd8bd25cd8 Removed unuseful and noisy test 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40540
 2008-12-23 01:37:32 +00:00
elpolilla
193fbfdc3d Updated scrapy's tutorial 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40539
 2008-12-19 14:46:56 +00:00
elpolilla
70af2e8e1d Modified Unquote adaptor to make use of the new unquote_markup function instead of remove_entities 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40538
 2008-12-19 11:09:51 +00:00
elpolilla
ac7ecb7810 Added adapt_response method to XMLFeedSpiders 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40537
 2008-12-19 10:16:58 +00:00
elpolilla
9d76ff606b Fixed bug in xmliter, which didnt add headers to the nodes it returned 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40536
 2008-12-19 10:16:31 +00:00