BackTrackZ/scrapy
BackTrackZ/scrapy
1
0
Fork 0
mirror of https://github.com/scrapy/scrapy.git synced 2025-02-26 16:44:22 +00:00
Code Releases Wiki Activity
993 Commits 39 Branches 119 Tags
Commit Graph

993 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Pablo Hoffman
d3c4d1f1e1 removed domain argument from Response constructor 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40743
 2009-01-18 16:38:01 +00:00
Pablo Hoffman
db91d26871 removed 'domain' argument from Response objects constructor. besides being a required first constructor argument, it wasn't actually needed and made the Response consturctor more complex 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40742
 2009-01-18 16:36:17 +00:00
Pablo Hoffman
654b49c86e added meta argument to Request & Response constructors 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40741
 2009-01-17 23:57:53 +00:00
Pablo Hoffman
8ecc6808e0 removed Request.context attribute (use Request.meta instead) 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40740
 2009-01-17 23:09:53 +00:00
Pablo Hoffman
7e640da433 renamed to_string() Request and Response methods to httprepr(). removed __len__() from Request and Response 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40739
 2009-01-17 22:11:54 +00:00
Pablo Hoffman
5dc1e7e5ca updated request/response reference doc 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40738
 2009-01-17 21:05:08 +00:00
Pablo Hoffman
da6a24b662 More Request/Response cleanup: 
* made status attribute an int
 * made engine use __str__ to display crawled requests
 * HTTP cache now inherits Response class to change __str__
 * added tests to check that the class is preserved on .copy() (for both Requests and Responses)
 * removed custom cached attribute (and passed to a Response.meta item)
 * removed some custom (and seldom used) methods from Response class: version(), info()
 * reinforced the privacy of the ResponseBody class, by renaming it to _ResponseBody and added a warning that it may be removed in the future
 * added tests for Request & Response to_string() methods
 * fixed minor (and harmless) bug in to_string() methods

--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40737
 2009-01-17 20:40:07 +00:00
Pablo Hoffman
b1745f49f1 removed deprecated original_url attribute from Response objects (it can be accessed through Response.request.url) 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40736
 2009-01-17 15:57:28 +00:00
Pablo Hoffman
7b545381bd changed log message and increased log level, when spiders return objects which are not Request or ScrapedItem 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40735
 2009-01-17 15:22:59 +00:00
Pablo Hoffman
6ba6238c83 Response class: 
* added meta and cache attributes to Response class
 * added tests for Response copy

Request class:
 * added meta attribute and renamed old _cache attribute to cache
 * moved depth and link_text to Request.meta
 * added tests for Request copy

* ResponseLibxml2 and ResponseSoup extensions now use Response.cache

Updated doc with changes

--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40734
 2009-01-15 03:24:48 +00:00
Pablo Hoffman
d26a54f541 added tests for ResponseSoup and ResponseLibxml2 extensions 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40733
 2009-01-15 03:06:00 +00:00
Pablo Hoffman
604af8e74f doc; removed referer argument from Request constructor 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40732
 2009-01-15 00:20:24 +00:00
Pablo Hoffman
2a7b41cdb2 removed referer argument from Request constructor. refs #48 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40731
 2009-01-15 00:10:31 +00:00
Daniel Grana
9513b1f465 Remove response referneces from pipelines. refs #51 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40730
 2009-01-14 23:59:45 +00:00
Pablo Hoffman
eef01a9fdd removed Request.method magic in Request constructor. refs #48 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40729
 2009-01-14 23:50:23 +00:00
Pablo Hoffman
ae95c1df68 removed unused (and broken) prepend_callback Request method 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40728
 2009-01-14 23:31:24 +00:00
Pablo Hoffman
3f89fc10b7 shortened some line widths 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40727
 2009-01-14 23:23:10 +00:00
Pablo Hoffman
1272b138ea moved HTTP auth functionality out of Request class and into scrapy.utils.request.request_authenticate function, added tests 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40726
 2009-01-14 22:02:58 +00:00
Andres Moreira
8fc4719d0c Added dns cache support for the crawler, improving the performance of the page download because this reduce the dns lookups. 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40725
 2009-01-14 18:59:52 +00:00
samus_
7be6ff0727 typo 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40724
 2009-01-14 12:12:37 +00:00
samus_
baf9a8d846 renamed expiration setting to the same used by the image pipeline 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40723
 2009-01-14 12:09:06 +00:00
Pablo Hoffman
0ea37f51db * moved request fingerprinting from Request class to scrapy.utils.request - closes #50 
* cleaned up fingerprint tests suite (only left relevant tests)

--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40722
 2009-01-14 01:17:40 +00:00
Pablo Hoffman
519458bdae added documentation for settings: ENGINE_DEBUG, DOWNLOADER_DEBUG 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40721
 2009-01-14 00:19:22 +00:00
Pablo Hoffman
64d1f67c57 decreased logging level of RequestLimitMiddleware to DEBUG 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40720
 2009-01-13 21:47:49 +00:00
Pablo Hoffman
4ed811b4d3 added DOWNLOAD_DELAY to default_settings and documentation, fixed some typo errors in settings reference 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40719
 2009-01-13 14:43:38 +00:00
Pablo Hoffman
0e78003c92 removed my email from CLOSEDOMAIN_NOTIFY setting 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40718
 2009-01-13 13:50:51 +00:00
Pablo Hoffman
e316722bb1 updated doc: ref/emails.rst and topics/downloader-middleware.rst 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40717
 2009-01-13 11:55:20 +00:00
Pablo Hoffman
468bfeb278 removed unused imports 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40716
 2009-01-13 10:10:19 +00:00
Pablo Hoffman
95d99d51b9 renamde old SchedulerStats web console module to ScheduleQueue and made it work with the new PriorityQueue 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40715
 2009-01-13 02:49:50 +00:00
Pablo Hoffman
deb960526b removed unused (Django) classes from scrapy.utils.datatypes: MergeDict, SortedDict, DotExpandedDict, FileDict. And also removed unused class gzStringIO 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40714
 2009-01-13 01:46:45 +00:00
Pablo Hoffman
ff637d9a0a added __len__ to PriorityQueue/Stack, and changed __iter__ implementation to return (item, priority) tuples, added more test cases 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40713
 2009-01-13 01:37:49 +00:00
Pablo Hoffman
2434000cda * ported PriorityQueue and PriorityStack to use heapq instead of queue.Queue + 
bisect which was up to 5x slower!
 * added test case for PriorityStack (only PriorityQueue had before)
 * changed Priority{Stack,Queue} API to just push(), pop(), and made them
   iterable

--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40712
 2009-01-13 01:14:40 +00:00
samus_
8b28d365b1 removed extra return 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40711
 2009-01-12 22:43:10 +00:00
Andres Moreira
eca60c7c4d Small change in canonicalize_url improved its performance a bit 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40710
 2009-01-12 17:18:54 +00:00
Pablo Hoffman
30e44c9a58 added settings: REQUEST_HEADER_ACCEPT, REQUEST_HEADER_ACCEPT_LANGUAGE. started built-in downloader middleware reference 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40709
 2009-01-12 00:53:37 +00:00
Pablo Hoffman
d0046196d8 ported MailSender class to use twisted non-blocking IO 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40708
 2009-01-11 23:04:50 +00:00
Pablo Hoffman
b0e37dc36a renamed StackTraceDebug extension to StackTraceDump 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40707
 2009-01-11 21:27:38 +00:00
Pablo Hoffman
73074721de improved settings doc 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40706
 2009-01-11 20:04:13 +00:00
Pablo Hoffman
dfdc04c28c some email doc improvments 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40705
 2009-01-11 19:49:11 +00:00
Pablo Hoffman
09459ed7f7 added logging doc 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40704
 2009-01-11 19:48:36 +00:00
Pablo Hoffman
9e93070382 moved email doc to reference (instead of topics) 
--HG--
rename : scrapy/trunk/docs/topics/email.rst => scrapy/trunk/docs/ref/email.rst
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40703
 2009-01-11 19:14:49 +00:00
Pablo Hoffman
5c15def3a5 added doc for scrapy.mail 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40702
 2009-01-11 19:11:17 +00:00
Pablo Hoffman
bf0050c321 added doc for extensions and web console (closes #29 and #33). also started stats doc 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40701
 2009-01-11 06:34:38 +00:00
Pablo Hoffman
300a0f4901 minor (and inoffensive) code improvements and fixes found while documenting 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40700
 2009-01-11 06:31:07 +00:00
Pablo Hoffman
7cfefc6e70 some minor doc improvements here and there 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40699
 2009-01-09 22:45:58 +00:00
Pablo Hoffman
45e812a8bd added misc section to doc 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40698
 2009-01-09 22:45:09 +00:00
Pablo Hoffman
0e6b518f35 added FAQ entry about Django 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40697
 2009-01-09 21:18:41 +00:00
elpolilla
71f7d62c68 Bugfix in AWSMiddleware regarding requests from local files 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40696
 2009-01-09 17:35:04 +00:00
elpolilla
5fa1009712 Improved adaptors documentation 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40695
 2009-01-09 13:45:11 +00:00
elpolilla
2cfd292f01 Added needed conversion from unicode to string before using twisted's logging system because it may trigger encoding issues 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40694
 2009-01-09 10:49:48 +00:00