BackTrackZ/scrapy
BackTrackZ/scrapy
1
0
Fork 0
mirror of https://github.com/scrapy/scrapy.git synced 2025-02-28 16:36:32 +00:00
Code Releases Wiki Activity
814 Commits 39 Branches 119 Tags
Commit Graph

694 Commits

Author SHA1 Message Date
Pablo Hoffman
005a642240 added SETTINGS_DISABLED environment variable to turn off custom settings (and only use Scrapy defaults) 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40814
 2009-01-30 23:18:23 +00:00
Pablo Hoffman
16efbf87fc added some references to documentation and fixed some doc typos (thanks Patrick) 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40813
 2009-01-30 22:33:50 +00:00
Pablo Hoffman
c672223091 splitted spiders doc from link extractor docs, moved the corresponding parts to ref and topics 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40812
 2009-01-30 21:53:40 +00:00
Pablo Hoffman
483ef3ba7f changed typo in Items docs 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40811
 2009-01-30 21:52:41 +00:00
Ismael Carnales
a834eef6c9 added BaseSpider attributes and method references and metadata (from tutorial) 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40810
 2009-01-30 19:24:43 +00:00
Ismael Carnales
7e6c9c2e25 formatting changes and references to spiders added in the tutorial 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40809
 2009-01-30 19:14:16 +00:00
Ismael Carnales
2526e11be0 changed formatting a little bit and added class references to spiders topic 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40808
 2009-01-30 19:12:09 +00:00
Ismael Carnales
591a4a2bb5 added information on default item in new tutorial 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40807
 2009-01-30 14:45:52 +00:00
Ismael Carnales
f27d07c1a4 added warning of spider naming in new tutorial 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40806
 2009-01-30 14:26:40 +00:00
Ismael Carnales
3b012d3981 modified default project template structure in proposed tutorial 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40805
 2009-01-30 13:15:50 +00:00
elpolilla
648753190c Several corrections made to items and adaptors documentation 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40804
 2009-01-30 13:05:52 +00:00
Ismael Carnales
d9a90a1f3f added proposed documentation, moved tutorial there 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40803
 2009-01-30 11:29:55 +00:00
elpolilla
6cc806a843 Added adaptors diagram svg to the repo 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40802
 2009-01-30 11:19:20 +00:00
Pablo Hoffman
bcf6e5476b added one more case to responsetypes 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40801
 2009-01-30 11:14:08 +00:00
elpolilla
769519c2aa Added missing adaptors diagram to documentation 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40800
 2009-01-30 10:57:12 +00:00
Pablo Hoffman
fb13d3d83b doc: added status argument to domain_closed signal 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40799
 2009-01-29 20:18:53 +00:00
Daniel Grana
b3050ca0e4 pipeline: remove open_domain/close_domain hooks, use domain_open and domain_closed signals instead. docs updated. 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40798
 2009-01-29 19:18:03 +00:00
Pablo Hoffman
73c924643c bundled mime.types file with scrapy for platforms that have poor support for mime.types (windows), added from_args() convenience method to ResponseTypes 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40797
 2009-01-29 18:52:46 +00:00
Daniel Grana
f4e6ed3c2f run without RobotTxtMiddleware by default and update docs 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40796
 2009-01-29 18:22:59 +00:00
Ismael Carnales
e6375c6a5b added some info on items pipeline on the tutorial 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40795
 2009-01-29 17:57:21 +00:00
elpolilla
9634af9aa7 Updated adaptors and items documentation 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40794
 2009-01-29 17:46:32 +00:00
Ismael Carnales
ff88171b05 added items to tutorial 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40793
 2009-01-29 14:35:54 +00:00
Ismael Carnales
04fc0471ec first version of the new tutorial (work in progress) 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40792
 2009-01-29 12:29:28 +00:00
elpolilla
3ca2187e3b Removed non-generic implementation of _add_single_attributes and modified test 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40791
 2009-01-29 02:21:08 +00:00
elpolilla
bc4fcbeab7 Added response type recognition to local file urls 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40790
 2009-01-28 11:45:39 +00:00
Andres Moreira
e9fbce4ee2 Improved code of utils.markup functions. 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40789
 2009-01-27 17:58:51 +00:00
elpolilla
c0601cd03e Refactored remove_entities and fixed the hex entities bug (they werent recognised). Also, tests were added 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40788
 2009-01-27 14:05:22 +00:00
Pablo Hoffman
c3d61dc999 changing extension in test from csv to txt (not all systems support the text/csv mimetype) 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40787
 2009-01-27 13:33:28 +00:00
Pablo Hoffman
134b867abe added support for unknown file extensions to ResponseTypes.from_filename 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40786
 2009-01-27 13:08:07 +00:00
elpolilla
14e67d59c2 Added test for encoding in csviter 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40785
 2009-01-27 12:35:09 +00:00
Pablo Hoffman
1e55e88111 fixed some typos in previous commit 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40784
 2009-01-27 12:18:23 +00:00
Pablo Hoffman
11a9236125 added example about creating Requests with cookies 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40783
 2009-01-27 12:17:40 +00:00
Pablo Hoffman
14ceca98bf added FormRequest example 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40782
 2009-01-27 12:10:49 +00:00
Pablo Hoffman
e25bfa5d88 added more cases to ResponseTypes, and tests for ResponseTypes 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40781
 2009-01-27 11:27:09 +00:00
elpolilla
b6246cbc37 Fixed encoding-related bug in csviter 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40780
 2009-01-27 11:10:42 +00:00
Pablo Hoffman
4013497edb more patches sent by Patrick 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40779
 2009-01-26 23:42:51 +00:00
Pablo Hoffman
c1a1b8945a some doc fixes suggested by Patrick Mézard 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40778
 2009-01-26 23:38:21 +00:00
Pablo Hoffman
57189e1b92 applied Patrick Mézard patch for loading local files 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40777
 2009-01-26 23:31:04 +00:00
Pablo Hoffman
ce1700dd8e added libxml2 installation steps for MacOSX (thanks Patrick Mézard) 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40776
 2009-01-26 23:28:19 +00:00
Pablo Hoffman
279b9ac40d updated adaptors docs to reflect its inestability and improved the ReST formatting (80 column lines, no ugly pipes at the beginnig of lines) 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40775
 2009-01-26 23:22:53 +00:00
elpolilla
e8e87bcb52 Fixed small bug in adaptor pipelines that tried to work with None values 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40774
 2009-01-26 16:20:58 +00:00
Pablo Hoffman
7ed88fd0f3 added Content-Disposition encoding discovery to ResponseTypes 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40773
 2009-01-26 15:36:53 +00:00
elpolilla
9de6ee5109 Added missing test for change in r769 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40772
 2009-01-26 15:25:53 +00:00
elpolilla
91eff31f18 . Modified the default value of the BOT_NAME setting to the project's name 
. Modified spider templates to use the already-generated example item instead of a ScrapedItem

--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40771
 2009-01-26 15:24:52 +00:00
elpolilla
f63a661320 Normalized the usage of ints for storing http status codes 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40770
 2009-01-26 15:03:47 +00:00
elpolilla
74661d54d0 Added application/xml mimetype to the known response types 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40769
 2009-01-26 15:03:14 +00:00
elpolilla
b9d3a2cb96 Improved adaptors debugging in order to make it clearer for reading 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40768
 2009-01-26 12:15:31 +00:00
Pablo Hoffman
bc4e80f640 reverted to IO-blocking MailSender implementation (using standard smtplib) until we fix some problems with deferred left unexectuted when stopping the engine 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40767
 2009-01-26 03:40:59 +00:00
Pablo Hoffman
5f0d5a1653 Big Response/Request refactoring: 
- added Response subclasses: TextResponse, HtmlResponse, XmlResponse
- made Response.body a str
- added Response.body_as_unicode() method
- added encoding attribute for TextResponse and subclasses
- added headers_encoding() and body_encoding() to TextResponse and subclasses
- added ResponseTypes class to guess the Response class to use based on
  mimetype and other criteria

- added and improved several Request/Response tests
- updated request/response documetnation to reflect the changes

Another changes not related to encoding:

- added lixbml2debug for debugging libxml2 memory leaks, which can be enabled
  by a environment variable
- added memoizemethod decorator (implemented using descriptors) to cache the
  result of methods
- moved DecompressionMiddleware to contrib_exp

--HG--
rename : scrapy/trunk/scrapy/tests/test_spiders/testplugin.py => scrapy/trunk/scrapy/tests/test_spiders/testspider.py
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40766
 2009-01-26 02:57:03 +00:00
elpolilla
dbaf602730 . Updated some adaptors docstrings 
. Turned Delist and Unquote adaptors into factory functions instead of classes
. Updated tests

--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40765
 2009-01-26 01:15:05 +00:00