BackTrackZ/scrapy
BackTrackZ/scrapy
1
0
Fork 0
mirror of https://github.com/scrapy/scrapy.git synced 2025-02-26 14:44:08 +00:00
Code Releases Wiki Activity
3,227 Commits 39 Branches 119 Tags
Commit Graph

3227 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Pablo Hoffman
e9cef3ae69 better description for scrapy check command 2012-10-12 20:01:43 -02:00
Pablo Hoffman
a0e2b6e3ff simplified backwards compatibility of scrapy.conf 2012-10-12 17:28:02 -02:00
Pablo Hoffman
9d7106219a added deprecation warning when importing scrapy.conf module 2012-10-12 17:24:25 -02:00
Pablo Hoffman
b038bb8504 minor message update: spider stats -> Scrapy stats 2012-10-12 16:53:44 -02:00
Artem Bogomyagkov
1d5967cee6 httpcompression middleware improvement 2012-10-10 20:20:13 +03:00
Pablo Hoffman
1a905d62f5 removed scrapy.log.started attribute, and avoid checking if log has already been started (since it should be called once anyway) 2012-10-09 16:05:19 -02:00
Pablo Hoffman
1f89eb59fe fixed doc reference to topics-contracts 2012-10-09 16:02:12 -02:00
Pablo Hoffman
9e4c6aba2f updated release notes with new stats recently added 2012-10-09 12:50:42 -02:00
Pablo Hoffman
bd3a109299 Track number of log messages via Scrapy stats 
Log messages count are grouped per LEVEL like:

    'log_count/DEBUG': 8,
    'log_count/ERROR': 1,
    'log_count/INFO': 4,

This required passing the Crawler object to scrapy logging facility, so
the scrapy.log module underwent a few changes which should be backwards
compatible (or as backwards compatible as they could be).
 2012-10-09 12:41:31 -02:00
Pablo Hoffman
1ae1a3ea52 don't reset scrapy stats when spider is opened 2012-10-09 12:40:51 -02:00
Pablo Hoffman
ceeea27d21 added response_received_count to scrapy stats 2012-10-09 12:39:30 -02:00
Pablo Hoffman
452595790e fix test-scrapyd.sh after changes made to website 2012-10-08 15:30:19 -02:00
Daniel Graña
3c3c0c1f3f notify #scrapy room of travis build results 2012-10-06 15:16:58 -03:00
Daniel Graña
05ce20dab3 more cases where response.meta is prefered 2012-10-06 15:16:58 -03:00
Pablo Hoffman
1e1cc76fb7 Merge pull request #179 from artem-dev/errback_fix 
This fixes the case where failure.request wasn't available when the failure was generated in a spider middleware (for example, the HttpError middleware).
 2012-10-06 08:47:26 -07:00
Artem Bogomyagkov
0256a34f97 fix for request's errback, allows to get failed requests correctly 2012-10-05 20:44:20 +03:00
Daniel Graña
eee8e70b7b response.meta exposes resquest.meta for binded responses 2012-10-05 10:57:58 -03:00
Pablo Hoffman
dd13dfe82b Raise error when settings module is missing. 
Previously, it failed silently if an ImportError was caught when trying
to import the scrapy settings module. This not only happened when the
scrapy settings module itself was missing, but also when it tried to
import a missing module, which made the whole thing a bad idea.

A side effect of this change (not required, but for simplification) is
that we no longer support the default "scrapy_settings" name for the
scrapy settings module, but this was never used afaik.
 2012-10-03 12:31:19 -03:00
Shane Evans
ce80e5c792 fix link to online installation instructions 2012-10-02 12:26:14 +01:00
Pablo Hoffman
7458092eef added spider contracts to release notes and warn that its API is still subject to change 2012-09-29 03:06:30 -03:00
Pablo Hoffman
34f1477317 make tests code python 2.6 compatible 2012-09-28 18:55:12 -03:00
Pablo Hoffman
c380910b40 Merge pull request #167 from alexcepoi/sep-017 
Spider contracts (SEP-017)
 2012-09-28 13:57:07 -07:00
Pablo Hoffman
a5f8943d3e added change introduced in previous commit to release notes 2012-09-26 13:17:01 -03:00
Pablo Hoffman
4ca35dba87 Merge pull request #175 from euphoris/master 
add options -o and -t to the runspider command
 2012-09-26 09:15:40 -07:00
Jae-Myoung Yu
e94bd8187c add options -o and -t to the runspider command 
copy the codes from commands/crawl.py to commands/runspider.py
 2012-09-25 09:38:00 +09:00
Pablo Hoffman
a6eacf2c9b Merge pull request #172 from artem-dev/new_git_version 
nicer deploy versions for git repos if tags are available
 2012-09-21 07:39:47 -07:00
Artem Bogomyagkov
7685cb14a4 nicer deploy versions for git repos if tags are available 2012-09-21 12:34:45 +03:00
Alex Cepoi
73e6bc1b10 remove unused import 2012-09-21 00:54:11 +02:00
Alex Cepoi
11d29c7005 SEP-017 contracts: add tests and minor improvements 2012-09-21 00:12:46 +02:00
Pablo Hoffman
b46b5a6ef0 Documented AutoThrottle extension and added to extensions available by 
default. Also deprecated concurrency and delay settings, in favour of
using the standard Scrapy ones.
 2012-09-20 18:52:57 -03:00
Pablo Hoffman
c1cbc5de3f renamed CONTRIBUTING to CONTRIBUTING.md so that links are rendered as links in github 2012-09-19 13:58:58 -03:00
Pablo Hoffman
f3516aed69 added CONTRIBUTING file, for github: https://github.com/blog/1184-contributing-guidelines 2012-09-19 13:54:16 -03:00
Pablo Hoffman
e8eacd6f9e minor code simplification 2012-09-19 13:28:34 -03:00
Pablo Hoffman
7afedb89e9 backwards compatibility support for scrapy.conf.settings singleton for when scrapy.conf is imported *before* calling scrapy.cmdline.execute(), such as in custom bot runners 2012-09-19 12:44:59 -03:00
Pablo Hoffman
768a483953 merge adding missing changes from previous commit 2012-09-19 03:19:03 -03:00
Pablo Hoffman
c7f8219901 - removed scrapy.conf singleton from scrapy.log, scrapy.responsetypes, 
scrapy.http.response.text, scrapy.selector
- fixed bug with scrapy.conf.settings backwards compatibility support
- added facility to notify (and provide some guidelines) about deprecated/obsolete settings
 2012-09-19 03:03:34 -03:00
Pablo Hoffman
aadf7ff143 - removed scrapy.conf singleton from scrapy.log, scrapy.responsetypes, 
scrapy.http.response.text, scrapy.selector
- fixed bug with scrapy.conf.settings backwards compatibility support
- added facility to notify (and provide some guidelines) about deprecated/obsolete settings
 2012-09-19 02:56:38 -03:00
Pablo Hoffman
391cc06063 Merge pull request #174 from stav/master 
Selector documentation typo fixes
 2012-09-18 11:19:23 -07:00
stav
303e13f616 selector documentation typos 2012-09-18 12:56:52 -05:00
Pablo Hoffman
3d736e657f fixed typo in doc 2012-09-18 10:51:01 -03:00
Pablo Hoffman
eed6eb49da make DBM the new default storage backend for HTTP cache middleware, simplified DBM storage backend code to avoid dealing with many spiders at once (not needed), and update httpcache stats names (hit -> hits, miss -> misses) 2012-09-17 10:11:07 -03:00
Pablo Hoffman
8f2dda12cc removed another instance of scrapy.conf.settings singleton, this time from scrapy.utils.trackref. From now on, trackrefs functionality will be always enabled as it imposes a very minimal performance overhead 2012-09-16 21:21:44 -03:00
Pablo Hoffman
cd823018aa removed another instance of scrapy.conf.settings singleton, this time from scrapy.cmdline (which manages scrapy command line tool), by moving the get_project_settings() function to scrapy.utils.project 2012-09-16 20:53:52 -03:00
Pablo Hoffman
9685c24059 restored stats_spider_{opened,closing,closed} signals in backwards compatibility mode 2012-09-14 12:38:00 -03:00
Pablo Hoffman
81ed2d2d0b major Stats Collection refactoring: removed separation of global/per-spider stats, removed stats-related signals (stats_spider_opened, etc). Stats are much simpler now, backwards compatibility is kept on the Stats Collector API. 2012-09-14 12:31:33 -03:00
Pablo Hoffman
8b48420559 better names for scheduler stats 2012-09-13 20:07:11 -03:00
Pablo Hoffman
263e302b6a added more scheduler stats 2012-09-13 19:46:45 -03:00
Pablo Hoffman
d6867f7984 removed sphinx warnings about duplicate reference names 'this page' 2012-09-13 15:26:45 -03:00
Pablo Hoffman
a874964ad4 renamed 'XPath Selectors' title to just 'Selectors' 2012-09-13 15:24:44 -03:00
Pablo Hoffman
acb8895e1a changed note in scrapyd doc to use sphinx notes 2012-09-13 15:22:59 -03:00