BackTrackZ/scrapy
BackTrackZ/scrapy
1
0
Fork 0
mirror of https://github.com/scrapy/scrapy.git synced 2025-02-24 07:03:52 +00:00
Code Releases Wiki Activity
6,008 Commits 39 Branches 119 Tags
Commit Graph

571 Commits

Author SHA1 Message Date
Julia Medina
6d1205063c Add a filter to replace '__name__' loggers with 'scrapy' 2015-04-22 17:24:41 -03:00
Julia Medina
c174d78f12 Deprecate scrapy/log.py 2015-04-22 17:24:40 -03:00
Julia Medina
6acb3848fb Stdout redirect in configure_logging 2015-04-22 17:24:40 -03:00
Julia Medina
1c8708eb82 Create a logger for every Spider and adapt Spider.log to log through it 2015-04-22 17:24:40 -03:00
Julia Medina
ac40ef611a Custom handler to count log level occurrences in a crawler 2015-04-22 17:24:40 -03:00
Julia Medina
b75556ef79 Add a logging filter to mimic Twisted's log.err formating for Failures 2015-04-22 17:24:40 -03:00
Julia Medina
6f9b423215 Restructure LogFormatter to comply with std logging calls 2015-04-22 17:24:40 -03:00
Julia Medina
c2d716807a Use LogCapture in testfixtures package for tests 
This allows to remove `get_testlog` helper, `flushLoggedErrors` from
twisted.trial.unittest.TestCase and Twisted log observers created for
each test on conftest.py.
 2015-04-22 17:24:40 -03:00
Julia Medina
7a958f90be Replace scrapy.log calls for their equivalents in the logging std module 
Changes:
 - Each module takes 'scrapy' logger and logs through it
 - Lazy string evaluation in all log messages
 - Added missing log messages in scrapy/core/engine.py
 - Contextual data such as crawler or spider instances, and failures
 2015-04-22 17:24:39 -03:00
Daniel Graña
571bf68d7d Merge pull request #353 from nramirezuy/item-multi_inherit 
[MRG] Item multi inheritance fix
 2015-04-22 05:37:41 -03:00
nramirezuy
7871acd665 Item multi inherit fixed 2015-04-21 18:10:52 -03:00
Mikhail Korobov
c3d3a94914 fixed backwards compatibility for SPIDER_MANAGER_CLASS option 2015-04-22 00:29:40 +05:00
Pablo Hoffman
0a5bbbaed3 Merge pull request #1159 from scrapy/feed-export-fields 
[MRG+1] FEED_EXPORT_FIELDS option
 2015-04-21 15:54:24 -03:00
Daniel Graña
e034947b54 Merge pull request #1166 from scrapy/spider-loader 
[MRG+1] rename SpiderManager to SpiderLoader
 2015-04-21 15:42:17 -03:00
Pablo Hoffman
06e1ca9ef2 Merge pull request #1177 from Curita/remove-djangoitem 
Remove djangoitem
 2015-04-21 15:35:47 -03:00
Nikolaos-Digenis Karagiannis
017fb25bee loader test with processors that use item's values 2015-04-20 11:04:31 +03:00
Julia Medina
ffc60910aa Remove djangoitem since we moved it to scrapy/scrapy-djangoitem 2015-04-19 13:09:25 -03:00
Mikhail Korobov
ad587ea792 rename CrawlerRunner.spiders to CrawlerRunner.spider_loader 2015-04-17 01:54:26 +05:00
Mikhail Korobov
403e7c7c70 rename scrapy.spidermanager.SpiderManager to scrapy.spiderloader.SpiderLoader 2015-04-16 20:07:53 +05:00
mrpandav
954c8fcecb changes test_from_response_formname_notexists_fallback_formid unit test for pull request #1137 - addition of new shortcut for html form election by formid attribute 2015-04-15 16:49:09 +05:30
mrpandav
020a32a3d9 Adding tests for pull request #1137 - addition of new shortcut for html form election by formid attribute 2015-04-15 11:23:25 +05:30
Julia Medina
526aa07f44 Merge pull request #1156 from kmike/crawlers-accept-dicts 
allow Crawler, CrawlerRunner and CrawlerProcess to accept dicts instead of Setting objects
 2015-04-15 02:15:32 -03:00
Daniel Graña
c013baa602 Merge pull request #1161 from scrapy/telnet-disable 
disable scrapy.telnet if twisted.conch is not available
 2015-04-15 00:32:42 -03:00
Mikhail Korobov
0b76487629 PY3 fix Python 3 syntax errors in ajaxcrawl.py 
Tests for _has_ajaxcrawlable_meta passin Python 3.
 2015-04-15 04:16:26 +05:00
Mikhail Korobov
378b6efc0e PY3 disable scrapy.telnet if twisted.conch is not available 2015-04-15 04:06:36 +05:00
Mikhail Korobov
abeb8e3a01 TST skip a test in Python 3 because it imports some parts of Twisted which are not ported yet 2015-04-15 03:27:11 +05:00
Mikhail Korobov
647eeaea61 TST add a test for GH-1050. 2015-04-15 03:23:30 +05:00
Mikhail Korobov
1534e8540b FEED_EXPORT_FIELDS option 2015-04-15 02:48:26 +05:00
Mikhail Korobov
7644813286 allow Crawler, CrawlerRunner and CrawlerProcess to accept dicts instead of Setting objects 2015-04-14 23:26:05 +05:00
Mikhail Korobov
973c31f735 TST cleanup: use assertIn instead of assert_ 2015-04-14 20:11:46 +05:00
Pablo Hoffman
71c0aface7 Merge pull request #1148 from Curita/verify-spidermanager-interface 
Verify SPIDER_MANAGER_CLASS interface while loading it in CrawlerRunner
 2015-04-13 14:25:32 -03:00
Jakob de Maeyer
cf9d848f0f Fixed deprecated settings API in cmdline test 2015-04-12 14:18:47 +02:00
Nikolaos-Digenis Karagiannis
bf301b7328 encode invalid xpath with unicode_escape under PY2 
The exception quotes an xpath string
which may be unicode.
 2015-04-10 22:23:40 +03:00
Julia Medina
24a07fd892 Verify SPIDER_MANAGER_CLASS interface in CrawlerRunner 2015-04-10 13:43:33 -03:00
nyov
6d48c19acd dissolve scrapy.contrib_exp 2015-04-07 21:20:48 +00:00
Pablo Hoffman
4b11501eb5 Merge pull request #963 from tpeng/fix-xmliter-lxml 
[MRG+1] support namespace prefix in xmliter_lxml
 2015-04-07 18:13:01 -03:00
Daniel Graña
9706119336 Merge pull request #1128 from Curita/move-spider-settings-population-to-crawler 
[MRG+1] Move spider settings population from CrawlerRunner to Crawler.__init__
 2015-04-04 18:08:05 -03:00
Julia Medina
6f9265b44b Move spider settings population to Crawler.__init__ 2015-04-01 23:29:10 -03:00
nyov
dc88be75e2 more python3 compatibility changes for urllib 2015-04-01 19:50:56 +00:00
Daniel Graña
ec4251afd7 Merge pull request #1089 from drack3800/master 
[MRG+1] Add test for webclient with POST method and no body given
 2015-03-31 17:30:08 -03:00
Pablo Hoffman
bb4c922d85 Merge pull request #1081 from scrapy/dict-items 
Allow spiders to return dicts.
 2015-03-27 15:19:27 -03:00
nyov
84edc2ebc7 Add Response.urljoin() testcase 
and add evaluation of base-url for HtmlResponse.
 2015-03-27 05:59:01 +00:00
drack3800
1b6d5a011a Added webclient test for checking Content-Length header in response for POST request with no given body 2015-03-21 04:02:51 +03:00
Mateusz Golewski
f92bc09bf4 Add re_first() to SelectorList and iflatten() to utils.python 2015-03-18 21:11:08 -03:00
Mateusz Golewski
2742b4d8c2 Add tests to extract_first() 2015-03-18 21:11:08 -03:00
Nicolás Alejandro Ramírez Quiros
ee82fe0e24 Merge pull request #1016 from SudShekhar/jsonProcessor 
[MRG+1] Added JmesSelect
 2015-03-18 08:11:25 -03:00
Mikhail Korobov
39635e5f55 Allow spiders to return dicts. See GH-1064. 2015-03-18 07:26:56 +05:00
Pablo Hoffman
934584a355 Merge pull request #1020 from jojje/gzip_http_cache 
[MRG+1] add gzip compression to filesystem http cache backend
 2015-03-17 14:32:06 -03:00
Pablo Hoffman
f924567591 Merge pull request #983 from ArturGaspar/linkextractor_css 
[MRG+1] CSS support in link extractors
 2015-03-17 01:07:47 -03:00
nramirezuy
c13e23641b httpcache dont_cache meta #19 #689 2015-03-16 11:50:04 -03:00