BackTrackZ/scrapy
BackTrackZ/scrapy
1
0
Fork 0
mirror of https://github.com/scrapy/scrapy.git synced 2025-02-28 19:17:40 +00:00
Code Releases Wiki Activity
5,141 Commits 39 Branches 119 Tags
Commit Graph

289 Commits

Author SHA1 Message Date
nramirezuy
773ea5a5ef item fields defined on fields class attribute fix 2015-05-13 15:51:23 -03:00
Julia Medina
d3f576a816 Move scrapy/spider.py to scrapy/spiders/__init__.py 2015-05-09 04:20:09 -03:00
bosnj
8ae05478be added docs and test case, fixed handling empty string vs None 2015-05-04 21:22:17 +02:00
Victoria Terenina (torymur)
667864ae94 testing robotstxt error 2015-04-30 16:28:57 +03:00
Julia Medina
b827097c91 Rename scrapy/statscol.py to scrapy/statscollectors.py 2015-04-29 21:36:40 -03:00
Julia Medina
9a3e3ba505 Move scrapy/contrib remaining top-level files to scrapy/extensions 2015-04-29 21:27:19 -03:00
Julia Medina
54a4ce069c Rename scrapy/dupefilter.py to scrapy/dupefilters.py 2015-04-29 21:27:19 -03:00
Julia Medina
fc346cba4d Move scrapy/contrib/spiders to scrapy/spiders 2015-04-29 21:27:19 -03:00
Julia Medina
f5bdf64f9c Rename scrapy/squeue.py to scrapy/squeues.py 2015-04-29 21:27:19 -03:00
Julia Medina
645cdcbf9e Rename scrapy/loader/processor.py to scrapy/loader/processors.py 2015-04-29 21:27:19 -03:00
Julia Medina
180272c092 Move scrapy/contrib/spidermiddleware to scrapy/spidermiddlewares 2015-04-29 21:26:35 -03:00
Julia Medina
8021df18d4 Move scrapy/contrib/pipeline to scrapy/pipelines 2015-04-29 21:26:35 -03:00
Julia Medina
b47228ada8 Move scrapy/contrib/loader to scrapy/loader 2015-04-29 21:24:30 -03:00
Julia Medina
cf064b1437 Move scrapy/contrib/linkextractors to scrapy/linkextractors 2015-04-29 21:24:30 -03:00
Julia Medina
7804b3d778 Move scrapy/contrib/exporter to scrapy/exporters 2015-04-29 21:24:30 -03:00
Julia Medina
d7c444fefb Move scrapy/contrib/downloadermiddleware to scrapy/downloadermiddlewares 2015-04-29 21:24:30 -03:00
Mikhail Korobov
fbb1078f58 Merge pull request #1060 from Curita/python-logging 
[MRG+1] Python logging
 2015-04-29 23:20:34 +05:00
Julia Medina
cc4c31e426 Relax SpiderLoader interface check 2015-04-23 15:08:04 -03:00
Julia Medina
6d1205063c Add a filter to replace '__name__' loggers with 'scrapy' 2015-04-22 17:24:41 -03:00
Julia Medina
c174d78f12 Deprecate scrapy/log.py 2015-04-22 17:24:40 -03:00
Julia Medina
6acb3848fb Stdout redirect in configure_logging 2015-04-22 17:24:40 -03:00
Julia Medina
1c8708eb82 Create a logger for every Spider and adapt Spider.log to log through it 2015-04-22 17:24:40 -03:00
Julia Medina
ac40ef611a Custom handler to count log level occurrences in a crawler 2015-04-22 17:24:40 -03:00
Julia Medina
b75556ef79 Add a logging filter to mimic Twisted's log.err formating for Failures 2015-04-22 17:24:40 -03:00
Julia Medina
6f9b423215 Restructure LogFormatter to comply with std logging calls 2015-04-22 17:24:40 -03:00
Julia Medina
c2d716807a Use LogCapture in testfixtures package for tests 
This allows to remove `get_testlog` helper, `flushLoggedErrors` from
twisted.trial.unittest.TestCase and Twisted log observers created for
each test on conftest.py.
 2015-04-22 17:24:40 -03:00
Julia Medina
7a958f90be Replace scrapy.log calls for their equivalents in the logging std module 
Changes:
 - Each module takes 'scrapy' logger and logs through it
 - Lazy string evaluation in all log messages
 - Added missing log messages in scrapy/core/engine.py
 - Contextual data such as crawler or spider instances, and failures
 2015-04-22 17:24:39 -03:00
Daniel Graña
571bf68d7d Merge pull request #353 from nramirezuy/item-multi_inherit 
[MRG] Item multi inheritance fix
 2015-04-22 05:37:41 -03:00
nramirezuy
7871acd665 Item multi inherit fixed 2015-04-21 18:10:52 -03:00
Mikhail Korobov
c3d3a94914 fixed backwards compatibility for SPIDER_MANAGER_CLASS option 2015-04-22 00:29:40 +05:00
Pablo Hoffman
0a5bbbaed3 Merge pull request #1159 from scrapy/feed-export-fields 
[MRG+1] FEED_EXPORT_FIELDS option
 2015-04-21 15:54:24 -03:00
Daniel Graña
e034947b54 Merge pull request #1166 from scrapy/spider-loader 
[MRG+1] rename SpiderManager to SpiderLoader
 2015-04-21 15:42:17 -03:00
Pablo Hoffman
06e1ca9ef2 Merge pull request #1177 from Curita/remove-djangoitem 
Remove djangoitem
 2015-04-21 15:35:47 -03:00
Nikolaos-Digenis Karagiannis
017fb25bee loader test with processors that use item's values 2015-04-20 11:04:31 +03:00
Julia Medina
ffc60910aa Remove djangoitem since we moved it to scrapy/scrapy-djangoitem 2015-04-19 13:09:25 -03:00
Mikhail Korobov
ad587ea792 rename CrawlerRunner.spiders to CrawlerRunner.spider_loader 2015-04-17 01:54:26 +05:00
Mikhail Korobov
403e7c7c70 rename scrapy.spidermanager.SpiderManager to scrapy.spiderloader.SpiderLoader 2015-04-16 20:07:53 +05:00
mrpandav
954c8fcecb changes test_from_response_formname_notexists_fallback_formid unit test for pull request #1137 - addition of new shortcut for html form election by formid attribute 2015-04-15 16:49:09 +05:30
mrpandav
020a32a3d9 Adding tests for pull request #1137 - addition of new shortcut for html form election by formid attribute 2015-04-15 11:23:25 +05:30
Julia Medina
526aa07f44 Merge pull request #1156 from kmike/crawlers-accept-dicts 
allow Crawler, CrawlerRunner and CrawlerProcess to accept dicts instead of Setting objects
 2015-04-15 02:15:32 -03:00
Daniel Graña
c013baa602 Merge pull request #1161 from scrapy/telnet-disable 
disable scrapy.telnet if twisted.conch is not available
 2015-04-15 00:32:42 -03:00
Mikhail Korobov
0b76487629 PY3 fix Python 3 syntax errors in ajaxcrawl.py 
Tests for _has_ajaxcrawlable_meta passin Python 3.
 2015-04-15 04:16:26 +05:00
Mikhail Korobov
378b6efc0e PY3 disable scrapy.telnet if twisted.conch is not available 2015-04-15 04:06:36 +05:00
Mikhail Korobov
abeb8e3a01 TST skip a test in Python 3 because it imports some parts of Twisted which are not ported yet 2015-04-15 03:27:11 +05:00
Mikhail Korobov
647eeaea61 TST add a test for GH-1050. 2015-04-15 03:23:30 +05:00
Mikhail Korobov
1534e8540b FEED_EXPORT_FIELDS option 2015-04-15 02:48:26 +05:00
Mikhail Korobov
7644813286 allow Crawler, CrawlerRunner and CrawlerProcess to accept dicts instead of Setting objects 2015-04-14 23:26:05 +05:00
Mikhail Korobov
973c31f735 TST cleanup: use assertIn instead of assert_ 2015-04-14 20:11:46 +05:00
Pablo Hoffman
71c0aface7 Merge pull request #1148 from Curita/verify-spidermanager-interface 
Verify SPIDER_MANAGER_CLASS interface while loading it in CrawlerRunner
 2015-04-13 14:25:32 -03:00
Jakob de Maeyer
cf9d848f0f Fixed deprecated settings API in cmdline test 2015-04-12 14:18:47 +02:00