BackTrackZ/scrapy
BackTrackZ/scrapy
1
0
Fork 0
mirror of https://github.com/scrapy/scrapy.git synced 2025-02-25 11:43:45 +00:00
Code Releases Wiki Activity
4,785 Commits 39 Branches 119 Tags
Commit Graph

151 Commits

Author SHA1 Message Date
Mikhail Korobov
d047665c02 make "settings" argument optional for Crawler, CrawlerRunner and CrawlerProcess 2015-06-06 03:23:13 +05:00
Mikhail Korobov
cc2f3e1b46 TST a test case to show custom_settings doesn't always work 2015-06-03 04:26:20 +05:00
Julia Medina
ffc7b7fd6c Add helper to update deprecated class paths 2015-06-01 17:01:33 -03:00
Mikhail Korobov
aa6a72707d fixed CrawlerProcess when settings are passed as dicts 
See https://github.com/scrapy/scrapy/pull/1156
 2015-05-30 06:59:15 +05:00
Mikhail Korobov
cc2258b2bb Merge pull request #1145 from bosnj/master 
[MRG+1] default return value for extract_first
 2015-05-21 22:03:54 +05:00
Daniel Graña
ee59112480 Merge pull request #1224 from scrapy/fix-empty-feed-export-fields 
[MRG] fixed FEED_EXPORT_FIELDS handling (see #1223)
 2015-05-19 16:36:05 -03:00
Mikhail Korobov
7a5b5ec4d6 TST remove unnecessary lines from py3-ignores 
scrapy/contrib is already skipped - see https://github.com/scrapy/scrapy/pull/1165
 2015-05-19 00:57:39 +05:00
Mikhail Korobov
9b0ca1b7a0 drop support for FEED_EXPORT_FIELD=[] meaning "no fields" 2015-05-18 17:13:25 +05:00
Mikhail Korobov
9fb318338b support FEED_EXPORT_FIELDS=[] 2015-05-18 16:44:02 +05:00
Mikhail Korobov
e1efd19175 TST, DOC document that Scrapy only infers field names for CSV 2015-05-18 16:43:23 +05:00
Mikhail Korobov
8d9e3b7e8d TST add a test for JSON lines item exporter 2015-05-18 16:43:23 +05:00
Julia Medina
6dccb3a9b3 Replace FailureFormatter with direct exc_info conversions in log calls 2015-05-15 05:46:14 -03:00
Julia Medina
157cec7f64 Replace logger.exception calls for Python compatibility 2015-05-15 01:26:45 -03:00
Daniel Graña
5e5e44e91e Merge pull request #1228 from nramirezuy/item_fields_defined_on_fields 
item fields defined on fields class attribute fix
 2015-05-14 23:09:24 -03:00
nramirezuy
773ea5a5ef item fields defined on fields class attribute fix 2015-05-13 15:51:23 -03:00
Julia Medina
d3f576a816 Move scrapy/spider.py to scrapy/spiders/__init__.py 2015-05-09 04:20:09 -03:00
bosnj
8ae05478be added docs and test case, fixed handling empty string vs None 2015-05-04 21:22:17 +02:00
Victoria Terenina (torymur)
667864ae94 testing robotstxt error 2015-04-30 16:28:57 +03:00
Julia Medina
b827097c91 Rename scrapy/statscol.py to scrapy/statscollectors.py 2015-04-29 21:36:40 -03:00
Julia Medina
54a4ce069c Rename scrapy/dupefilter.py to scrapy/dupefilters.py 2015-04-29 21:27:19 -03:00
Julia Medina
f5bdf64f9c Rename scrapy/squeue.py to scrapy/squeues.py 2015-04-29 21:27:19 -03:00
Julia Medina
645cdcbf9e Rename scrapy/loader/processor.py to scrapy/loader/processors.py 2015-04-29 21:27:19 -03:00
Julia Medina
9a3e3ba505 Move scrapy/contrib remaining top-level files to scrapy/extensions 2015-04-29 21:27:19 -03:00
Julia Medina
fc346cba4d Move scrapy/contrib/spiders to scrapy/spiders 2015-04-29 21:27:19 -03:00
Julia Medina
180272c092 Move scrapy/contrib/spidermiddleware to scrapy/spidermiddlewares 2015-04-29 21:26:35 -03:00
Julia Medina
8021df18d4 Move scrapy/contrib/pipeline to scrapy/pipelines 2015-04-29 21:26:35 -03:00
Julia Medina
b47228ada8 Move scrapy/contrib/loader to scrapy/loader 2015-04-29 21:24:30 -03:00
Julia Medina
cf064b1437 Move scrapy/contrib/linkextractors to scrapy/linkextractors 2015-04-29 21:24:30 -03:00
Julia Medina
7804b3d778 Move scrapy/contrib/exporter to scrapy/exporters 2015-04-29 21:24:30 -03:00
Julia Medina
d7c444fefb Move scrapy/contrib/downloadermiddleware to scrapy/downloadermiddlewares 2015-04-29 21:24:30 -03:00
Mikhail Korobov
fbb1078f58 Merge pull request #1060 from Curita/python-logging 
[MRG+1] Python logging
 2015-04-29 23:20:34 +05:00
Julia Medina
cc4c31e426 Relax SpiderLoader interface check 2015-04-23 15:08:04 -03:00
Julia Medina
6d1205063c Add a filter to replace '__name__' loggers with 'scrapy' 2015-04-22 17:24:41 -03:00
Julia Medina
c174d78f12 Deprecate scrapy/log.py 2015-04-22 17:24:40 -03:00
Julia Medina
6acb3848fb Stdout redirect in configure_logging 2015-04-22 17:24:40 -03:00
Julia Medina
1c8708eb82 Create a logger for every Spider and adapt Spider.log to log through it 2015-04-22 17:24:40 -03:00
Julia Medina
ac40ef611a Custom handler to count log level occurrences in a crawler 2015-04-22 17:24:40 -03:00
Julia Medina
b75556ef79 Add a logging filter to mimic Twisted's log.err formating for Failures 2015-04-22 17:24:40 -03:00
Julia Medina
6f9b423215 Restructure LogFormatter to comply with std logging calls 2015-04-22 17:24:40 -03:00
Julia Medina
c2d716807a Use LogCapture in testfixtures package for tests 
This allows to remove `get_testlog` helper, `flushLoggedErrors` from
twisted.trial.unittest.TestCase and Twisted log observers created for
each test on conftest.py.
 2015-04-22 17:24:40 -03:00
Julia Medina
7a958f90be Replace scrapy.log calls for their equivalents in the logging std module 
Changes:
 - Each module takes 'scrapy' logger and logs through it
 - Lazy string evaluation in all log messages
 - Added missing log messages in scrapy/core/engine.py
 - Contextual data such as crawler or spider instances, and failures
 2015-04-22 17:24:39 -03:00
Daniel Graña
571bf68d7d Merge pull request #353 from nramirezuy/item-multi_inherit 
[MRG] Item multi inheritance fix
 2015-04-22 05:37:41 -03:00
nramirezuy
7871acd665 Item multi inherit fixed 2015-04-21 18:10:52 -03:00
Mikhail Korobov
c3d3a94914 fixed backwards compatibility for SPIDER_MANAGER_CLASS option 2015-04-22 00:29:40 +05:00
Pablo Hoffman
0a5bbbaed3 Merge pull request #1159 from scrapy/feed-export-fields 
[MRG+1] FEED_EXPORT_FIELDS option
 2015-04-21 15:54:24 -03:00
Daniel Graña
e034947b54 Merge pull request #1166 from scrapy/spider-loader 
[MRG+1] rename SpiderManager to SpiderLoader
 2015-04-21 15:42:17 -03:00
Pablo Hoffman
06e1ca9ef2 Merge pull request #1177 from Curita/remove-djangoitem 
Remove djangoitem
 2015-04-21 15:35:47 -03:00
Nikolaos-Digenis Karagiannis
017fb25bee loader test with processors that use item's values 2015-04-20 11:04:31 +03:00
Julia Medina
ffc60910aa Remove djangoitem since we moved it to scrapy/scrapy-djangoitem 2015-04-19 13:09:25 -03:00
Mikhail Korobov
ad587ea792 rename CrawlerRunner.spiders to CrawlerRunner.spider_loader 2015-04-17 01:54:26 +05:00