BackTrackZ/scrapy
BackTrackZ/scrapy
1
0
Fork 0
mirror of https://github.com/scrapy/scrapy.git synced 2025-02-26 12:23:44 +00:00
Code Releases Wiki Activity
5,567 Commits 39 Branches 119 Tags
Commit Graph

467 Commits

Author SHA1 Message Date
Mikhail Korobov
41bcae5d00 TST fix to_bytes and to_unicode tests in Python 3.x 2015-07-23 17:35:23 +02:00
Mikhail Korobov
61cd27e5c7 Rename unicode_to_str and str_to_unicode method. Fixes GH-778. 2015-07-23 15:00:06 +02:00
Jakob de Maeyer
c153217bfb Fix RedirectMiddleware test 2015-07-17 16:11:53 +02:00
Jakob de Maeyer
c908d31660 Make RedirectMiddleware respect Spider.handle_httpstatus_list 2015-07-16 12:50:26 +02:00
nyov
4f56c6c79b Unset environment proxies for tests 
because urllib doesn't handle $no_proxy correctly
and the unittest webserver is always local.
 2015-07-11 11:30:08 +00:00
Julia Medina
d706310d8b Merge pull request #1151 from marven/cache-control 
[MRG+1] RFC2616 policy enhancements + tests
 2015-07-11 08:06:20 -03:00
Julia Medina
d3d61fde87 Fix a recent mock regression by not using it in a test 2015-07-10 19:58:38 -03:00
Demelziraptor
9a15fcf89a interpreting application/x-json as TextResponse 2015-07-02 19:56:30 +02:00
Daniel Graña
07f4f12e8b Merge pull request #1313 from eliasdorneles/support-empty-passwd-for-http-proxy 
[MRG+1] Support empty password for http_proxy config
 2015-06-30 19:58:16 -03:00
Mikhail Korobov
ea4b175c2d Merge pull request #1254 from scrapy/bugfix-1232 
[MRG +1] Unquote request path before passing to FTPClient, it already escape paths
 2015-06-24 18:04:45 +05:00
Elias Dorneles
eb8fed4ecd support empty password for http_proxy config 2015-06-23 00:43:15 -03:00
Berker Peksag
8a48d9c6a8 Ignore ScrapyDeprecationWarning warnings properly. 
Conflicts:

	tests/test_utils_deprecate.py
 2015-06-09 18:22:03 +03:00
Mikhail Korobov
d047665c02 make "settings" argument optional for Crawler, CrawlerRunner and CrawlerProcess 2015-06-06 03:23:13 +05:00
Mikhail Korobov
cc2f3e1b46 TST a test case to show custom_settings doesn't always work 2015-06-03 04:26:20 +05:00
Julia Medina
ffc7b7fd6c Add helper to update deprecated class paths 2015-06-01 17:01:33 -03:00
Marven Sanchez
bb3ebf13f9 Add tests for RFC2616 policy enhancements 
Add `scrapy/downloadermiddlewares/httpcache.py` to `tests/py3-ignores.txt
 2015-06-01 18:20:12 +08:00
Mikhail Korobov
aa6a72707d fixed CrawlerProcess when settings are passed as dicts 
See https://github.com/scrapy/scrapy/pull/1156
 2015-05-30 06:59:15 +05:00
Daniel Graña
ebe889a663 Unquote request path before passing to FTPClient, it already escape paths 2015-05-23 20:50:30 -03:00
Mikhail Korobov
cc2258b2bb Merge pull request #1145 from bosnj/master 
[MRG+1] default return value for extract_first
 2015-05-21 22:03:54 +05:00
Daniel Graña
ee59112480 Merge pull request #1224 from scrapy/fix-empty-feed-export-fields 
[MRG] fixed FEED_EXPORT_FIELDS handling (see #1223)
 2015-05-19 16:36:05 -03:00
Mikhail Korobov
7a5b5ec4d6 TST remove unnecessary lines from py3-ignores 
scrapy/contrib is already skipped - see https://github.com/scrapy/scrapy/pull/1165
 2015-05-19 00:57:39 +05:00
Mikhail Korobov
9b0ca1b7a0 drop support for FEED_EXPORT_FIELD=[] meaning "no fields" 2015-05-18 17:13:25 +05:00
Mikhail Korobov
9fb318338b support FEED_EXPORT_FIELDS=[] 2015-05-18 16:44:02 +05:00
Mikhail Korobov
e1efd19175 TST, DOC document that Scrapy only infers field names for CSV 2015-05-18 16:43:23 +05:00
Mikhail Korobov
8d9e3b7e8d TST add a test for JSON lines item exporter 2015-05-18 16:43:23 +05:00
Julia Medina
6dccb3a9b3 Replace FailureFormatter with direct exc_info conversions in log calls 2015-05-15 05:46:14 -03:00
Julia Medina
157cec7f64 Replace logger.exception calls for Python compatibility 2015-05-15 01:26:45 -03:00
Daniel Graña
5e5e44e91e Merge pull request #1228 from nramirezuy/item_fields_defined_on_fields 
item fields defined on fields class attribute fix
 2015-05-14 23:09:24 -03:00
nramirezuy
773ea5a5ef item fields defined on fields class attribute fix 2015-05-13 15:51:23 -03:00
Julia Medina
d3f576a816 Move scrapy/spider.py to scrapy/spiders/__init__.py 2015-05-09 04:20:09 -03:00
bosnj
8ae05478be added docs and test case, fixed handling empty string vs None 2015-05-04 21:22:17 +02:00
Victoria Terenina (torymur)
667864ae94 testing robotstxt error 2015-04-30 16:28:57 +03:00
Julia Medina
b827097c91 Rename scrapy/statscol.py to scrapy/statscollectors.py 2015-04-29 21:36:40 -03:00
Julia Medina
9a3e3ba505 Move scrapy/contrib remaining top-level files to scrapy/extensions 2015-04-29 21:27:19 -03:00
Julia Medina
54a4ce069c Rename scrapy/dupefilter.py to scrapy/dupefilters.py 2015-04-29 21:27:19 -03:00
Julia Medina
fc346cba4d Move scrapy/contrib/spiders to scrapy/spiders 2015-04-29 21:27:19 -03:00
Julia Medina
f5bdf64f9c Rename scrapy/squeue.py to scrapy/squeues.py 2015-04-29 21:27:19 -03:00
Julia Medina
645cdcbf9e Rename scrapy/loader/processor.py to scrapy/loader/processors.py 2015-04-29 21:27:19 -03:00
Julia Medina
180272c092 Move scrapy/contrib/spidermiddleware to scrapy/spidermiddlewares 2015-04-29 21:26:35 -03:00
Julia Medina
8021df18d4 Move scrapy/contrib/pipeline to scrapy/pipelines 2015-04-29 21:26:35 -03:00
Julia Medina
b47228ada8 Move scrapy/contrib/loader to scrapy/loader 2015-04-29 21:24:30 -03:00
Julia Medina
cf064b1437 Move scrapy/contrib/linkextractors to scrapy/linkextractors 2015-04-29 21:24:30 -03:00
Julia Medina
7804b3d778 Move scrapy/contrib/exporter to scrapy/exporters 2015-04-29 21:24:30 -03:00
Julia Medina
d7c444fefb Move scrapy/contrib/downloadermiddleware to scrapy/downloadermiddlewares 2015-04-29 21:24:30 -03:00
Mikhail Korobov
fbb1078f58 Merge pull request #1060 from Curita/python-logging 
[MRG+1] Python logging
 2015-04-29 23:20:34 +05:00
Julia Medina
cc4c31e426 Relax SpiderLoader interface check 2015-04-23 15:08:04 -03:00
Julia Medina
6d1205063c Add a filter to replace '__name__' loggers with 'scrapy' 2015-04-22 17:24:41 -03:00
Julia Medina
c174d78f12 Deprecate scrapy/log.py 2015-04-22 17:24:40 -03:00
Julia Medina
6acb3848fb Stdout redirect in configure_logging 2015-04-22 17:24:40 -03:00
Julia Medina
1c8708eb82 Create a logger for every Spider and adapt Spider.log to log through it 2015-04-22 17:24:40 -03:00