BackTrackZ/scrapy
BackTrackZ/scrapy
1
0
Fork 0
mirror of https://github.com/scrapy/scrapy.git synced 2025-03-02 21:17:58 +00:00
Code Releases Wiki Activity
3,778 Commits 39 Branches 119 Tags
Commit Graph

3778 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Mikhail Korobov
a27d91f0a6 Rename BaseSpider to Spider. See GH-495. 2013-12-30 19:46:41 +06:00
Mikhail Korobov
439a141aae utility for showing a warning when a class is subclassed 2013-12-30 19:46:41 +06:00
Daniel Graña
74433a17e7 Merge pull request #505 from kmike/scrapy-shell-docs 
[MRG] minor fixes to scrapy shell docs
 2013-12-30 04:08:50 -08:00
Mikhail Korobov
e713733edf minor fixes to scrapy shell docs 
* better IPython links;
* MDC link instead of w3schools;
* small formatting fixes;
* show quoted URL in example
 2013-12-30 10:27:39 +06:00
Daniel Graña
f9dd2986d4 Merge pull request #503 from kmike/better-tox-ini 
[MRG] allow running individual tests via tox
 2013-12-29 14:03:29 -08:00
Mikhail Korobov
b15df73a0f allow running individual tests via tox, e.g. «tox -e trunk -- scrapy.tests.test_spider» 2013-12-29 01:59:20 +06:00
Mikhail Korobov
f18ac02987 remove duplicated link extractors link 
Check http://doc.scrapy.org/en/latest/topics/link-extractors.html - two menu items are highlighted at the left.
 2013-12-28 05:40:10 +05:00
Pablo Hoffman
99f02b11f5 Merge pull request #498 from mbacho/master 
Removing extra 'doc' extension, adding 'pptx' and 'xlsx' extensions
 2013-12-24 06:36:06 -08:00
Chomba Ng'ang'a
95dc46f2bc Removing extra 'doc' extension, adding 'pptx' and 'xlsx' extensions 2013-12-24 17:29:40 +03:00
Pablo Hoffman
e42e3743fe quick documentation for #475 2013-12-24 12:19:15 -02:00
Daniel Graña
c4cd0f45a0 reindent travis.yml as per travis gem defaults 2013-12-24 11:26:48 -02:00
Daniel Graña
e91025536f Merge pull request #475 from dangra/473-do-not-send-header 
[MRG] Do not set Referer by default when its value is None
 2013-12-24 05:20:36 -08:00
Daniel Graña
2b7fea26a5 Do not set Referer by default when its value is None 
closes #473
 2013-12-24 11:05:06 -02:00
Daniel Graña
2c2ce20878 Merge pull request #490 from max-arnold/master 
add new pipeline methods to get file/image/thumbnail paths
 2013-12-23 15:11:53 -08:00
Pablo Hoffman
80bb9fb5d2 Merge pull request #497 from kmike/utils-unique 
[MRG] no need to use dict in scrapy.utils.python.unique
 2013-12-19 05:40:34 -08:00
Mikhail Korobov
84aa7599a4 no need to use dict in scrapy.utils.python.unique 2013-12-19 15:06:27 +06:00
Mikhail Korobov
93eee9d1c6 Merge pull request #479 from citizen-stig/unicode_spider_name 
fix logging error with unicode spider name
 2013-12-18 12:17:57 -08:00
Mikhail Korobov
51f340e1b9 add distutils folders to gitignore 2013-12-18 23:29:07 +06:00
Max Arnold
87559a1240 fix missing keyword arguments for file_path in media_downloaded() 2013-12-18 23:23:01 +07:00
Max Arnold
d8ca8f83bb test for new response and info keyword arguments 2013-12-18 23:09:37 +07:00
Max Arnold
2dfeff7eca rename ImagesPipelineTestCase image_path var to file_path for clarity 2013-12-18 22:54:47 +07:00
Max Arnold
6959523338 implement DeprecatedImagesPipelineTestCase 2013-12-18 22:54:39 +07:00
Max Arnold
65c017c5fe implement DeprecatedFilesPipelineTestCase 2013-12-18 22:25:42 +07:00
Max Arnold
14b68b47d6 rename FilesPipelineTestCase image_path var to file_path for clarity 2013-12-18 22:13:49 +07:00
Max Arnold
23c50bd973 rename key to path in FilesPipelineTestCase 2013-12-18 10:52:13 +07:00
Max Arnold
d58a4639cd switch existing pipeline tests to use new file_path/image_path methods 2013-12-17 23:57:22 +07:00
Max Arnold
220ced163c simplify porting to Python 3 2013-12-17 23:19:40 +07:00
Max Arnold
86a6d6898b better marker name to detect overridden methods 2013-12-17 22:44:12 +07:00
Max Arnold
c29939862a add deprecation warnings for old file_key/image_key/thumb_key methods 2013-12-17 22:35:31 +07:00
Max Arnold
270e9190ae add new pipeline methods to get file/image/thumbnail paths 
This change allows to pass request, response and spider context to filename
construction methods.
 2013-12-17 21:50:40 +07:00
Pablo Hoffman
462e40acd0 Merge pull request #489 from RasPat1/patch-2 
Note about selector class import
 2013-12-16 06:40:59 -08:00
RasPat1
ff21281b95 Note about selector class import 
This is the salient point of this code compared to the last example.  We have a selector now and this is how we use it.  Especially since the user has just come from the shell where the pre-instantiated selector is taken for granted.
 2013-12-15 13:46:42 -05:00
Daniel Graña
8a7c5b5d81 Add 0.20.2 release notes 
Conflicts:
	docs/news.rst
 2013-12-09 18:33:46 -02:00
Daniel Graña
11359188a5 Merge pull request #484 from nyov/nyov/crawl-tmpl-selector 
Update CrawlSpider Template with Selector changes
 2013-12-09 12:21:27 -08:00
nyov
b6a200d02a Update CrawlSpider Template with Selector changes 2013-12-06 20:48:25 +00:00
Daniel Graña
72543c9ef0 Merge pull request #397 from duendex/duendex/proxyTunnel 
Adds the functionality to do HTTPS downloads behind proxies using an
 2013-12-03 14:48:42 -08:00
duendex
8ada8f5f36 Added a test case to ensure that passing the noconnect paramenter avoids trigerring the creation of a connect tunnel when downloading from a site with https scheme. 2013-12-03 12:55:44 -02:00
duendex
6427d60fd8 Fixed the location of the certificate required by libmproxy. 2013-12-03 11:45:02 -02:00
duendex
500490ee73 Corrected a test that used a dummy URL that unpurposedly had an https scheme and failed with PR 397. 2013-12-03 03:10:16 -02:00
duendex
f8dea74948 Added a delay to wait for the proxy to start. 2013-12-03 03:09:06 -02:00
duendex
247b330f08 Corrected typo in tox.ini 2013-12-02 21:34:35 -02:00
duendex
02bab270e8 Added mitmproxy as a requirement. 2013-12-02 20:25:12 -02:00
duendex
d69ba7c1ae Changed the proxy tests to use libmproxy instead of starting mitmdump as a separate process. 2013-12-02 20:21:43 -02:00
duendex
88bec496f2 The response matching re is now compiled once at module load time. 2013-12-02 20:19:42 -02:00
duendex
23c3288a6d Adds the option to omit the usage of a CONNECT tunnel by adding the noconnect 
parameter to the URL of the proxy.
 2013-12-02 20:19:42 -02:00
duendex
7f053cc1d2 Adds support for proxy authentication when openning a CONNECT tunnel. 2013-12-02 20:19:42 -02:00
duendex
628bfbcc3e Raises a custom TunnelError when the tunnel cannot be opened. Removed unnecesary comments. 2013-12-02 20:19:42 -02:00
duendex
58a98b0c04 Improved error handling. 2013-12-02 20:19:42 -02:00
duendex
36e4fc3785 Removed some trailing spaces that I left. 2013-12-02 20:19:42 -02:00
duendex
ae28c7d698 Adds the functionality to do HTTPS downloads behind proxies using an 
HTTP CONNECT.
 2013-12-02 20:19:41 -02:00