BackTrackZ/scrapy
BackTrackZ/scrapy
1
0
Fork 0
mirror of https://github.com/scrapy/scrapy.git synced 2025-02-25 16:43:47 +00:00
Code Releases Wiki Activity
3,483 Commits 39 Branches 119 Tags
Commit Graph

3483 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Pablo Hoffman
9feb65865c Merge pull request #284 from nramirezuy/cmd-parse-pipelines 
Command parse, --pipelines argument added
 2013-04-09 06:26:24 -07:00
Pablo Hoffman
adf38a65e9 Merge pull request #283 from opyate/patch-1 
Update overview.rst, Torrent referenced as TorrentItem in spider
 2013-04-08 13:48:05 -07:00
Nicolás Ramírez
2b39527f72 pipelines argument added 2013-04-08 14:55:28 -03:00
Juan M Uys
4de3aa4932 Update overview.rst 2013-04-08 14:13:15 +02:00
Pablo Hoffman
96c2332e0e fix inaccurate downloader middleware documentation. refs #280 2013-04-02 11:35:32 -03:00
Pablo Hoffman
b0ea457c7c Merge pull request #277 from nramirezuy/cmd-parse-args 
Spider Arguments support for parse command
 2013-03-28 15:38:35 -07:00
Nicolás Ramírez
df19693ed2 Spider Arguments support for parse command and test 2013-03-28 16:49:06 -03:00
Steven Almeroth
70179c7c0c doc: remove trailing spaces 2013-03-21 13:57:39 -06:00
Steven Almeroth
0d7747d353 doc: Response.replace() cannot take meta argument 
>>> response.replace(meta={'foo':1})
Traceback (most recent call last):
  File "<input>", line 1, in <module>
  File "/srv/scrapy/scrapy-fork/scrapy/scrapy/http/response/text.py", line 45, in replace
    return Response.replace(self, *args, **kwargs)
  File "/srv/scrapy/scrapy-fork/scrapy/scrapy/http/response/__init__.py", line 77, in replace
    return cls(*args, **kwargs)
  File "/srv/scrapy/scrapy-fork/scrapy/scrapy/http/response/text.py", line 22, in __init__
    super(TextResponse, self).__init__(*args, **kwargs)
TypeError: __init__() got an unexpected keyword argument 'meta'
 2013-03-21 13:49:55 -06:00
Pablo Hoffman
21c8b89422 Revert "replaced use of depricated module scrapy.settings with the method get_project_settings()" 
This reverts commit 1b4d14c8f635b28d5f72d37f924c4e71d71520ca.

Calling `get_project_settings()` generates a new independent Settings
object that doesn't contain the overrides passed by command line
arguments, for example.

Proper port would require implementing the from_crawler() class method
and making sure the settings object is passed to all internal objects
(probably breaking some minor backwards compatibility).
 2013-03-21 12:27:07 -03:00
Pablo Hoffman
8c181d87f6 Merge pull request #274 from brunsgaard/master 
Updated settings import in contrib/feedexport.py class S3FeedStorage
 2013-03-21 08:25:29 -07:00
Jonas Brunsgaard
1b4d14c8f6 replaced use of depricated module scrapy.settings with the method get_project_settings() 2013-03-21 16:09:14 +01:00
Pablo Hoffman
d0a81d369f initial version of crawl tests using a mock HTTP server (in separate process). This can also be used to benchmark scrapy performance, although a script (specially suited for that task) would be more convenient 2013-03-20 14:48:59 -03:00
Pablo Hoffman
2a5c7ed4da make Crawler.start() return a deferred that is fired when the crawl is finished 2013-03-20 14:48:59 -03:00
Daniel Graña
c43931ea8c Use latest pyOpenSSL for all travis tests environments 
SSLv2 was removed from OpenSSL 1.0 and above but it is still referenced
by pyOpenSSL < 0.13. Travis workers are precise hosts with OpenSSL 1.0
and pyOpenSSL 0.12 (!) with a debian patch to workaround this problem
that is not present in pyOpenSSL 0.12 shipped by PyPi.

Trying to install pyOpenSSL 0.10 or 0.12 from packages at PyPi under a
system with OpenSSL >= 1.0 will success but fails at import time with a
message similar to:

    ImportError: .../lib/python2.7/site-packages/OpenSSL/SSL.so: undefined symbol: SSLv2_method
 2013-03-20 11:44:48 -03:00
Pablo Hoffman
9968f99e06 remove ssl from optional_features to simplify code, as it is now required. also deprecate optional_features set 2013-03-20 09:52:40 -03:00
Pablo Hoffman
320bdfe391 Merge pull request #269 from kalessin/settingdict 
added support for explicitly interpret a setting value as dict
 2013-03-19 11:10:54 -07:00
Martin Olveyra
bf480015f4 added support generic python literals in settings, and for explicitly 
interpret a setting value as dict
 2013-03-19 16:02:53 -02:00
Pablo Hoffman
d246b926bf Merge pull request #273 from plainas/master 
Accept ajax requests from other hosts (CORS support)
 2013-03-19 07:27:56 -07:00
Pedro
a80ed769d9 allow remote ajax requests to the webservice 2013-03-19 12:00:49 +01:00
Pablo Hoffman
b347c14b5f update engine status output on telnet console documentation 2013-03-18 19:12:12 -03:00
Pablo Hoffman
6f9f6f1f16 Merge pull request #271 from nramirezuy/scraper-6017 
Slots removed, now Scraper can handle just one spider
 2013-03-18 15:05:34 -07:00
Shane Evans
5c2a82f1f7 fix typo 2013-03-17 19:34:55 +00:00
Nicolás Ramírez
58975abeab Slots removed, now Scraper can handle just one spider 2013-03-15 11:11:51 -03:00
Pablo Hoffman
e630126b82 scrapy deploy: return non-zero exit code if deploy fails 2013-03-14 16:47:35 -03:00
Pablo Hoffman
bb20907254 minor updated to faq 2013-03-14 16:43:00 -03:00
Pablo Hoffman
098ccff862 added FAQ about error: "cannot import name crawler" 2013-03-14 12:57:59 -03:00
Pablo Hoffman
a862f23376 added Nicolas Ramirez to AUTHORS 2013-03-14 12:44:39 -03:00
Pablo Hoffman
46b305ef7d Merge pull request #267 from nramirezuy/formrequest 
Override Request 'method' in FormRequest
 2013-03-14 08:43:53 -07:00
Nicolás Ramírez
f043bba049 Override request method in FormRequest 
Changes proposed in the comments
 2013-03-14 10:39:17 -03:00
Pablo Hoffman
a2e9d031f9 removed duplicated test 2013-03-14 10:32:19 -03:00
Pablo Hoffman
8391b36251 minor updates to contributing doc 2013-03-13 03:24:25 -03:00
Pablo Hoffman
51c301b3a2 added link to python binary libs, for windows installation 2013-03-13 03:18:33 -03:00
Pablo Hoffman
8e72730792 Merge pull request #261 from stav/allowed_domains 
allow spider allowed_domains to be set/tuple, #259
 2013-03-12 20:44:51 -07:00
Pablo Hoffman
296db1dc09 log (just once) when duplicate requests are filtered out. closes #105, #249 2013-03-12 19:28:43 -03:00
Pablo Hoffman
a4507f62c3 sep-019: fixed typo 2013-03-12 18:44:09 -03:00
Steven Almeroth
650eda68da doc: add comment about commit history cleanliness 2013-03-10 18:51:04 -06:00
Steven Almeroth
5828179c5f remove over-testing and dict testing for test_utils_url.py 2013-03-10 18:30:49 -06:00
Steven Almeroth
1514b3b5db pylint clean-ups for test_utils_url.py 2013-03-10 16:47:24 -06:00
Steven Almeroth
a613e15154 add tests for url_is_from_spider() with allowed_domains 2013-03-10 16:41:27 -06:00
Pablo Hoffman
b2b256dc37 added nicolas to sep-019 authors, changed main title formatting for sep-019 and sep-020 2013-03-10 14:22:08 -03:00
Pablo Hoffman
b11a1326b9 Merge pull request #264 from stav/sep 
add new sep-20 and update sep-19 with table at the top
 2013-03-08 11:23:15 -08:00
Steven Almeroth
e66db6fa56 add new sep-20 and update sep-19 with table at the top 2013-03-08 13:04:11 -06:00
Pablo Hoffman
eeb69d2f70 added #260 to release notes 2013-03-08 11:59:38 -02:00
Pablo Hoffman
42499c72b1 updated AUTHORS 2013-03-08 11:57:54 -02:00
Pablo Hoffman
b5b37197bf Merge pull request #260 from llonchj/entry_point 
Commands module
 2013-03-08 05:49:08 -08:00
Pablo Hoffman
b560f4d1c1 sep-019: remove TODO section (settled down with class method) 2013-03-08 09:57:12 -02:00
Pablo Hoffman
a44757179b sep-019: other minor fixes 2013-03-07 19:38:12 -02:00
Pablo Hoffman
c7add1e5bd sep-019: fix typos 2013-03-07 19:24:44 -02:00
Pablo Hoffman
d806acdf50 sep-019: minor tide up 2013-03-07 12:39:40 -02:00