BackTrackZ/scrapy
BackTrackZ/scrapy
1
0
Fork 0
mirror of https://github.com/scrapy/scrapy.git synced 2025-02-22 23:24:11 +00:00
Code Releases Wiki Activity
7,530 Commits 39 Branches 119 Tags
Commit Graph

58 Commits

Author SHA1 Message Date
Eugenio Lacuesta
c5612f387b
Remove deprecated xlib module 2019-09-13 14:32:05 -03:00
Aditya
9d9d83a8c3 Use https links wherever possible 2017-10-28 16:24:40 +05:30
Mikhail Korobov
936266a61a Merge pull request #934 from Dineshs91/zsh-support 
[MRG+1] Added zsh completion for Scrapy command-line tool and updated bash completion
 2015-07-31 03:18:19 +05:00
Daniel Graña
47b1b748c2 Generated version as pep440 and dpkg compatible 2015-06-16 00:16:09 +00:00
agusc
140f58251b removed SUFFIX from scrapy name package 2015-06-15 17:02:09 -03:00
Elias Dorneles
3d3633f3d2 favoring web scraping over screen scraping in the descriptions 2015-04-25 11:20:20 -03:00
dinesh
e2b4fabbf5 Fix zsh completion path 2015-04-03 08:34:16 +05:30
dinesh
9d0ea5a396 Added zsh completion for the Scrapy command-line tool 2015-04-03 08:18:47 +05:30
Pablo Hoffman
5846d6154c emphasize web crawling over screen scraping on scrapy description. closes #586 2015-03-18 00:24:16 -03:00
Daniel Graña
3b64b2449e update other places where w3lib version is mentioned 2014-08-04 15:56:41 -03:00
Daniel Graña
e17866aaab there is not support for python3 yet 2014-07-01 17:12:01 +00:00
Daniel Graña
e58c23e4a9 Update python compatible version set to debian packages 2014-07-01 12:12:31 -03:00
alexanderlukanin13
ab6496647c Added "six>=1.5.2" to requirements 2014-01-15 13:05:00 +06:00
Daniel Graña
2df8156431 Drop Python 2.6 support 2013-10-29 13:44:00 -02:00
Daniel Graña
bd79b6e1d3 debian package requires python-cssselect 2013-10-24 16:47:46 -02:00
Pablo Hoffman
ca12886acb update copyright notes 2013-05-16 15:05:52 -03:00
Pablo Hoffman
d02da2f31f ported code to use queuelib 2013-04-23 17:48:09 -03:00
Daniel Graña
5db45b3825 remove scrapyd, it was migrated to its own repository 2013-02-06 05:24:07 +00:00
Daniel Graña
d74a067227 require w3lib 1.2 or greater 2012-05-15 17:11:51 -03:00
Pablo Hoffman
abcac4fcbd updated maintainer to scrapinghub 2012-05-02 03:25:35 -03:00
Pablo Hoffman
78185921c2 renamed and improved README to provide a more helpful github landing page 2012-04-28 23:30:00 -03:00
Pablo Hoffman
6e8edbd72e switched default selectors backend to lxml 2012-04-10 15:52:14 -03:00
Pablo Hoffman
9006227358 bumped required python-w3lib version in debian/control 2012-03-05 20:25:38 -02:00
Pablo Hoffman
85e2b493b4 make scrapyd debian package dependent on the same (or higher) version of scrapy package 2012-01-13 10:55:20 -02:00
Pablo Hoffman
8a45dd121b scrapyd: fixed issue with ubuntu package: /var/lib/scrapyd/items dir not being created by default 2012-01-12 17:17:50 -02:00
Pablo Hoffman
76af0cdd44 updated documentation and code to use -s instead of --set option 2011-09-01 14:35:37 -03:00
Pablo Hoffman
718428c0ab debian/control: added python-setuptools to Recommends, because it's need by 'scrapy deploy' command 2011-05-01 11:00:02 -03:00
Pablo Hoffman
bf73002428 removed googledir example, replaced by dirbot project on github. updated docs accordingly 2011-04-28 02:28:39 -03:00
Pablo Hoffman
ebcbb9f453 debian: added python-w3lib package to dependencies 2011-04-19 00:55:08 -03:00
Pablo Hoffman
b10f4fae35 Moved several functions from scrapy.utils.{http,markup,multipart,response,url} (and their tests) to a new library called 'w3lib'. Scrapy will now depend on w3lib. 2011-04-18 22:37:19 -03:00
Pablo Hoffman
60f6a9b054 moved scrapyd python module to scrapy debian package. left scrapyd package only for installing service (upstart script) and scrapy user 2011-03-23 15:45:40 -03:00
Pablo Hoffman
b1c89508f5 fixed wrong changes commited in previous changeset 2011-01-28 16:22:39 -02:00
Pablo Hoffman
4361150494 added missing scrapyd/default_scrapyd.conf file to MANIFEST.in 2011-01-28 16:21:00 -02:00
Pablo Hoffman
7dc521c56e make scrapyd package depend on specific scrapy version 2011-01-11 17:03:51 -02:00
Pablo Hoffman
ac8166251e make scrapy*-0.12 packages conflict with scrapy*-0.11 packages 2011-01-04 13:56:28 -02:00
Pablo Hoffman
2897061b98 Make scrapy conflict with previous versionsof the debian package 2010-11-17 17:30:22 -02:00
Pablo Hoffman
3926ca45b3 debian/control: Added scrapy/scrapyd to Provides 2010-11-17 17:08:21 -02:00
Pablo Hoffman
a034d078c8 Changed Debian packaging to use the scrapy version in the package name, so we can have multiple Scrapy versions in the same apt repo 
--HG--
rename : debian/scrapy.1 => debian/scrapy-files/scrapy.1
rename : debian/000-default => debian/scrapyd-files/000-default
rename : debian/scrapyd.upstart => debian/scrapyd.scrapyd.upstart
rename : debian/scrapy.1 => extras/scrapy.1
 2010-11-17 17:03:00 -02:00
Pablo Hoffman
1d5c56089c Automated merge with http://hg.scrapy.org/scrapy-0.10 2010-10-27 14:42:47 -02:00
Pablo Hoffman
c3e5b4bb03 changed pid file name to scrapyd 2010-10-27 14:42:22 -02:00
Pablo Hoffman
2bba87f69f Automated merge with http://hg.scrapy.org/scrapy-0.10 2010-10-27 14:20:21 -02:00
Pablo Hoffman
f47b9f608c simplified lockfile used by scrapyd (/var/run/scrapyd.pid instead of /var/run/scrapyd/scrapyd.pid). closes #271 2010-10-27 14:18:40 -02:00
Pablo Hoffman
d15a97ff61 Updated Scrapy version in debian/changelog 2010-09-28 16:45:05 -03:00
Pablo Hoffman
08952562e9 Make Ubuntu package conflict with standard Scrapy Debian/Ubuntu package 2010-09-17 16:25:02 -03:00
Pablo Hoffman
9158e9d682 Some changes to Scrapyd to support multiple configuration files, to make it easier to deploy Scrapyd applications. Also documented 'egg_runner' and 'application' options 
--HG--
rename : debian/scrapyd.cfg => debian/000-default
rename : scrapyd/default_scrapyd.cfg => scrapyd/default_scrapyd.conf
 2010-09-07 09:17:25 -03:00
Pablo Hoffman
37e9c5d78e Added new Scrapy service with support for: 
* multiple projects
* uploading scrapy projects as Python eggs
* scheduling spiders using a JSON API

Documentation is added along with the code.

Closes #218.

--HG--
rename : debian/scrapy-service.default => debian/scrapyd.default
rename : debian/scrapy-service.dirs => debian/scrapyd.dirs
rename : debian/scrapy-service.install => debian/scrapyd.install
rename : debian/scrapy-service.lintian-overrides => debian/scrapyd.lintian-overrides
rename : debian/scrapy-service.postinst => debian/scrapyd.postinst
rename : debian/scrapy-service.postrm => debian/scrapyd.postrm
rename : debian/scrapy-service.upstart => debian/scrapyd.upstart
rename : extras/scrapy.tac => extras/scrapyd.tac
 2010-09-03 15:54:42 -03:00
Pablo Hoffman
e14cc2c12a Moved scrapy-ws script to extras/ and fixed broken methods due to changes in web service API 
--HG--
rename : bin/scrapy-ws.py => extras/scrapy-ws.py
 2010-08-27 00:33:08 -03:00
Pablo Hoffman
8e83f527b3 Removed scrapy-sqs script, as it has been superseded by the new scrapy 'queue' command 2010-08-23 22:04:49 -03:00
Pablo Hoffman
0d9e75c684 Added bash completion for the Scrapy command-line tool. Closes #210 2010-08-21 03:23:45 -03:00
Pablo Hoffman
34554da201 Deprecated scrapy-ctl.py command in favour of simpler "scrapy" command. Closes #199. Also updated documenation accordingly and added convenient scrapy.bat script for running from Windows. 
--HG--
rename : debian/scrapy-ctl.1 => debian/scrapy.1
rename : docs/topics/scrapy-ctl.rst => docs/topics/cmdline.rst
 2010-08-18 19:48:32 -03:00