Daniel Graña
b58285b635
fix a reference to unexistent engine.slots. closes #593
2014-02-14 15:31:05 -02:00
Pablo Hoffman
c886d7459f
add SEP-021 (Add-ons) - work in progress
2014-02-11 20:16:09 -02:00
Mikhail Korobov
9caf37553e
Merge pull request #590 from Digenis/master
...
downloaderMW doc typo (spiderMW doc copy remnant)
2014-02-12 02:49:30 +05:00
Nikolaos-Digenis Karagiannis
43a797e2f7
downloaderMW doc typo (spiderMW doc copy remnant)
2014-02-11 22:30:00 +02:00
tracicot
b2f4b296df
Correct typos
2014-02-10 11:46:23 -02:00
Daniel Graña
783d8e3ae1
Add 0.22.1 release notes
2014-02-08 17:09:04 -02:00
Daniel Graña
877e345d79
localhost666 can resolve under certain circumstances
...
localhost666 is resolvable if the host running the tests contains a
"search mydomain.com" in /etc/resolv.conf and a wildcard dns entry
exists for *.mydomain.com
2014-02-08 16:38:22 -02:00
Daniel Graña
45c9dab15b
Merge pull request #582 from kmike/inspect-workaround
...
[WIP] Handle cases when inspect.stack() fails
2014-02-06 19:41:32 -08:00
Daniel Graña
9e95b36a09
test inspect.stack failure
2014-02-07 01:36:43 -02:00
Daniel Graña
c12e3a58b4
Merge pull request #585 from kmike/mitmproxy-reqs-cleanup
...
[MRG] testing PIL dependency is removed because there is a new mitmproxy version
2014-02-05 14:39:56 -08:00
Daniel Graña
3d4fe60e47
Merge pull request #584 from dangra/581-deprecated-subclass-fix
...
Fix wrong checks on subclassing of deprecated classes
2014-02-05 13:48:40 -08:00
Daniel Graña
ee896b154c
Fix wrong checks on subclassing of deprecated classes. closes #581
2014-02-05 19:47:04 -02:00
Mikhail Korobov
8e88fbc255
testing PIL dependency is removed because there is a new mitmproxy version
2014-02-06 00:49:26 +06:00
Mikhail Korobov
8a1905e6ee
Handle cases when inspect.stack() fails
2014-02-05 02:28:51 +06:00
Daniel Graña
45417e6b03
Merge pull request #574 from redapple/htmllx-tests
...
Fix HtmlParserLinkExtractor and tests after #485 merge
2014-02-02 19:09:02 -08:00
Mikhail Korobov
b74e7dd30a
Merge pull request #575 from redapple/docs-tutorialindent
...
Docs: 4-space indent for final spider example
2014-02-01 15:07:01 -08:00
Paul Tremberth
57f30bcb04
Docs: 4-space indent for final spider example
2014-02-01 23:34:55 +01:00
Paul Tremberth
8017d85d8a
Fix HtmlParserLinkExtractor and tests after #485 merge
2014-02-01 22:52:56 +01:00
Pablo Hoffman
e31fb49320
make 'basic' the default template spider in genspider, and added info with next steps to startproject. closes #488
2014-02-01 17:38:31 -02:00
Pablo Hoffman
c928eef585
Merge pull request #485 from bblanchon/sgml-extractor-link-text
...
SgmlLinkExtractor: Fixed link text when there is an inner tag
2014-02-01 10:28:56 -08:00
Paul Tremberth
38a96fba2d
CrawSpider: support process_links as generator
...
Adding tests for CrawlSpider's process_links
2014-01-31 21:25:05 +01:00
Daniel Graña
e1c6d3ffdf
Merge pull request #566 from redapple/offsite-stats
...
OffsiteMiddleware: add 2 stats counters
2014-01-31 10:05:35 -08:00
Daniel Graña
1e553ec0cd
Merge pull request #570 from redapple/travistests
...
[MRG] Fix tests for Travis-CI build
2014-01-31 09:16:54 -08:00
Paul Tremberth
cc157f182b
Fix tests for Travis-CI build
2014-01-31 01:27:54 +01:00
Mikhail Korobov
05158803df
Merge pull request #571 from darkrho/httpcache-storage-doc
...
DOC Fixed HTTPCACHE_STORAGE typo in the default value which is now Files...
2014-01-30 08:48:39 -08:00
Rolando Espinoza
a6279fe95b
DOC Fixed HTTPCACHE_STORAGE typo in the default value which is now Filesystem instead Dbm.
2014-01-30 11:53:42 -04:00
Paul Tremberth
fd5b40593a
Always enable offsite stats + refactor test to initialize crawler
2014-01-29 17:37:24 +01:00
Paul Tremberth
1a545157c6
Offsite: add 2 stats counters
2014-01-29 17:37:24 +01:00
Pablo Hoffman
21f0e4048b
Merge pull request #565 from dangra/562-sgmlinkextractor
...
replace unencodeable codepoints with html entities
2014-01-27 22:01:39 -08:00
Pablo Hoffman
116a1df4b0
Merge pull request #557 from darkrho/expose-crawler-in-shell
...
Expose current crawler in the scrapy shell.
2014-01-27 21:08:02 -08:00
Daniel Graña
66829c962f
replace unencodeable codepoints with html entities. fixes #562 and #285
2014-01-27 11:37:09 -02:00
Daniel Graña
8ecf0b786d
Merge pull request #556 from darkrho/item-loader-nones
...
Make `ItemLoader` ignore `None` values from processors.
2014-01-27 04:21:09 -08:00
Daniel Graña
b14dabb281
Merge pull request #561 from redapple/regexlx-encoding
...
RegexLinkExtractor: encode URL unicode value when creating Links
2014-01-24 07:33:23 -08:00
Rolando Espinoza
4255e12bc7
Updated the tutorial crawl output with latest output.
2014-01-23 18:18:56 -04:00
Rolando Espinoza
9aab9224cb
Updated shell docs with the crawler reference and fixed the actual shell output.
...
Also updated the shell example with a reproducible code example.
2014-01-23 18:04:57 -04:00
Paul Tremberth
f87859a627
RegexLinkExtractor: encode URL unicode value when creating Links
2014-01-23 19:41:18 +01:00
Rolando Espinoza
4081ba238d
PEP8 minor edits.
2014-01-23 11:13:54 -04:00
Rolando Espinoza
240fdde667
Expose current crawler in the scrapy shell.
2014-01-23 11:08:56 -04:00
Rolando Espinoza
b93412059d
Unused re import and PEP8 minor edits.
2014-01-23 10:37:33 -04:00
Rolando Espinoza
420efe77b2
Ignore None's values when using the ItemLoader.
2014-01-23 10:36:06 -04:00
Pablo Hoffman
2d60f86084
Merge pull request #535 from redapple/xpath-smartstrings
...
Disable smart strings in lxml XPath evaluations
2014-01-22 07:41:31 -08:00
Daniel Graña
677afe7e54
show ubuntu setup instructions as literal code
2014-01-20 16:22:53 -02:00
Daniel Graña
1c514c5fe9
Merge pull request #549 from dangra/509-scrapy-apt-repo
...
[MRG] update instruction to install using ubuntu packages
2014-01-20 09:27:57 -08:00
Daniel Graña
ebfb5b7096
replace warning about updating package lists by a note on package upgrade
2014-01-20 15:18:34 -02:00
Daniel Graña
52c3ff9190
fix apt-get line
2014-01-20 14:39:26 -02:00
Paul Tremberth
a0e25aec00
Use assertTrue/False
2014-01-20 17:29:16 +01:00
Daniel Graña
eb73ddd301
Update Ubuntu installation instructions
2014-01-20 14:15:01 -02:00
stray-leone
7f30a671c3
modify the version of scrapy ubuntu package
...
latest version is 0.22.
with scrapy-0.18, tutorial project provides error
relative issue : https://github.com/scrapy/scrapy/issues/511
2014-01-20 10:24:34 -02:00
Daniel Graña
431f2d109f
fix 0.22.0 release date
2014-01-17 17:54:01 -02:00
Mikhail Korobov
dfd13f9941
fix typos in news.rst and remove (not released yet) header
2014-01-18 01:50:19 +06:00
