Ismael Carnales
|
5fac909670
|
try to import json from python 2.6 or fallback to simplejson
|
2009-08-11 15:23:38 -03:00 |
|
Daniel Grana
|
9a531b552e
|
remove compiled pys before running tests
|
2009-08-11 15:11:31 -03:00 |
|
Daniel Grana
|
edb7150b27
|
Automated merge with ssh://hg.scrapy.org/scrapy
|
2009-08-11 12:39:05 -03:00 |
|
Daniel Grana
|
743d326184
|
returning None from process_response is not allowed, ignore the request using exception instead
|
2009-08-11 12:39:02 -03:00 |
|
Ismael Carnales
|
3614998053
|
fixed error in doc
|
2009-08-11 09:23:45 -03:00 |
|
Pablo Hoffman
|
10a527daca
|
removed unused scrapy.contrib.codecs module
|
2009-08-10 21:02:46 -03:00 |
|
Pablo Hoffman
|
7ac7597950
|
removed obsolete scrapy.contrib.cluster
|
2009-08-10 21:02:46 -03:00 |
|
Pablo Hoffman
|
a0e2086b1b
|
moved deprecated scrapy.item.adaptors to scrapy.contrib.item, and added deprecation warning
--HG--
rename : scrapy/item/adaptors.py => scrapy/contrib/item/adaptors.py
|
2009-08-10 21:02:46 -03:00 |
|
Pablo Hoffman
|
e96622166e
|
removed backwards compatibility support for importing link extractors from scrapy.link.extractors
--HG--
rename : scrapy/link/__init__.py => scrapy/link.py
|
2009-08-10 21:02:46 -03:00 |
|
Pablo Hoffman
|
c0532712f9
|
removed unnecesary response ResponseSoup extension, and replaced by a utils function with cache support
|
2009-08-10 21:02:24 -03:00 |
|
Pablo Hoffman
|
d334c035c0
|
removed unnecesary ResponseLibxml2 extension and moved libxml2 document caching functionality to Libxml2Document using weak references
|
2009-08-10 20:52:43 -03:00 |
|
Daniel Grana
|
21f2bb6797
|
Automated merge with ssh://hg.scrapy.org/scrapy
|
2009-08-10 20:28:37 -03:00 |
|
Daniel Grana
|
7078cdc336
|
remove unmantained web server code
|
2009-08-10 20:28:32 -03:00 |
|
Pablo Hoffman
|
49bab4777b
|
XPathSelector: added 're' argument to add_xpath method, exposed selector attribute
|
2009-08-10 19:42:20 -03:00 |
|
Pablo Hoffman
|
dfa4a4846c
|
removed unused module: scrapy.xpath.types
|
2009-08-10 10:23:42 -03:00 |
|
Pablo Hoffman
|
50fcc67278
|
improved reducers examples
|
2009-08-10 10:13:30 -03:00 |
|
Pablo Hoffman
|
5be0cee577
|
added TreeExpander example
|
2009-08-09 20:54:22 -03:00 |
|
Pablo Hoffman
|
38f82e3993
|
loaders doc: added information about expanders/reducers declaration precendece, and other minor improvements
|
2009-08-09 18:06:12 -03:00 |
|
Pablo Hoffman
|
b296d4169e
|
minor doc update for making it more windows-friendly
|
2009-08-09 17:08:42 -03:00 |
|
Pablo Hoffman
|
80b96fc195
|
minor changes to referer logging when crawling
|
2009-08-08 16:07:10 -03:00 |
|
Pablo Hoffman
|
b8eb08f444
|
additional cleanup to scrapy.xpath module
|
2009-08-08 15:29:50 -03:00 |
|
Pablo Hoffman
|
467dbef20e
|
fixed bug when no project module setting is defined
|
2009-08-08 15:12:40 -03:00 |
|
Pablo Hoffman
|
91eea82eef
|
added XPathLoader for working with XPath Selectors more conveniently
|
2009-08-08 07:26:20 -03:00 |
|
Pablo Hoffman
|
10cdc70fc4
|
some cleanup to scrapy.xpath module
--HG--
rename : scrapy/xpath/constructors.py => scrapy/xpath/factories.py
|
2009-08-08 06:03:46 -03:00 |
|
Pablo Hoffman
|
661aeb5cda
|
moved ItemPipelineManager from scrapy.item.pipeline to scrapy.contrib.pipeline
|
2009-08-08 05:01:18 -03:00 |
|
Pablo Hoffman
|
174ef14d0a
|
some cleanup to item pipeline code
|
2009-08-08 04:57:18 -03:00 |
|
Pablo Hoffman
|
a67dc6c73c
|
removed unused module
|
2009-08-08 04:42:14 -03:00 |
|
Pablo Hoffman
|
5786bbd26a
|
cleaned up scrapy.command.cmdline module
|
2009-08-08 04:29:54 -03:00 |
|
Pablo Hoffman
|
6b20f1f7d9
|
added "Global Options" group to command line options, improved help display, splitted --profile option in --profile + --lsprof
|
2009-08-08 04:02:49 -03:00 |
|
Pablo Hoffman
|
1339b18218
|
some changes to command line options: use 'resolve' conflict_handler, improved option help messages, removed -f option, changed -o option to -L, removed redundant --nolog option
|
2009-08-08 03:08:19 -03:00 |
|
Daniel Grana
|
417f0a3a20
|
remove stat of warning level notification not reached
|
2009-08-07 21:29:19 -03:00 |
|
Daniel Grana
|
3901e06f44
|
Automated merge with ssh://hg.scrapy.org/scrapy
|
2009-08-07 21:24:41 -03:00 |
|
Daniel Grana
|
9aaf010af3
|
add stats of memory usage
|
2009-08-07 21:24:36 -03:00 |
|
Pablo Hoffman
|
14148b6ed8
|
fixed unittest codes broken in previous commit
|
2009-08-07 14:45:28 -03:00 |
|
Pablo Hoffman
|
db90e26a8b
|
renamed ItemLoader class to Loader
|
2009-08-07 14:39:30 -03:00 |
|
Pablo Hoffman
|
e585c6cac4
|
relocated experimental newitems/loaders doc, and added example for extending fields metadata
--HG--
rename : docs/experimental/newitem-loader.rst => docs/experimental/loaders.rst
rename : docs/experimental/newitem.rst => docs/experimental/newitems.rst
|
2009-08-07 14:28:58 -03:00 |
|
Pablo Hoffman
|
d95e99f585
|
Added documentation for Items and Loaders, removed obsolete Item Adaptors documentation
--HG--
rename : docs/experimental/topics/newitem/index.rst => docs/experimental/newitem.rst
|
2009-08-07 03:50:09 -03:00 |
|
Pablo Hoffman
|
efa08318be
|
renamed JoinStrings reducer to Join, accept item as first positional argument in ItemLoader constructor, removed expanders and reducers docstrings (will be moved to documentation)
|
2009-08-07 03:48:42 -03:00 |
|
Pablo Hoffman
|
3658acd9da
|
newitem: reverting to use 'default' Field key instead of 'default_factory'
|
2009-08-06 21:29:40 -03:00 |
|
Pablo Hoffman
|
78b69ec97e
|
merge
|
2009-08-06 14:35:52 -03:00 |
|
Pablo Hoffman
|
d142489b81
|
remove_entities: added test for encoding argument
|
2009-08-06 14:35:24 -03:00 |
|
Pablo Hoffman
|
f2c2609e42
|
remove_entities: added support for common browser hack for numeric character references in the 80-9F range
|
2009-08-06 14:31:59 -03:00 |
|
Pablo Hoffman
|
c3f7bfea1f
|
remove_entities: added encoding argument, and removed some empty lines
|
2009-08-06 14:26:38 -03:00 |
|
Daniel Grana
|
82dc2096fe
|
Automated merge with ssh://hg.scrapy.org/scrapy
|
2009-08-06 12:28:49 -03:00 |
|
Daniel Grana
|
6ef991b71f
|
normalize times used for stats to UTC
|
2009-08-06 12:07:22 -03:00 |
|
Pablo Hoffman
|
80db8abee2
|
use time.time() instead of datetime in SpiderProfiler extensions, which is faster and simpler
|
2009-08-06 11:56:32 -03:00 |
|
Pablo Hoffman
|
de48406f10
|
added 3 common content-types (for feeds) to ResponseTypes class
|
2009-08-06 11:37:20 -03:00 |
|
Pablo Hoffman
|
a23ff37050
|
ItemLoader: added one more test and improved other test names
|
2009-08-05 11:38:01 -03:00 |
|
Pablo Hoffman
|
7bc7af0162
|
ItemLoader: some more code cleanups, and added many more tests
|
2009-08-05 00:41:02 -03:00 |
|
Pablo Hoffman
|
9081e84e27
|
ItemLoader: sorted out module locations, and added more tests
--HG--
rename : scrapy/newitem/reducers.py => scrapy/newitem/loader/reducers.py
|
2009-08-04 20:02:49 -03:00 |
|