Pablo Hoffman
d334c035c0
removed unnecesary ResponseLibxml2 extension and moved libxml2 document caching functionality to Libxml2Document using weak references
2009-08-10 20:52:43 -03:00
Daniel Grana
21f2bb6797
Automated merge with ssh://hg.scrapy.org/scrapy
2009-08-10 20:28:37 -03:00
Daniel Grana
7078cdc336
remove unmantained web server code
2009-08-10 20:28:32 -03:00
Pablo Hoffman
49bab4777b
XPathSelector: added 're' argument to add_xpath method, exposed selector attribute
2009-08-10 19:42:20 -03:00
Pablo Hoffman
dfa4a4846c
removed unused module: scrapy.xpath.types
2009-08-10 10:23:42 -03:00
Pablo Hoffman
50fcc67278
improved reducers examples
2009-08-10 10:13:30 -03:00
Pablo Hoffman
5be0cee577
added TreeExpander example
2009-08-09 20:54:22 -03:00
Pablo Hoffman
38f82e3993
loaders doc: added information about expanders/reducers declaration precendece, and other minor improvements
2009-08-09 18:06:12 -03:00
Pablo Hoffman
b296d4169e
minor doc update for making it more windows-friendly
2009-08-09 17:08:42 -03:00
Pablo Hoffman
80b96fc195
minor changes to referer logging when crawling
2009-08-08 16:07:10 -03:00
Pablo Hoffman
b8eb08f444
additional cleanup to scrapy.xpath module
2009-08-08 15:29:50 -03:00
Pablo Hoffman
467dbef20e
fixed bug when no project module setting is defined
2009-08-08 15:12:40 -03:00
Pablo Hoffman
91eea82eef
added XPathLoader for working with XPath Selectors more conveniently
2009-08-08 07:26:20 -03:00
Pablo Hoffman
10cdc70fc4
some cleanup to scrapy.xpath module
...
--HG--
rename : scrapy/xpath/constructors.py => scrapy/xpath/factories.py
2009-08-08 06:03:46 -03:00
Pablo Hoffman
661aeb5cda
moved ItemPipelineManager from scrapy.item.pipeline to scrapy.contrib.pipeline
2009-08-08 05:01:18 -03:00
Pablo Hoffman
174ef14d0a
some cleanup to item pipeline code
2009-08-08 04:57:18 -03:00
Pablo Hoffman
a67dc6c73c
removed unused module
2009-08-08 04:42:14 -03:00
Pablo Hoffman
5786bbd26a
cleaned up scrapy.command.cmdline module
2009-08-08 04:29:54 -03:00
Pablo Hoffman
6b20f1f7d9
added "Global Options" group to command line options, improved help display, splitted --profile option in --profile + --lsprof
2009-08-08 04:02:49 -03:00
Pablo Hoffman
1339b18218
some changes to command line options: use 'resolve' conflict_handler, improved option help messages, removed -f option, changed -o option to -L, removed redundant --nolog option
2009-08-08 03:08:19 -03:00
Daniel Grana
417f0a3a20
remove stat of warning level notification not reached
2009-08-07 21:29:19 -03:00
Daniel Grana
3901e06f44
Automated merge with ssh://hg.scrapy.org/scrapy
2009-08-07 21:24:41 -03:00
Daniel Grana
9aaf010af3
add stats of memory usage
2009-08-07 21:24:36 -03:00
Pablo Hoffman
14148b6ed8
fixed unittest codes broken in previous commit
2009-08-07 14:45:28 -03:00
Pablo Hoffman
db90e26a8b
renamed ItemLoader class to Loader
2009-08-07 14:39:30 -03:00
Pablo Hoffman
e585c6cac4
relocated experimental newitems/loaders doc, and added example for extending fields metadata
...
--HG--
rename : docs/experimental/newitem-loader.rst => docs/experimental/loaders.rst
rename : docs/experimental/newitem.rst => docs/experimental/newitems.rst
2009-08-07 14:28:58 -03:00
Pablo Hoffman
d95e99f585
Added documentation for Items and Loaders, removed obsolete Item Adaptors documentation
...
--HG--
rename : docs/experimental/topics/newitem/index.rst => docs/experimental/newitem.rst
2009-08-07 03:50:09 -03:00
Pablo Hoffman
efa08318be
renamed JoinStrings reducer to Join, accept item as first positional argument in ItemLoader constructor, removed expanders and reducers docstrings (will be moved to documentation)
2009-08-07 03:48:42 -03:00
Pablo Hoffman
3658acd9da
newitem: reverting to use 'default' Field key instead of 'default_factory'
2009-08-06 21:29:40 -03:00
Pablo Hoffman
78b69ec97e
merge
2009-08-06 14:35:52 -03:00
Pablo Hoffman
d142489b81
remove_entities: added test for encoding argument
2009-08-06 14:35:24 -03:00
Pablo Hoffman
f2c2609e42
remove_entities: added support for common browser hack for numeric character references in the 80-9F range
2009-08-06 14:31:59 -03:00
Pablo Hoffman
c3f7bfea1f
remove_entities: added encoding argument, and removed some empty lines
2009-08-06 14:26:38 -03:00
Daniel Grana
82dc2096fe
Automated merge with ssh://hg.scrapy.org/scrapy
2009-08-06 12:28:49 -03:00
Daniel Grana
6ef991b71f
normalize times used for stats to UTC
2009-08-06 12:07:22 -03:00
Pablo Hoffman
80db8abee2
use time.time() instead of datetime in SpiderProfiler extensions, which is faster and simpler
2009-08-06 11:56:32 -03:00
Pablo Hoffman
de48406f10
added 3 common content-types (for feeds) to ResponseTypes class
2009-08-06 11:37:20 -03:00
Pablo Hoffman
a23ff37050
ItemLoader: added one more test and improved other test names
2009-08-05 11:38:01 -03:00
Pablo Hoffman
7bc7af0162
ItemLoader: some more code cleanups, and added many more tests
2009-08-05 00:41:02 -03:00
Pablo Hoffman
9081e84e27
ItemLoader: sorted out module locations, and added more tests
...
--HG--
rename : scrapy/newitem/reducers.py => scrapy/newitem/loader/reducers.py
2009-08-04 20:02:49 -03:00
Pablo Hoffman
ac9f4c9cc2
Refatored scrapy.newitem package:
...
- left only one type of Field - just a dict wrapper to contain field metadata
- removed Item Builder and tests
- adapted Item Loader to work with new Field class
2009-08-04 19:26:31 -03:00
Pablo Hoffman
114dba2850
some minor simplifications to tree_expander() function
2009-08-04 09:10:21 -03:00
Pablo Hoffman
8d705ec302
added ItemLoader class, an alternative implementation of ItemBuilder with a slightly different API
2009-08-03 22:53:08 -03:00
Ismael Carnales
f05695d75e
added first implementation of ItemBuilder
2009-08-03 17:27:57 -03:00
Ismael Carnales
32894643a0
added ListField documentation, ordered field reference alphabetically
2009-08-03 15:00:04 -03:00
Ismael Carnales
cf638e682c
made ListField init with an instance (not a class) of Field
2009-08-03 15:00:02 -03:00
Ismael Carnales
d8b85ae7ad
moved MultiValuedField to ListField
2009-08-02 18:43:35 -03:00
Pablo Hoffman
fcb33c7988
removed obsolete pipeline
2009-07-31 17:24:30 -03:00
Pablo Hoffman
7c049d2ef5
use standard 'mcs' for first argument of meta class __new__ method
2009-07-31 16:51:29 -03:00
Pablo Hoffman
02c454c26e
newitem: added warning when trying to access item field value via getattr instead of getitem
2009-07-31 16:49:27 -03:00
