Daniel Grana
1f9e0f10eb
don't log ignored requests errors at images pipeline
2009-09-28 12:38:49 -03:00
Daniel Grana
c795771282
remove unused line from project settings template
2009-09-23 14:53:40 -03:00
Daniel Grana
dfd35c5280
Add common javascript mimetypes to be detected as TextResponses, also improve mimetype detection to avoid querying twice classes dict
2009-09-23 14:26:58 -03:00
Ismael Carnales
ef0e5d368f
updated spider templates to reflect the new spider callback return policy
2009-09-22 11:25:42 -03:00
Ismael Carnales
5862ba7db7
modified doc to reflect the new spider callback return policy (lists not needed)
2009-09-22 11:25:40 -03:00
Ismael Carnales
802f918b69
removed obsolete doc static file
2009-09-22 11:25:38 -03:00
Daniel Grana
6467c7055d
check iterable output of crawl spider and add tests for common function used to iterate spider output
2009-09-21 13:57:16 -03:00
Pablo Hoffman
b6155801f0
Added tag 0.7.0-rc1 for changeset f4a14daa04e6
2009-09-18 00:52:41 -03:00
Pablo Hoffman
bc778a0e5b
make_release.sh: added warning about broken bdist_wininst
2009-09-17 15:28:36 -03:00
Daniel Grana
1b28858bf9
improve XmlRpcRequest and update tests to cover more cases
2009-09-17 14:26:42 -03:00
Pablo Hoffman
48c3fb3ace
made XmlRpcRequestTest inherit from RequestTest and removed redundant test
2009-09-17 12:21:43 -03:00
Pablo Hoffman
6845505e7a
added make_release.sh script
2009-09-17 11:46:47 -03:00
Pablo Hoffman
6e10914454
added pypi classifiers to setup.py
2009-09-17 11:22:48 -03:00
Pablo Hoffman
6e93872955
updated installation guide for using releases
2009-09-17 11:06:55 -03:00
Pablo Hoffman
653db1d673
added bitmap for windows installer
2009-09-17 02:01:40 -03:00
Pablo Hoffman
132557dd14
some deployment changes in preparation for the 0.7.0 release candidate
2009-09-16 22:40:36 -03:00
Ismael Carnales
fd41f06056
added doc on how to enable an Item Pipeline component
2009-09-16 14:19:16 -03:00
Ismael Carnales
404e7e09d7
changed spider doc references in BaseSpider class
2009-09-16 14:10:11 -03:00
Daniel Grana
062730cbd8
fix csv exporter documentation
2009-09-16 00:17:50 -03:00
Daniel Grana
08aedcbe76
Automated merge with ssh://hg.scrapy.org/scrapy
2009-09-15 09:35:24 -03:00
Daniel Grana
e8514445dc
bugfix image thumb when no thumb is configured
2009-09-15 09:35:21 -03:00
Pablo Hoffman
02e228ad76
added support for returning deferreds in spider manager close_domain() method, and making sure engine_stopped signal is always sent (even when no spiders have run)
2009-09-15 09:27:30 -03:00
Pablo Hoffman
66ff3b3013
renamed defer_failed to defer_fail
2009-09-15 08:44:31 -03:00
Pablo Hoffman
56b292e057
XmlItemExporter: added built-in support for exporting multi-valued fields (for convenience)
2009-09-14 22:05:52 -03:00
Pablo Hoffman
e8960bf616
added runspider command to run spiders directly, without having to create a project
2009-09-14 22:05:14 -03:00
Pablo Hoffman
fcbbb5001e
ported spiderctl web console extensin to work with new core based on spider references
2009-09-14 20:35:47 -03:00
Pablo Hoffman
47aa716630
adapted web console to use unix timestamps for uptime instead of datetime
2009-09-14 20:35:07 -03:00
Pablo Hoffman
bc463bc9e8
using time.time() instead of datetime.utcnow() in engine.start_time atttribute
2009-09-14 20:28:46 -03:00
Pablo Hoffman
7e07f76edd
made pending_spiders attribute protected in spider scheduler
2009-09-14 20:27:54 -03:00
Pablo Hoffman
9b68432624
commented out line that was preventing errors from propagating to the request errback
2009-09-14 12:27:29 -03:00
Pablo Hoffman
2322312c63
Logging requests instead of responses in 'Crawled ...' messages
2009-09-14 10:31:29 -03:00
Pablo Hoffman
6f64dfe579
renamed spider manager close_domain() method to close_spider()
2009-09-14 10:06:54 -03:00
Pablo Hoffman
99467d4e6e
Changed (unstable) scheduler middleware API to receive spider (instead of domain) in enqueue_request method
2009-09-13 20:51:43 -03:00
Pablo Hoffman
00873cd16c
Another Spider Manager simplification: removed add_spider() method
2009-09-12 21:27:58 -03:00
Pablo Hoffman
dc82550058
Do not impose an arbitrary encoding in spider templates, because we don't know beforehand what enconding our users will use in their editors.
2009-09-12 21:04:21 -03:00
Pablo Hoffman
58bdd7303c
fixed deprecated import in 'crawl' spider template (thanks Anibal)
2009-09-12 20:59:04 -03:00
Pablo Hoffman
90407d2789
added missing colon
2009-09-12 19:33:16 -03:00
Pablo Hoffman
3f30fee6ea
added first (not yet stable) revision of Crawler class, which allows to use the Scrapy crawler from stand-alone scripts
2009-09-12 19:32:23 -03:00
Pablo Hoffman
1381c1e50a
removed (no longer needed) hack in setup.py
2009-09-12 14:50:05 -03:00
Pablo Hoffman
921fc4f3bf
Big Scrapy core refactoring to pass around spider references instead of domains.
...
This is to avoid accessing the scrapy.spider.spiders singleton for "resolving"
spiders, which is considered an "evil" practice because it ties us to the
singleton model for the spider resolver, which is a bad thing.
This change will also work as the foundation for the API cleaning that we'll
perform for 0.8. We decided to introduce this change now to have a more common
basecode between 0.7 and 0.8, which will allow us to better support 0.7 until
0.8 is released.
However, this change doesn't modify the stable/documented API, nor does it
change the core logic. Those changes will land on the 0.8 branch, after 0.7 is
released.
--HG--
rename : scrapy/contrib/domainsch.py => scrapy/contrib/spiderscheduler.py
2009-09-12 14:34:18 -03:00
Pablo Hoffman
655cfe138d
removed unused imports
2009-09-11 19:38:31 -03:00
Pablo Hoffman
0c292e3350
removed hacky --callback option to crawl command
2009-09-11 19:36:00 -03:00
Pablo Hoffman
e854d0d6ef
removed redundant --nopipelines function. same behaviour can be obtained by clearing the ITEM_PIPELINES setting
2009-09-11 19:32:05 -03:00
Pablo Hoffman
8d49dc2fb5
changed IMAGES_THUMBS setting to a dict instead of a list of tuples, and more improvements to images pipeline doc
2009-09-11 17:36:00 -03:00
Pablo Hoffman
e20f766792
fixed some typos
2009-09-11 16:55:37 -03:00
Pablo Hoffman
c2fe350f72
more changes to images pipeline doc
2009-09-11 16:53:36 -03:00
Ismael Carnales
ada46a2dbb
styled imagesp doc
2009-09-11 15:30:46 -03:00
Pablo Hoffman
be0f2beef0
more cleanup to scheduler middelware doc, and permanentely moved to experimental doc
2009-09-11 13:27:31 -03:00
Pablo Hoffman
0af052b68f
removed confusing title
2009-09-11 12:19:18 -03:00
Pablo Hoffman
f3240748cb
changed link to scheduler middleware doc, now in experimental
2009-09-11 12:03:23 -03:00
