Daniel Graña
19bcb44c25
Merge pull request #165 from andrix/master
...
Just a simple refactoring on *MemoryQueue classes that improve the performance
2012-08-10 15:04:30 -07:00
Andrés Moreira
ed8721059d
Small improvement on *MemoryQueue performance (mainly on push time)
2012-08-10 10:38:20 -03:00
Daniel Graña
483a70cb5b
Merge pull request #160 from midiotthimble/patch-1
...
Pass into the model only existing fields
test passing OK and changes makes sense
2012-08-09 05:41:50 -07:00
Pablo Hoffman
3bdd9b7b89
added more extensions to ignore on link extractors
2012-08-07 14:23:50 -03:00
Pablo Hoffman
16bee46c70
added ppt to scrapy.linkextractor.IGNORED_EXTENSIONS
2012-08-07 14:18:55 -03:00
Daniel Graña
abcc8c9f63
Recommend pypi as single way to install on Windows
2012-08-06 10:21:13 -03:00
Daniel Graña
b7b0c49520
append parse command to example code sections in docs. closes #162
2012-08-06 09:10:16 -03:00
Vladislav Poluhin
0497ac2fc8
Simple test for default values of model in DjangoItem
2012-07-26 20:55:55 +08:00
Vladislav
e14c41636f
Pass into the model only existing fields
...
Model fields has default values and when field doesn't exists in item container, added `None` instead default value. My patch solves this problem.
2012-07-25 10:10:46 +08:00
Pablo Hoffman
832e45073b
fixed typo in stats documentation. closes #159
2012-07-20 17:13:06 -03:00
Pablo Hoffman
831a45009a
Merge pull request #155 from warvariuc/import_settings_module_verbose
...
Spider not found - confusion
2012-07-09 12:39:01 -07:00
Victor Varvariuc
eb91cad0da
if you put in settings.py something like import local_settings and local_settings doesn't exist or contains errors you get:
...
/usr/local/lib/python2.7/dist-packages/Scrapy-0.15.1-py2.7.egg/scrapy/utils/project.py:17: UserWarning: Cannot import scrapy settings module settings
warnings.warn("Cannot import scrapy settings module %s" % scrapy_module)
...
File "/usr/local/lib/python2.7/dist-packages/Scrapy-0.15.1-py2.7.egg/scrapy/cmdline.py", line 117, in _run_command
cmd.run(args, opts)
File "/usr/local/lib/python2.7/dist-packages/Scrapy-0.15.1-py2.7.egg/scrapy/commands/crawl.py", line 43, in run
spider = self.crawler.spiders.create(spname, **opts.spargs)
File "/usr/local/lib/python2.7/dist-packages/Scrapy-0.15.1-py2.7.egg/scrapy/spidermanager.py", line 43, in create
raise KeyError("Spider not found: %s" % spider_name)
KeyError: 'Spider not found: fb_spider'
Which is not very descriptive.
Now showing details about the exception when ImportError was raises
2012-07-09 08:39:14 +04:00
Pablo Hoffman
99bfd1e590
Merge pull request #154 from vially/patch-1
...
Fix typo in tutorial
2012-07-04 04:36:16 -07:00
Valentin-Costel Hăloiu
00bfb37e79
Update master
2012-07-04 06:55:01 +03:00
Daniel Graña
a40090a6a7
Merge pull request #151 from msabramo/tox
...
Add support for tox (http://tox.testrun.org/ )
2012-06-29 05:37:24 -07:00
Marc Abramowitz
61952c3be6
Add .tox to .gitignore
2012-06-28 12:43:33 -07:00
Marc Abramowitz
9ae8ea96c4
Add tox.ini for tox ( http://tox.testrun.org/ )
2012-06-28 12:43:12 -07:00
Daniel Graña
277ed0ae23
Merge pull request #145 from alexcepoi/cookies-changes
...
domain and path support for request cookies
2012-06-25 11:29:04 -07:00
Pablo Hoffman
8b30575dd3
Merge pull request #147 from tonal/cfg-jsons
...
Configure json services members
2012-06-25 11:27:13 -07:00
Alexandru Cepoi
177c81745d
domain and path support for request cookies
2012-06-25 20:17:59 +02:00
Alexandr N Zamaraev (aka tonal)
cf968c328f
Configure json services members
2012-06-25 12:50:48 +07:00
Pablo Hoffman
179e3810dc
fixed links to doc. closes #150
2012-06-24 01:00:33 -03:00
Pablo Hoffman
700f20b28f
Merge pull request #149 from alexcepoi/parse-changes
...
documentation for `parse` command, debugging spiders section
2012-06-22 17:50:40 -07:00
Alexandru Cepoi
2e05cf5685
fix small bug with parse command
2012-06-21 20:06:50 +02:00
Alexandru Cepoi
f4faa19e31
added docs topic debugging spiders
2012-06-21 20:03:33 +02:00
Pablo Hoffman
4eeda53b0d
Merge pull request #148 from tonal/cfg-launcher
...
Add launcher class to config
2012-06-19 08:14:58 -07:00
Alexandr N Zamaraev (aka tonal)
3bddedfc6d
Add launcher class to config
2012-06-19 22:01:53 +07:00
Daniel Graña
27689009f1
fix urlparse monkeypatches for python 2.7.4. closes #144
2012-06-18 11:50:43 -03:00
Alexandru Cepoi
3e05a2ecf6
update docs for parse command
2012-06-12 18:28:10 +02:00
Pablo Hoffman
e1be9c01bc
updated FAQ about bot bans
2012-06-08 18:33:53 -03:00
Daniel Graña
3d6edc2f40
Merge pull request #135 from alexcepoi/parse-changes
...
Add --depth option to parse
2012-06-07 07:52:27 -07:00
Alexandru Cepoi
3b5cf31198
add --verbose option to parse command
2012-05-24 16:35:09 +02:00
Alexandru Cepoi
8e3c5f1bf7
add --depth field to parse command
2012-05-24 15:10:03 +02:00
Pablo Hoffman
8d77005047
scrapy shell: start shell in main thread and crawler in secondary thread, instead of the other way around. fixes #100
2012-05-22 19:15:54 -03:00
Pablo Hoffman
b33303779a
scrapyd.launcher: make SCRAPY_LOG_FILE and SCRAPY_FEED_URI optional
2012-05-21 14:29:15 -03:00
Daniel Graña
35ef7de546
add travis-ci build status to README
2012-05-17 09:07:36 -03:00
Daniel Graña
e77e4b5f6e
add precise to travis-ci build enviroments
2012-05-17 09:03:28 -03:00
Daniel Graña
3096e46401
add requirements file per travis env
2012-05-17 08:53:18 -03:00
Daniel Graña
7740581f88
Merge remote-tracking branch 'upstream/master'
2012-05-17 08:50:28 -03:00
Daniel Graña
c3a3108799
multiple build enviroments for travis-ci
2012-05-17 08:49:45 -03:00
Pablo Hoffman
1bc18434f5
removed obsolete entries from MANIFEST.in
2012-05-17 01:09:48 -03:00
Daniel Graña
7fc573a230
fix libxml2 test
2012-05-16 18:54:02 -03:00
Daniel Graña
f530b0b3eb
make libxml2 optional now that lxml is the default
2012-05-16 18:17:51 -03:00
Daniel Graña
8376d95ce8
add travis-ci build configuration file
2012-05-16 16:54:12 -03:00
Pablo Hoffman
b4f368c37e
warn if Link objects are instantiated with unicode urls
2012-05-16 13:12:25 -03:00
Pablo Hoffman
30b6c77ce5
fixed typo in previous commit
2012-05-16 09:16:46 -03:00
Pablo Hoffman
b53bc66c76
added lxml/libxml2 versions to 'scrapy version' output
2012-05-16 09:12:02 -03:00
Daniel Graña
d74a067227
require w3lib 1.2 or greater
2012-05-15 17:11:51 -03:00
Daniel Graña
ae2ff4d33a
update news file with 0.14.4 release notes
2012-05-15 16:16:02 -03:00
Pablo Hoffman
9686f97242
added precise to supported ubuntu distros
2012-05-12 19:54:36 -03:00
