Andrey Rakhmatullin
cc480680d7
Merge pull request #6650 from wRAR/cleanup-cache-tests
...
Remove a duplicate test.
2025-02-03 23:33:07 +04:00
Andrey Rakhmatullin
ba5df629a2
Refactor downloader tests ( #6647 )
...
* Make download handler test base classes abstract.
* Small cleanup.
* Don't run the full test suite for special HTTP cases.
* Don't run tests in imported base classes.
* Remove an obsolete service_identity check.
* Move FTP imports back to the top level.
* Simplify the H2DownloadHandler import.
* Forbig pytest 8.2.x.
* Revert "Simplify the H2DownloadHandler import."
This reverts commit ed187046ac53c395c7423c0f5e6fb2bc7c27838f.
2025-02-03 20:11:47 +05:00
Andrey Rakhmatullin
4842bcbf1d
Deprecate and disable escape_ajax().
2025-02-03 14:08:05 +05:00
Andrey Rakhmatullin
393ff96e45
Deprecate AjaxCrawlMiddleware.
2025-02-03 14:08:05 +05:00
Mikhail Korobov
b4c2531021
Merge pull request #6648 from wRAR/mockserver-calls
...
Optimize mockserver calls
2025-02-03 14:06:35 +05:00
Andrey Rakhmatullin
df688910e0
Remove a duplicate test.
2025-02-02 18:48:26 +05:00
Andrey Rakhmatullin
783b98deda
Make mockserver instances per-class.
2025-02-02 14:10:09 +05:00
Andrey Rakhmatullin
1a0dfbd32e
Reuse mockserver instances in test_feedexport.py.
2025-02-02 13:28:34 +05:00
Andrey Rakhmatullin
200d76afa9
Refactor EngineTest tests.
2025-02-01 16:07:55 +05:00
Andrey Rakhmatullin
0a80871c3a
Remove scrapy.core.downloader.webclient._parse().
2025-01-28 22:22:09 +05:00
Andrey Rakhmatullin
0d2d2892ba
Silence the readBody warning.
2025-01-28 02:08:49 +05:00
Andrey Rakhmatullin
bc1aeeefc9
Deprecate overriding ScrapyClientContextFactory.getContext().
2025-01-28 02:07:47 +05:00
Andrey Rakhmatullin
16b998f9ca
Sort out webclient tests.
2025-01-28 01:35:04 +05:00
Lidiane T
98a57e2418
Fix error when running scrapy bench ( #6633 )
2025-01-27 11:21:30 +01:00
Andrey Rakhmatullin
cec0aeca58
Bump ruff, switch from black to ruff-format ( #6631 )
2025-01-27 11:07:09 +01:00
anubhav
c03fb2abb8
fix: added feed_options as a keyword argument to GCSFeedStorage. ( #6628 )
2025-01-23 21:06:45 +05:00
Andrey Rakhmatullin
499b6c66b4
Merge pull request #6626 from Laerte/master
...
fix: test_s3_export fails with boto3 >= 1.36.0
2025-01-22 15:14:33 +04:00
guillermo-bondonno
9bc0029d27
Allow updating pre-crawler settings from add-ons ( #6568 )
2025-01-22 12:07:44 +01:00
Laerte Pereira
14219b1fca
fix: test_s3_export fails with boto3 >= 1.36.0
2025-01-22 07:16:22 -03:00
anubhav
ca345a3b73
Flexible severity of logging level when items are dropped ( #6608 )
2025-01-15 11:08:18 +01:00
anubhav
1c1e83895c
Fix _pop_command_name ( #6606 )
2025-01-14 16:40:24 +01:00
Adrián Chaves
98ba61256d
Deprecate BaseDupeFilter.log() and improve dupefilter docs ( #4151 )
...
* Remove BaseDupeFilter.log()
It is never called because request_seen() always returns False
* Document the interface of DUPEFILTER_CLASS classes
* Remove unnecessary BaseDupeFilter comments and add a short class description
* Improve the documentation related to the DUPEFILTER_CLASS setting
* Deprecate BaseDupeFilter.log
* Update the docs
* Fix the new code example
* Remove typing to keep the example short
Otherwise, it would have required yet another import line (from __future__ or typing).
---------
Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
2025-01-14 19:36:56 +05:00
Ionut-Cezar Ciubotariu
402500b164
Change unknown cmd message when outside project ( #3426 )
...
* Change unknown cmd message when outside project
* Simplification.
* Move the import to the top level.
* Reword the message.
---------
Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
2025-01-10 23:08:27 +05:00
Kevin Lloyd Bernal
1fc91bb462
new allow_offsite parameter in OffsiteMiddleware ( #6151 )
...
* new 'allow_offsite' parameter in OffsiteMiddleware
* document deprecated dont_filter flag in OffsiteMiddleware
* avoid deprecating dont_filter in OffsiteMiddleware
* Copy the code to the downloader mw.
* Add tests for allow_offsite in the downloader mw.
* Mark allow_offsite with reqmeta.
---------
Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
2025-01-08 21:28:51 +05:00
Andrey Rakhmatullin
7dfbecd392
Fix tracking of coverage in subprocesses.
2025-01-07 20:41:48 +05:00
Andrey Rakhmatullin
4d31277bc6
Explicitly mark re-exports. ( #6579 )
2025-01-02 23:48:14 +01:00
Andrey Rakhmatullin
c330a399dc
Merge pull request #6601 from wRAR/ruff-rules-5
...
Add more Ruff rules, do some pylint cleanups
2025-01-02 17:38:15 +04:00
Andrey Rakhmatullin
176ae348c5
Reformat long REFERRER_POLICY.
2025-01-02 18:14:18 +05:00
Andrey Rakhmatullin
6ae5b92671
Drop the remaining unittest.main() blocks. ( #6602 )
2025-01-02 12:45:04 +01:00
Arthur
b10d46d280
Fix the calculate_final_stats method ( #6599 )
2025-01-02 12:36:28 +01:00
Andrey Rakhmatullin
dc706d4fc3
Remove useless pylint: disable lines.
2025-01-02 12:32:25 +05:00
Andrey Rakhmatullin
b70443f2d0
Split ruff and pylint ignores into two categories, some pylint cleanup.
2025-01-02 01:03:42 +05:00
Andrey Rakhmatullin
c87354cd46
Enable SIM Ruff rules.
2025-01-02 01:03:36 +05:00
Andrey Rakhmatullin
273620488c
Enable PTH Ruff rules.
2025-01-01 22:21:51 +05:00
Andrey Rakhmatullin
f44ca39fa2
Enable FLY Ruff rules.
2025-01-01 21:50:02 +05:00
Andrey Rakhmatullin
838ff99f37
Enable RUF Ruff rules.
2025-01-01 21:41:08 +05:00
Andrey Rakhmatullin
cc484efd43
Replace isort with the ruff isort rules.
2024-12-29 14:15:16 +05:00
Laerte Pereira
a195af304d
Deprecate w3lib objects importable from scrapy.utils.url ( #6586 )
2024-12-18 11:50:44 +05:00
Adrián Chaves
21b9ba717c
Allow customizing logged software versions ( #6582 )
...
Co-authored-by: Grammy Jiang <grammy.jiang@gmail.com>
Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
2024-12-16 14:46:23 +01:00
Andrey Rakhmatullin
7dd92e6e43
Add pylint rules to ruff, refresh the ignore list of pylint itself ( #6584 )
2024-12-16 09:44:36 +01:00
Andrey Rakhmatullin
1e4c81e9dc
Add Perflint rules to ruff.
2024-12-12 21:03:34 +05:00
Andrey Rakhmatullin
c2832ed131
Add flake8-pyi rules to ruff.
2024-12-12 20:44:28 +05:00
Andrey Rakhmatullin
93644f2c30
Add flake8-pie rules to ruff.
2024-12-12 20:33:35 +05:00
Andrey Rakhmatullin
e7595837a6
Add flake8-raise rules to ruff.
2024-12-12 20:33:35 +05:00
Andrey Rakhmatullin
897e124a27
Add flake8-return rules to ruff.
2024-12-12 20:32:25 +05:00
Andrey Rakhmatullin
cde0845ab2
Ruff: migrate pyupgrade and bandit, enable some other rules ( #6577 )
2024-12-10 19:53:27 +01:00
Andrey Rakhmatullin
f4d8d6d8ac
Tidy up noqa comments.
2024-12-10 14:58:45 +05:00
Andrey Rakhmatullin
cc146b9df7
Add ruff with basic rules. ( #6565 )
2024-12-09 10:47:47 +01:00
Andrey Rakhmatullin
4dcc04be48
Add tests for DOWNLOADER_CLIENT_TLS_METHOD, remove dead code.
2024-11-24 12:44:48 +04:00
Swayam Gupta
8c23da943c
Integrating configs into pyproject.toml ( #6547 )
2024-11-19 19:21:15 +05:00
