This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
Scrapy Release Procedure
This page outlines the procedure used to release a new version of Scrapy.
You will need bump2version installed.
Release notes
First the release notes page of the documentation must be udpated to include an
entry for the new version. Do this in the target branch, which will be master
if you are releasing a major or minor version, or a version-specific branch in
case of a patch version.
The changes must include looking for the following references in the code base and replacing them with the corresponding values where appropriate:
-
VERSION: new major or minor version being released, without the patch version number (x.y, notx.y.0) -
PREVIOUS_VERSION: the major or minor version before the new version being released, without the patch version number (x.y, notx.y.0)
A pull request for this can be created at any point, to allow for discussions, and updated as new changes are accepted into the target branch. Once the code is frozen and the release date set, the pull request can be updated accordingly and merged into the target branch.
Version and git branch
Note: The code below assumes that the Scrapy repository is configured as
the upstream remote in your local clone, which you can do with:
git remote add upstream git@github.com:scrapy/scrapy.git
If you are releasing major version x.0.0:
git checkout master
git pull --ff-only upstream master
git log -1 # Check that the last commit is the release notes update
bump2version major
git checkout -b x.0
git push upstream master x.0 x.0.0
If you are releasing minor version x.y.0:
git checkout master
git pull --ff-only upstream master
git log -1 # Check that the last commit is the release notes update
bump2version minor
git checkout -b x.y
git push upstream master x.y x.y.0
If you are releasing patch version x.y.z:
git checkout x.y
git pull --ff-only upstream x.y
git log -1 # Check that the last commit is the release notes update
bump2version patch
git push upstream x.y x.y.z
Python Package Index
New Scrapy versions are automatically deployed on the
Python Package Index through
GitHub Actions, at the
end of the tag build job where PYPI_RELEASE_JOB=true is defined.
Read the Docs
For new major and minor versions:
-
Enable the new version:
-
Find the entry of the new branch (not tag) under Inactive Versions, at the bottom of the page, and select Edit.
-
Mark the Active checkbox, and select Save.
-
Set the new version as default:
-
Change the Default branch to the new branch (not tag), and select Save at the bottom of the page.
Github release
Create a Github release entry, with a description including the highlights from the release notes and a link to the release notes of the new version in Read the Docs.
This step can be skipped in the case of new tags created just to trigger re-packagings in the Python Package Index.
scrapy.org website
Update scrapy versions
Send announcements
In the case or major and minor versions, or patch versions that users should install with urgency (e.g. patch versions fixing security issues), leave a message with a link to the release notes in:
Update Conda packages
Create a pull request in https://github.com/conda-forge/scrapy-feedstock
against master which updates
recipe/meta.yaml
as follows:
-
Update the
versionstring. -
Update the
sha256sumandblake2b256values from the PyPI's source release tarball. -
Reset the
build/numberto0. This number is only increased when we re-build a package with no version change. -
Update the requirements if needed. This can be done by checking the requirements declared in
setup.py.
The pull request will trigger builds in Travis CI, Circle CI and Appveyor. This
may take a while depending on the conda-forge build queue. Once builds are OK,
the pull request is ready to merge and packages will be uploaded to the
conda-forge channel automatically.
Alternatively, you can wait for https://github.com/regro-cf-autotick-bot to create such a pull request, and just edit it as needed (you should be able to commit directly to it, even though the GitHub online editor).
Merge into the main branch
If you’ve released a patch version, you should merge its branch into the main branch (making sure you exclude version-bumping changes with e.g. git merge --no-commit if the version in master is higher than the released one).
Prepare the new Scrapy Cloud stack
As a different team produces images for Scrapy Cloud stacks, you should ping them to do that (preferably before the release so that they can plan in advance) and start the process by making a PR following the procedure at https://github.com/scrapinghub/scrapinghub-stack-scrapy#setting-up-local-environment-to-update-requirements.
Upgrade CI packages
To avoid the risk of CI suddenly failing and confusing pull request authors, we freeze some CI packages in tox.ini and .pre-commit-config.yaml.
In tox.ini you can find them looking for ==, but mind that some freezes are to be kept (e.g. to test minimum versions of Scrapy dependencies).
Right after a major or minor release would be a good time to update those packages, solving or allow-listing any issue raised by their new versions.