Switch Djblets's packaging to pyproject.toml.

Review Request #14137 — Created Sept. 3, 2024 and submitted — Latest diff uploaded

Information

Djblets
release-5.x

Reviewers

Djblets has historically been a setuptools-based project, relying
heavily on the dynamic ability of setup.py. Over the years, the Python
ecosystem has moved to pyproject.toml files, with pluggable build
backends. This has reached a point of maturity, and pip will soon
remove support for installing either production or editable installs
from a legacy source tree.

In theory, modernization requires just providing a pyproject.toml that
specifies setuptools as a build backend. A project can still use
setup.py for the project definition and dynamic capabilities. However,
since packages are also now built in a virtualenv, it's become clear
that we needed to address about our packaging.

We now use pyproject.toml to define most of the metadata of the
package. The build backend is then a specialization of
setuptools.build_meta, living in build-backend.py at the root of the
tree. This specializes a few things about our build process:

  1. It uses all of Djblets's dependencies as build-system dependencies,
    needed in order to build static media for the package.

  2. It builds the static media, including them in both sdist and wheel
    distributions.

  3. It writes out a package-requirements.txt at build time with the
    dependencies from djblets/dependencies.py, which setuptools can
    then consume. This is also bundled in the sdist.

With this, we no longer need setup.py, and constrain all custom logic
to build-backend.py.

Some things to note:

  1. python -m build . will default to building an sdist and then a
    wheel from that sdist, whereas build . -w will build a wheel
    straight from the tree. Due to differences in the prep stages, and
    what's built from what, we need to micromanage some state (like
    package-requirements.txt) in different places.

  2. Other build backends (hatch, PDM, flit, and poetry) were evaluated
    and discarded. These are great backends, but don't solve the core
    problems we've had to work around out of the box, and sort of want to
    manage more of the development and build process. setuptools is a
    known entity for us, and will be needed for extension packaging
    anyway, so we're sticking with it.

  3. Makefile installs the package in editable-compat mode. This uses
    standard .pth files instead of newer setuptools-based import
    hooks, in order to allow mypy and pyright to find type
    definitions. See https://github.com/python/mypy/issues/13392 and
    https://github.com/pypa/setuptools/issues/3518.

Tested isolated builds in the following setups:

  • python -m build . (builds sdist, then a wheel from the sdist)
  • python -m build . -s (builds sdist)
  • python -m build . -w (builds wheel)

Tested isolated editable installs using pip install -e . and
non-isolated editable installs using pip install -e --no-build-isolation
and with make develop.

Performed full tree diffs of generated wheels from ./setup.py bdist_wheel
(prior to this change) and both wheel-producing python -m build methods.
Verified that all content was identical, with the exception of differences
in source map paths and some modern metadata for the package.

Changes between revision 4 and 5

orig
1
2
3
4
5

Commits

Summary ID Author
Switch Djblets's packaging to pyproject.toml.
Djblets has historically been a setuptools-based project, relying heavily on the dynamic ability of `setup.py`. Over the years, the Python ecosystem has moved to `pyproject.toml` files, with pluggable build backends. This has reached a point of maturity, and `pip` will soon remove support for installing either production or editable installs from a legacy source tree. In theory, modernization requires just providing a `pyproject.toml` that specifies `setuptools` as a build backend. A project can still use `setup.py` for the project definition and dynamic capabilities. However, since packages are also now built in a virtualenv, it's become clear that we needed to address about our packaging. We now use `pyproject.toml` to define most of the metadata of the package. The build backend is then a specialization of `setuptools.build_meta`, living in `build-backend.py` at the root of the tree. This specializes a few things about our build process: 1. It uses all of Djblets's dependencies as build-system dependencies, needed in order to build static media for the package. 2. It builds the static media, including them in both sdist and wheel distributions. 3. It writes out a `package-requirements.txt` at build time with the dependencies from `djblets/dependencies.py`, which setuptools can then consume. This is also bundled in the sdist. With this, we no longer need `setup.py`, and constrain all custom logic to `build-backend.py`. Some things to note: 1. `python -m build .` will default to building an sdist and then a wheel from that sdist, whereas `build . -w` will build a wheel straight from the tree. Due to differences in the prep stages, and what's built from what, we need to micromanage some state (like `package-requirements.txt`) in different places. 2. Other build backends (hatch, PDM, flit, and poetry) were evaluated and discarded. These are great backends, but don't solve the core problems we've had to work around out of the box, and sort of want to manage more of the development and build process. `setuptools` is a known entity for us, and will be needed for extension packaging anyway, so we're sticking with it.
c80f60e84f90cf9135837dcde4e2310e9ff1fee0 Christian Hammond
Switch Djblets's packaging to pyproject.toml.
Djblets has historically been a setuptools-based project, relying heavily on the dynamic ability of `setup.py`. Over the years, the Python ecosystem has moved to `pyproject.toml` files, with pluggable build backends. This has reached a point of maturity, and `pip` will soon remove support for installing either production or editable installs from a legacy source tree. In theory, modernization requires just providing a `pyproject.toml` that specifies `setuptools` as a build backend. A project can still use `setup.py` for the project definition and dynamic capabilities. However, since packages are also now built in a virtualenv, it's become clear that we needed to address about our packaging. We now use `pyproject.toml` to define most of the metadata of the package. The build backend is then a specialization of `setuptools.build_meta`, living in `build-backend.py` at the root of the tree. This specializes a few things about our build process: 1. It uses all of Djblets's dependencies as build-system dependencies, needed in order to build static media for the package. 2. It builds the static media, including them in both sdist and wheel distributions. 3. It writes out a `package-requirements.txt` at build time with the dependencies from `djblets/dependencies.py`, which setuptools can then consume. This is also bundled in the sdist. With this, we no longer need `setup.py`, and constrain all custom logic to `build-backend.py`. Some things to note: 1. `python -m build .` will default to building an sdist and then a wheel from that sdist, whereas `build . -w` will build a wheel straight from the tree. Due to differences in the prep stages, and what's built from what, we need to micromanage some state (like `package-requirements.txt`) in different places. 2. Other build backends (hatch, PDM, flit, and poetry) were evaluated and discarded. These are great backends, but don't solve the core problems we've had to work around out of the box, and sort of want to manage more of the development and build process. `setuptools` is a known entity for us, and will be needed for extension packaging anyway, so we're sticking with it. 3. `Makefile` installs the package in editable-compat mode. This uses standard `.pth` files instead of newer `setuptools`-based import hooks, in order to allow `mypy` and `pyright` to find type definitions. See https://github.com/python/mypy/issues/13392 and https://github.com/pypa/setuptools/issues/3518.
47577360a1a78dbbb25249f077410668dfbc3372 Christian Hammond

Files

Makefile
Loading...