Summary

Make Djblets package develop mode compatible with wheels and npm packages.

Review Request #8493 — Created Oct. 26, 2016 and submitted Oct. 27, 2016, 4:32 p.m.

Information

Owner

chipx86

Repository

Djblets

Branch

release-0.10.x

Bugs

Depends On

Blocks

8494

Commit

0645a30...

Reviewers

Groups

djblets

People

Description

Djblets's packaging, and the world of Python packaging in general, has
gotten increasingly complex with time. Wheels are the preferred (and
better supported) format in the packaging world, and eggs are on their
way out. We also require a handful of node.js modules for static media
building.

This change adds significant improvements to our packaging setup, in
order to cover these issues and more.

setup.py is now driven almost entirely by metadata within the package.
It no longer lists dependencies, but rather makes use of a new
djblets.dependencies module, which lists all the npm and Python package
dependencies needed by Djblets. This allows for a couple of things.
Primarily, it allows us to consolidate all dependencies/version ranges
in one place. It also allows consumers of Djblets to refer to the same
versions for their own uses.

The setup.py develop command has been enhanced to install dependencies
via wheels. It does this by first ensuring we have a modern
pip/setuptools, and then invokes pip install -e ., which will perform
the equivalent of setup.py develop but with Wheels. (Behind the
scenes, it actually uses setup.py develop with shims to install the
wheel versions, so it's otherwise fully compatible.)

After that, it installs all the packages required for development (using
dev-requirements.txt), and then installs required modules from npm
(or npm-cache, if --use-npm-cache is installed).

The end result is that contributors to Djblets now only need to run
setup.py develop to have a fully working environment going.

Testing Done

Set up a brand new virtual environment and started fresh in the Djblets

source tree (no Djblets.egg-info, node_modules, dist, or build).
Ran ./setup.py develop. Saw that it upgraded setuptools and pip

and then installed the main package dependencies and development

dependencies. Following that, it installed the node.js modules.
Checked that all expected packages and files were installed. Successfully

ran all unit tests.

Issues

Description	From	Last Updated
Col: 1 E402 module level import not at top of file	reviewbot	Oct. 26, 2016, 2:15 p.m.
Col: 1 E402 module level import not at top of file	reviewbot	Oct. 26, 2016, 2:15 p.m.
Col: 1 E402 module level import not at top of file	reviewbot	Oct. 26, 2016, 2:15 p.m.
We are requiring pip, so I think it is safe to assume that setuptools exists on the machine. Can we …	brennie	Oct. 26, 2016, 2:15 p.m.
Docstring?	david	Oct. 26, 2016, 2:07 p.m.
Docstring?	david	Oct. 26, 2016, 2:07 p.m.
Docstring?	david	Oct. 26, 2016, 2:14 p.m.
Docstring?	david	Oct. 26, 2016, 2:14 p.m.
Docstring?	david	Oct. 26, 2016, 2:14 p.m.
Docstring?	david	Oct. 26, 2016, 2:15 p.m.
Col: 1 E402 module level import not at top of file	reviewbot	Oct. 26, 2016, 2:29 p.m.
Col: 1 E402 module level import not at top of file	reviewbot	Oct. 26, 2016, 2:29 p.m.
Col: 1 E402 module level import not at top of file	reviewbot	Oct. 26, 2016, 2:29 p.m.
Col: 1 E402 module level import not at top of file	reviewbot	Oct. 26, 2016, 3:28 p.m.
Col: 1 E402 module level import not at top of file	reviewbot	Oct. 26, 2016, 3:28 p.m.
Col: 1 E402 module level import not at top of file	reviewbot	Oct. 26, 2016, 3:27 p.m.
Col: 1 E402 module level import not at top of file	reviewbot	Oct. 27, 2016, 1:04 a.m.
'pkg_resources' imported but unused	reviewbot	Oct. 27, 2016, 1:04 a.m.
Col: 1 E402 module level import not at top of file	reviewbot	Oct. 26, 2016, 4:03 p.m.
Col: 1 E402 module level import not at top of file	reviewbot	Oct. 26, 2016, 4:03 p.m.
Col: 1 E402 module level import not at top of file	reviewbot	Oct. 26, 2016, 4:03 p.m.

Tool: Pyflakes
Processed Files:
    djblets/dependencies.py
    setup.py
    djblets/__init__.py
    docs/djblets/conf.py
    docs/releasenotes/conf.py

Ignored Files:
    package.json



Tool: PEP8 Style Checker
Processed Files:
    djblets/dependencies.py
    setup.py
    djblets/__init__.py
    docs/djblets/conf.py
    docs/releasenotes/conf.py

Ignored Files:
    package.json

docs/djblets/conf.py (Diff revision 1)
The issue has been dropped. Show all issues
```
Col: 1
 E402 module level import not at top of file
```
docs/djblets/conf.py (Diff revision 1)
The issue has been dropped. Show all issues
```
Col: 1
 E402 module level import not at top of file
```
docs/releasenotes/conf.py (Diff revision 1)
The issue has been dropped. Show all issues
```
Col: 1
 E402 module level import not at top of file
```

setup.py (Diff revision 1)
The issue has been resolved. Show all issues
```
Docstring?
```
setup.py (Diff revision 1)
The issue has been resolved. Show all issues
```
Docstring?
```
setup.py (Diff revision 1)
The issue has been resolved. Show all issues
```
Docstring?
```
setup.py (Diff revision 1)
The issue has been resolved. Show all issues
```
Docstring?
```
setup.py (Diff revision 1)
The issue has been resolved. Show all issues
```
Docstring?
```
setup.py (Diff revision 1)
The issue has been resolved. Show all issues
```
Docstring?
```

setup.py (Diff revision 1)

The issue has been dropped. Show all issues

We are requiring pip, so I think it is safe to assume that setuptools exists on the machine. Can we just nuke this?

chipx86 Oct. 26, 2016, 2:15 p.m.

I'll take care of this in a follow-up change.

Change Summary:


Added docstrings.
Added a --no-npm flag to setup.py develop.

Commit:

-	634be0889ec1e085941d31756abcfd32aa3dcc4e
+	f5a849a7dce0a0212223e04b93ea030bcc70af99

Diff:

Revision 2 (+228 -43)

Show changes

	package.json
	setup.py
	djblets/__init__.py
	djblets/dependencies.py
	docs/djblets/conf.py
	docs/releasenotes/conf.py

Tool: Pyflakes
Processed Files:
    djblets/dependencies.py
    setup.py
    djblets/__init__.py
    docs/djblets/conf.py
    docs/releasenotes/conf.py

Ignored Files:
    package.json



Tool: PEP8 Style Checker
Processed Files:
    djblets/dependencies.py
    setup.py
    djblets/__init__.py
    docs/djblets/conf.py
    docs/releasenotes/conf.py

Ignored Files:
    package.json

docs/djblets/conf.py (Diff revision 2)
The issue has been dropped. Show all issues
```
Col: 1
 E402 module level import not at top of file
```
docs/djblets/conf.py (Diff revision 2)
The issue has been dropped. Show all issues
```
Col: 1
 E402 module level import not at top of file
```
docs/releasenotes/conf.py (Diff revision 2)
The issue has been dropped. Show all issues
```
Col: 1
 E402 module level import not at top of file
```

Change Summary:

npm-cache is now only used if passing --use-npm-cache.

Description:

		Djblets's packaging, and the world of Python packaging in general, has
		gotten increasingly complex with time. Wheels are the preferred (and
		better supported) format in the packaging world, and eggs are on their
		way out. We also require a handful of node.js modules for static media
		building.

		This change adds significant improvements to our packaging setup, in
		order to cover these issues and more.

		setup.py is now driven almost entirely by metadata within the package.
		It no longer lists dependencies, but rather makes use of a new
		`djblets.dependencies` module, which lists all the npm and Python package
		dependencies needed by Djblets. This allows for a couple of things.
		Primarily, it allows us to consolidate all dependencies/version ranges
		in one place. It also allows consumers of Djblets to refer to the same
		versions for their own uses.

		The `setup.py develop` command has been enhanced to install dependencies
		via wheels. It does this by first ensuring we have a modern
		pip/setuptools, and then invokes `pip install -e .`, which will perform
		the equivalent of `setup.py develop` but with Wheels. (Behind the
		scenes, it actually uses `setup.py develop` with shims to install the
		wheel versions, so it's otherwise fully compatible.)

		After that, it installs all the packages required for development (using
		dev-requirements.txt), and then installs required modules from npm
~		(preferred npm-cache, if installed, to speed up installation).
	~	(or npm-cache, if `--use-npm-cache` is installed).

		The end result is that contributors to Djblets now only need to run
		`setup.py develop` to have a fully working environment going.

Commit:

-	f5a849a7dce0a0212223e04b93ea030bcc70af99
+	99c794b7d451a51dba693f7c617360a362c8ebc0

Diff:

Revision 3 (+238 -43)

Show changes

	package.json
	setup.py
	djblets/__init__.py
	djblets/dependencies.py
	docs/djblets/conf.py
	docs/releasenotes/conf.py

Tool: Pyflakes
Processed Files:
    djblets/dependencies.py
    setup.py
    djblets/__init__.py
    docs/djblets/conf.py
    docs/releasenotes/conf.py

Ignored Files:
    package.json



Tool: PEP8 Style Checker
Processed Files:
    djblets/dependencies.py
    setup.py
    djblets/__init__.py
    docs/djblets/conf.py
    docs/releasenotes/conf.py

Ignored Files:
    package.json

docs/djblets/conf.py (Diff revision 3)
The issue has been dropped. Show all issues
```
Col: 1
 E402 module level import not at top of file
```
docs/djblets/conf.py (Diff revision 3)
The issue has been dropped. Show all issues
```
Col: 1
 E402 module level import not at top of file
```
docs/releasenotes/conf.py (Diff revision 3)
The issue has been dropped. Show all issues
```
Col: 1
 E402 module level import not at top of file
```

Change Summary:


Fixed media building due to a bad variable reference.
Fixed building packages with wheels. (We got lucky in that we always built eggs before wheels.)

Commit:

-	99c794b7d451a51dba693f7c617360a362c8ebc0
+	0645a30dbde21cfcbfd4835e9a4399dcd5e204f6

Diff:

Revision 4 (+244 -45)

Show changes

	package.json
	setup.py
	contrib/internal/build-media.py
	djblets/__init__.py
	djblets/dependencies.py
	docs/djblets/conf.py
	docs/releasenotes/conf.py

Tool: Pyflakes
Processed Files:
    djblets/dependencies.py
    docs/releasenotes/conf.py
    docs/djblets/conf.py
    contrib/internal/build-media.py
    setup.py
    djblets/__init__.py

Ignored Files:
    package.json



Tool: PEP8 Style Checker
Processed Files:
    djblets/dependencies.py
    docs/releasenotes/conf.py
    docs/djblets/conf.py
    contrib/internal/build-media.py
    setup.py
    djblets/__init__.py

Ignored Files:
    package.json

contrib/internal/build-media.py (Diff revision 4)
The issue has been dropped. Show all issues
```
Col: 1
 E402 module level import not at top of file
```
contrib/internal/build-media.py (Diff revision 4)
The issue has been dropped. Show all issues
```
 'pkg_resources' imported but unused
```
docs/djblets/conf.py (Diff revision 4)
The issue has been dropped. Show all issues
```
Col: 1
 E402 module level import not at top of file
```
docs/djblets/conf.py (Diff revision 4)
The issue has been dropped. Show all issues
```
Col: 1
 E402 module level import not at top of file
```
docs/releasenotes/conf.py (Diff revision 4)
The issue has been dropped. Show all issues
```
Col: 1
 E402 module level import not at top of file
```

Ship it!

```
Ship It!
```

Status: Closed (submitted)

Change Summary:

Pushed to release-0.10.x (cd80e9b)