Reduce startup times for rbssh and make custom SSH backends persistant.

Review Request #11002 — Created April 13, 2020 and submitted — Latest diff uploaded

Information

Review Board
release-3.0.x

Reviewers

rbssh has historically executed within a Review Board environment in
order to take advantage of extension support, necessary in order for the
flexible SSH key storage backends to take effect. Originally, startup
time was fairly reasonable, but over time this became more expensive,
and on large deployments (particularly those using many extensions, or
expensive-to-load extensions) it could be a bottleneck.

This change reworks execution by running in a minimal Django
environment. It now uses a custom settings.py, which loads only the
bare-minimum Django apps needed to manage and implement SSH key storage
backends.

This means that extensions are no longer loaded. In order for an
extension-supplied storage backend to still work, the supplier of that
backend must set the siteconfig-provided ssh_storage_backend to the
path of the backend class. This supercedes the now-legacy
RBSSH_STORAGE_BACKEND key in settings.py. It also means that
extensions must be updated to set the new key in order to continue
working. Unlike the old process-local setting, the new setting
persists in the database, allowing rbssh to look it up without loading
any extensions.

Backend loading will try the new setting and the old setting, which will
help when performing SSH-backed requests from within the Review Board
process, or in rbssh if settings_local.py defines
RBSSH_STORAGE_BACKEND.

SSH communication within rbssh has also received some small speed
tweaks. We no longer attempt to log anything to debug if the
$DEBUG_RBSSH environment variable isn't set. We previously would send
to the logger, which would go through the internal logging processes and
then get ignored. We now just don't do anything.

NOTE: This is a breaking change:

  1. Extension developers providing custom SSH key storage backends will
    need to ensure their backends run within a bare-minimum environment.
    They can rely on the Django Site and Djblets SiteConfiguration
    models, but nothing else. If this impacts anyone, they will need to
    talk to us about a future-proof solution, but this change is not
    currently attempting to over-engineer this, since we're only aware of
    the custom backend provided by Power Pack.

  2. Power Pack users will need to update to Power Pack 3.0.3 (still
    in-development at the time of this change) in order to continue using
    DB-backend SSH keys, or will need to set the following in
    settings_local.py:

RBSSH_STORAGE_BACKEND = 'rbpowerpack.sshdb.storage.DBSSHStorage'

Unit tests pass.

Manually invoked rbssh, running speed tests and comparing the old
execution time to the new one. Saw a drastic speed reduction, particularly
with extensions enabled.

Tested usage from within Review Board.

Tested with the default local keys backend and the Power Pack SSHDB backend
(using an updated Power Pack with support for the new setting), verifying
that the INSTALLED_APPS changes didn't break anything.

Tested these changes on both Review Board 3.0 and 4.0.

Commits

Files