Skip SSH tests when a system-wide executable is missing.

Review Request #12147 — Created March 14, 2022 and submitted

Information

Review Board
release-5.0.x

Reviewers

A long-standing, annoying issue that has plagued many of us developing
on Review Board have been Bazaar SSH unit tests failing with Repository
Not Found errors.

This is actually due to Bazaar not being installed locally in a
directory part of the system path established when running a command via
SSH, without the shell's configuration setting up a normal path. For
instance, ssh localhost bzr won't result in a ~/.zshrc being
executed (just a ~/.zshenv). It certainly won't be found in a
virtualenv. This leads to the error.

This is actually not limited to Bazaar. CVS, Git, and Subversion tests
are suceptible to the same problem, but are more likely to have their
corresponding command line tools available in standard system paths.

This change reworks the SSH unit tests to perform tool lookups in order
to avoid this issue. Now, when computing whether SSH tests can be
performed for a given setup, it will SSH locally and run which <tool>
commands, caching results. If a tool can't be found, tests will be
skippped, along with a helpful error message aimed at resolving the
problem.

This requires opt-in from the test suites. ssh_required_system_exes
can be set to a list of command line tools needed for a successful SSH
test run. All will be checked, and if any are missing, the tests will
fail.

This should avoid annoying, hard-to-diagnose test failures going
forward.

Altered the system path on login (in ~/.zshenv) to avoid having any
useful directories.

Ran the test suite before this fix and saw errors with the SSH tests.

Ran after the fix and saw skip messages instead.

Added all necessary paths and re-ran. Saw 100% test completion.

Using pytest -rs to verify the presence and content of skip messages.

Summary ID
Skip SSH tests when a system-wide executable is missing.
A long-standing, annoying issue that has plagued many of us developing on Review Board have been Bazaar SSH unit tests failing with Repository Not Found errors. This is actually due to Bazaar not being installed locally in a directory part of the system path established when running a command via SSH, without the shell's configuration setting up a normal path. For instance, `ssh localhost bzr` won't result in a `~/.zshrc` being executed (just a `~/.zshenv`). It certainly won't be found in a virtualenv. This leads to the error. This is actually not limited to Bazaar. CVS, Git, and Subversion tests are suceptible to the same problem, but are more likely to have their corresponding command line tools available in standard system paths. This change reworks the SSH unit tests to perform tool lookups in order to avoid this issue. Now, when computing whether SSH tests can be performed for a given setup, it will SSH locally and run `which <tool>` commands, caching results. If a tool can't be found, tests will be skippped, along with a helpful error message aimed at resolving the problem. This requires opt-in from the test suites. `ssh_required_system_exes` can be set to a list of command line tools needed for a successful SSH test run. All will be checked, and if any are missing, the tests will fail. This should avoid annoying, hard-to-diagnose test failures going forward.
72c0b8e3b0308c16e0dc31fac82bd07b11f543ce
david
  1. Ship It!
  2. 
      
chipx86
Review request changed
Status:
Completed
Change Summary:
Pushed to release-5.0.x (1ed45e0)