Speed up unit tests by about 80%.

Review Request #11247 — Created Oct. 26, 2020 and submitted — Latest diff uploaded

Information

Django Evolution
master

Reviewers

Django Evolution has historically needed to run most operations in a
TransactionTestCase, in order to ensure a full database wipe
in-between each test run, and to allow transactions to be set up within
a test. This is very slow, since a database wipe takes significant time
that really adds up.

Most projects use TestCase instead, which is much faster but runs
everything in a transaction. While that's fine for some tests, most
tests need to set up their own top-level transactions, so it's not
really suitable for this project.

With this change, we're using a new approach for unit tests. We're using
TestCase as a base, but turning off all transaction management. This
means that every unit test is, effectively, sharing the same database.
Tests aren't as isolated, and in theory this could lead to data leakage
issues between tests. In practice, though, this won't be much of a
problem, for two reasons:

  1. We're clearing out/installing the only project-wide tables
    (Evolution and Version) on each test run, preventing that state
    from leaking.

  2. Most tests were already setting up their test models and then
    deleting them as part of their test run, using ensure_test_db().
    That prevents any non-project-wide tables from remaining between unit
    tests.

There are some changes that were required for a few unit tests.
Previously, a full migrate/evolve would have run at the start of each
test for all apps (which is also slow -- especially the migrations
step), but now we only bootstrap Evolution and Version, so any unit
tests that need an app bootstrapped now has to do this manually with
ensure_evolved_apps(). This actually simplified some code, though,
since we had tests that had to clear out unwanted state first, and now
that state just simply isn't there at the start.

This significantly speeds up the test suite. While this will differ
between versions of Django and Python, on average it speds up the test
suite by about 80%, bringing some test run times down from 9 minutes to
less than 1 or 2 in my local tests. This is with a remote database, so
testing against a local database may be even faster.

Unit tests pass for all supported versions of Python and Django across
all databases.

Commits

Files