Speed up unit tests by about 80%.

Review Request #11247 — Created Oct. 26, 2020 and submitted — Latest diff uploaded

Information

Django Evolution
master

Reviewers

Django Evolution has historically needed to run most operations in a
TransactionTestCase, in order to ensure a full database wipe
in-between each test run, and to allow transactions to be set up within
a test. This is very slow, since a database wipe takes significant time
that really adds up.

Most projects use TestCase instead, which is much faster but runs
everything in a transaction. While that's fine for some tests, most
tests need to set up their own top-level transactions, so it's not
really suitable for this project.

With this change, we're using a new approach for unit tests. We're using
TestCase as a base, but turning off all transaction management. This
means that every unit test is, effectively, sharing the same database.
Tests aren't as isolated, and in theory this could lead to data leakage
issues between tests. In practice, though, this won't be much of a
problem, for two reasons:

  1. We're clearing out/installing the only project-wide tables
    (Evolution and Version) on each test run, preventing that state
    from leaking.

  2. Most tests were already setting up their test models and then
    deleting them as part of their test run, using ensure_test_db().
    That prevents any non-project-wide tables from remaining between unit
    tests.

There are some changes that were required for a few unit tests.
Previously, a full migrate/evolve would have run at the start of each
test for all apps (which is also slow -- especially the migrations
step), but now we only bootstrap Evolution and Version, so any unit
tests that need an app bootstrapped now has to do this manually with
ensure_evolved_apps(). This actually simplified some code, though,
since we had tests that had to clear out unwanted state first, and now
that state just simply isn't there at the start.

This significantly speeds up the test suite. While this will differ
between versions of Django and Python, on average it speds up the test
suite by about 80%, bringing some test run times down from 9 minutes to
less than 1 or 2 in my local tests. This is with a remote database, so
testing against a local database may be even faster.

Unit tests pass for all supported versions of Python and Django across
all databases.

Diff Revision 1

This is not the most recent revision of the diff. The latest diff is revision 2. See what's changed.

orig
1
2

Commits

First Last Summary ID Author
Speed up unit tests by about 80%.
Django Evolution has historically needed to run most operations in a `TransactionTestCase`, in order to ensure a full database wipe in-between each test run, and to allow transactions to be set up within a test. This is very slow, since a database wipe takes significant time that really adds up. Most projects use `TestCase` instead, which is much faster but runs everything in a transaction. While that's fine for some tests, most tests need to set up their own top-level transactions, so it's not really suitable for this project. With this change, we're using a new approach for unit tests. We're using `TestCase` as a base, but turning off all transaction management. This means that every unit test is, effectively, sharing the same database. Tests aren't as isolated, and in theory this could lead to data leakage issues between tests. In practice, though, this won't be much of a problem, for two reasons: 1. We're clearing out/installing the only project-wide tables (`Evolution` and `Version`) on each test run, preventing that state from leaking. 2. Most tests were already setting up their test models and then deleting them as part of their test run, using `ensure_test_db()`. That prevents any non-project-wide tables from remaining between unit tests. There are some changes that were required for a few unit tests. Previously, a full migrate/evolve would have run at the start of each test for all apps (which is also slow -- especially the migrations step), but now we only bootstrap `Evolution` and `Version`, so any unit tests that need an app bootstrapped now has to do this manually with `ensure_evolved_apps()`. This actually simplified some code, though, since we had tests that had to clear out unwanted state first, and now that state just simply isn't there at the start. This significantly speeds up the test suite. While this will differ between versions of Django and Python, on average it speds up the test suite by about 80%, bringing some test run times down from 9 minutes to less than 1 or 2 in my local tests. This is with a remote database, so testing against a local database may be even faster.
fea8893e359068bd2802b3c448ebb7500db372b7 Christian Hammond
django_evolution/tests/base_test_case.py
django_evolution/tests/test_evolution_utils.py
django_evolution/tests/test_evolver.py
Loading...