Speed up unit tests by about 80%.

Review Request #11247 — Created Oct. 26, 2020 and submitted

Information

Django Evolution
master

Reviewers

Django Evolution has historically needed to run most operations in a
TransactionTestCase, in order to ensure a full database wipe
in-between each test run, and to allow transactions to be set up within
a test. This is very slow, since a database wipe takes significant time
that really adds up.

Most projects use TestCase instead, which is much faster but runs
everything in a transaction. While that's fine for some tests, most
tests need to set up their own top-level transactions, so it's not
really suitable for this project.

With this change, we're using a new approach for unit tests. We're using
TestCase as a base, but turning off all transaction management. This
means that every unit test is, effectively, sharing the same database.
Tests aren't as isolated, and in theory this could lead to data leakage
issues between tests. In practice, though, this won't be much of a
problem, for two reasons:

  1. We're clearing out/installing the only project-wide tables
    (Evolution and Version) on each test run, preventing that state
    from leaking.

  2. Most tests were already setting up their test models and then
    deleting them as part of their test run, using ensure_test_db().
    That prevents any non-project-wide tables from remaining between unit
    tests.

There are some changes that were required for a few unit tests.
Previously, a full migrate/evolve would have run at the start of each
test for all apps (which is also slow -- especially the migrations
step), but now we only bootstrap Evolution and Version, so any unit
tests that need an app bootstrapped now has to do this manually with
ensure_evolved_apps(). This actually simplified some code, though,
since we had tests that had to clear out unwanted state first, and now
that state just simply isn't there at the start.

This significantly speeds up the test suite. While this will differ
between versions of Django and Python, on average it speds up the test
suite by about 80%, bringing some test run times down from 9 minutes to
less than 1 or 2 in my local tests. This is with a remote database, so
testing against a local database may be even faster.

Unit tests pass for all supported versions of Python and Django across
all databases.

Summary ID
Speed up unit tests by about 80%.
Django Evolution has historically needed to run most operations in a `TransactionTestCase`, in order to ensure a full database wipe in-between each test run, and to allow transactions to be set up within a test. This is very slow, since a database wipe takes significant time that really adds up. Most projects use `TestCase` instead, which is much faster but runs everything in a transaction. While that's fine for some tests, most tests need to set up their own top-level transactions, so it's not really suitable for this project. With this change, we're using a new approach for unit tests. We're using `TestCase` as a base, but turning off all transaction management. This means that every unit test is, effectively, sharing the same database. Tests aren't as isolated, and in theory this could lead to data leakage issues between tests. In practice, though, this won't be much of a problem, for two reasons: 1. We're clearing out/installing the only project-wide tables (`Evolution` and `Version`) on each test run, preventing that state from leaking. 2. Most tests were already setting up their test models and then deleting them as part of their test run, using `ensure_test_db()`. That prevents any non-project-wide tables from remaining between unit tests. There are some changes that were required for a few unit tests. Previously, a full migrate/evolve would have run at the start of each test for all apps (which is also slow -- especially the migrations step), but now we only bootstrap `Evolution` and `Version`, so any unit tests that need an app bootstrapped now has to do this manually with `ensure_evolved_apps()`. This actually simplified some code, though, since we had tests that had to clear out unwanted state first, and now that state just simply isn't there at the start. This significantly speeds up the test suite. While this will differ between versions of Django and Python, on average it speds up the test suite by about 80%, bringing some test run times down from 9 minutes to less than 1 or 2 in my local tests. This is with a remote database, so testing against a local database may be even faster.
f8b933f42f449605dfc183a91ccd7d8f9f40c6e8
Description From Last Updated

E722 do not use bare except'

reviewbotreviewbot
Checks run (1 failed, 1 succeeded)
flake8 failed.
JSHint passed.

flake8

chipx86
david
  1. Ship It!
  2. 
      
chipx86
Review request changed
Status:
Completed
Change Summary:
Pushed to master (98336f3)