• 
      

    Speed up unit tests by about 80%.

    Review Request #11247 — Created Oct. 26, 2020 and submitted — Latest diff uploaded

    Information

    Django Evolution
    master

    Reviewers

    Django Evolution has historically needed to run most operations in a
    TransactionTestCase, in order to ensure a full database wipe
    in-between each test run, and to allow transactions to be set up within
    a test. This is very slow, since a database wipe takes significant time
    that really adds up.

    Most projects use TestCase instead, which is much faster but runs
    everything in a transaction. While that's fine for some tests, most
    tests need to set up their own top-level transactions, so it's not
    really suitable for this project.

    With this change, we're using a new approach for unit tests. We're using
    TestCase as a base, but turning off all transaction management. This
    means that every unit test is, effectively, sharing the same database.
    Tests aren't as isolated, and in theory this could lead to data leakage
    issues between tests. In practice, though, this won't be much of a
    problem, for two reasons:

    1. We're clearing out/installing the only project-wide tables
      (Evolution and Version) on each test run, preventing that state
      from leaking.

    2. Most tests were already setting up their test models and then
      deleting them as part of their test run, using ensure_test_db().
      That prevents any non-project-wide tables from remaining between unit
      tests.

    There are some changes that were required for a few unit tests.
    Previously, a full migrate/evolve would have run at the start of each
    test for all apps (which is also slow -- especially the migrations
    step), but now we only bootstrap Evolution and Version, so any unit
    tests that need an app bootstrapped now has to do this manually with
    ensure_evolved_apps(). This actually simplified some code, though,
    since we had tests that had to clear out unwanted state first, and now
    that state just simply isn't there at the start.

    This significantly speeds up the test suite. While this will differ
    between versions of Django and Python, on average it speds up the test
    suite by about 80%, bringing some test run times down from 9 minutes to
    less than 1 or 2 in my local tests. This is with a remote database, so
    testing against a local database may be even faster.

    Unit tests pass for all supported versions of Python and Django across
    all databases.

    Commits

    Files