Optimize the loading of unit test fixtures.

Review Request #10378 — Created Jan. 11, 2019 and submitted


We have a number of optimizations in place for Django unit test
fixtures, designed to prevent unnecessary repeated deserialization of
fixture data. This change improves this further, reducing the work
needed to save the deserialized data to the database.

Previously, we'd call save() on every deserialized object for every
test run. This would perform an UPDATE call (to update a previous
database instance with the object's ID, to reflect the new state),
falling back to a new INSERT if the UPDATE didn't find any existing
rows. This meant, in the worst-case scenario, 2 SQL queries per fixture
object per unit test.

We now simply delete any existing entries all at once (if any exist) and
then proceed to bulk-create the replacements. This saves us all those
SQL queries, requiring only 2 per model instead of 2 per object per

On top of this, we no longer directly assign a list of objects to
ManyToManyFields. An assignment results in a SELECT, possible
DELETE, and an INSERT, and this is wasteful when we know we have new
entries. So instead, we call .add() on the relation so that only an
INSERT is done.

In testing against the Review Board unit test suite (of ~4000 unit
tests), this change resulted in a ~45 second reduction in test time with
no regressions introduced.

Djblets and Review Board unit tests pass.

  1. Ship It!
Review request changed

Status: Closed (submitted)

Change Summary:

Pushed to release-1.0.x (3268c22)