Optimize the loading of unit test fixtures.
Review Request #10378 — Created Jan. 11, 2019 and submitted
We have a number of optimizations in place for Django unit test
fixtures, designed to prevent unnecessary repeated deserialization of
fixture data. This change improves this further, reducing the work
needed to save the deserialized data to the database.Previously, we'd call
save()
on every deserialized object for every
test run. This would perform anUPDATE
call (to update a previous
database instance with the object's ID, to reflect the new state),
falling back to a newINSERT
if theUPDATE
didn't find any existing
rows. This meant, in the worst-case scenario, 2 SQL queries per fixture
object per unit test.We now simply delete any existing entries all at once (if any exist) and
then proceed to bulk-create the replacements. This saves us all those
SQL queries, requiring only 2 per model instead of 2 per object per
model.On top of this, we no longer directly assign a list of objects to
ManyToManyField
s. An assignment results in aSELECT
, possible
DELETE
, and anINSERT
, and this is wasteful when we know we have new
entries. So instead, we call.add()
on the relation so that only an
INSERT
is done.In testing against the Review Board unit test suite (of ~4000 unit
tests), this change resulted in a ~45 second reduction in test time with
no regressions introduced.
Djblets and Review Board unit tests pass.