Optimize queries when fetching objects in the API.

Review Request #4826 — Created Oct. 21, 2013 and submitted

Information

Djblets
master

Reviewers

Optimize queries when fetching objects in the API.

Fetching lists could be wildly inefficient in terms of query counts, due
to related objects for each object being fetched when serializing items.
select_related() didn't help us here, as it can't traverse
ManyToManyFields.

We now use Django's somewhat new prefetch_related() to fetch all related
fields that we know will be serialized when getting a list resource. We
start off by building and caching a list of fields, based on the
resource's fields dictionary, and using that when fetching the list.

A new _get_queryset method handles this logic. It's called internally
instead of calling get_queryset() and tacking on select_related().

This dramatically cuts down on some query counts. It doesn't make as big
a difference in most unit tests, as they don't tend to create large
batches of objects for testing, but in real-world usage, there's a
noticeable difference.

Ran a large series of tests, before/after, with various optimization
attempts. Also ran lots of manual tests with the more expensive
resources in Review Board (such as the Review Request resource).

In automated tests, given how few tests created multiple objects,
there weren't savings all around, but there were never more queries
than we had for any test. Many actually did go down by a few.

The biggest savings were in a more real production test. Here, I saw
query counts go from 100+ to 9, or so. These resources had a fixed
cost, and not one that scaled with the number of items.

david
  1. Ship It!

  2. 
      
chipx86
Review request changed

Status: Closed (submitted)

Loading...