• 
      

    Optimize queries when fetching objects in the API.

    Review Request #4826 — Created Oct. 21, 2013 and submitted

    Information

    Djblets
    master

    Reviewers

    Optimize queries when fetching objects in the API.

    Fetching lists could be wildly inefficient in terms of query counts, due
    to related objects for each object being fetched when serializing items.
    select_related() didn't help us here, as it can't traverse
    ManyToManyFields.

    We now use Django's somewhat new prefetch_related() to fetch all related
    fields that we know will be serialized when getting a list resource. We
    start off by building and caching a list of fields, based on the
    resource's fields dictionary, and using that when fetching the list.

    A new _get_queryset method handles this logic. It's called internally
    instead of calling get_queryset() and tacking on select_related().

    This dramatically cuts down on some query counts. It doesn't make as big
    a difference in most unit tests, as they don't tend to create large
    batches of objects for testing, but in real-world usage, there's a
    noticeable difference.

    Ran a large series of tests, before/after, with various optimization
    attempts. Also ran lots of manual tests with the more expensive
    resources in Review Board (such as the Review Request resource).

    In automated tests, given how few tests created multiple objects,
    there weren't savings all around, but there were never more queries
    than we had for any test. Many actually did go down by a few.

    The biggest savings were in a more real production test. Here, I saw
    query counts go from 100+ to 9, or so. These resources had a fixed
    cost, and not one that scaled with the number of items.

    david
    1. Ship It!

    2. 
        
    chipx86
    Review request changed
    Status:
    Completed