• 
      

    Add smarter datagrid pagination, query augmentation, and distinct handling.

    Review Request #13367 — Created Oct. 22, 2023 and submitted — Latest diff uploaded

    Information

    Djblets
    release-3.x

    Reviewers

    Datagrids have historically used the same queryset for both page data
    and for pagination. This led to expensive pagination detection, as the
    nature of the paginator and Django's ORM results in the total page count
    be computed as a SELECT COUNT(*) FROM (<subquery>), which is very
    slow and gets worse the more data is matched.

    To resolve this, we're now using dual querysets. One queryset is
    responsible purely for the filtering. The other builds on this and is
    responsible for the page data.

    We use a new DataGridPaginator class to handle this, since Django's
    paginator doesn't give us any way to set the total item count. This
    class gives us that ability. It's now deprecated for subclasses to
    return a paginator instance that is not a subclass of
    DataGridPaginator, but for the moment, it's still supported.

    The old way of augmenting querysets through the datagrid or columns is
    also deprecated, as it was never built for this dual queryset method in
    mind. Instead, we have two new augmentation pathways: One for filtering,
    one for data collection. The filtering path is used for both querysets,
    and the data collection path is only used for the page data queryset.
    We'll be able to make further use of this later on when proper support
    for filtering arrives.

    And last, but not least, querysets are no longer forced into a
    DISTINCT mode. This is still on by default, but datagrids can turn it
    off. When on, the database resolves duplicates, but at a higher cost. If
    a queryset is written well, it can avoid duplicates without the need for
    this. Future versions of Djblets may switch this off by default.

    Unit tests pass in Djblets and Review Board.

    Made use of the new support to improve queryset counts in Review Board.

    Commits

    Files