• 
      

    Add smarter datagrid pagination, query augmentation, and distinct handling.

    Review Request #13367 — Created Oct. 22, 2023 and submitted

    Information

    Djblets
    release-3.x

    Reviewers

    Datagrids have historically used the same queryset for both page data
    and for pagination. This led to expensive pagination detection, as the
    nature of the paginator and Django's ORM results in the total page count
    be computed as a SELECT COUNT(*) FROM (<subquery>), which is very
    slow and gets worse the more data is matched.

    To resolve this, we're now using dual querysets. One queryset is
    responsible purely for the filtering. The other builds on this and is
    responsible for the page data.

    We use a new DataGridPaginator class to handle this, since Django's
    paginator doesn't give us any way to set the total item count. This
    class gives us that ability. It's now deprecated for subclasses to
    return a paginator instance that is not a subclass of
    DataGridPaginator, but for the moment, it's still supported.

    The old way of augmenting querysets through the datagrid or columns is
    also deprecated, as it was never built for this dual queryset method in
    mind. Instead, we have two new augmentation pathways: One for filtering,
    one for data collection. The filtering path is used for both querysets,
    and the data collection path is only used for the page data queryset.
    We'll be able to make further use of this later on when proper support
    for filtering arrives.

    And last, but not least, querysets are no longer forced into a
    DISTINCT mode. This is still on by default, but datagrids can turn it
    off. When on, the database resolves duplicates, but at a higher cost. If
    a queryset is written well, it can avoid duplicates without the need for
    this. Future versions of Djblets may switch this off by default.

    Unit tests pass in Djblets and Review Board.

    Made use of the new support to improve queryset counts in Review Board.

    Summary ID
    Add smarter datagrid pagination, query augmentation, and distinct handling.
    Datagrids have historically used the same queryset for both page data and for pagination. This led to expensive pagination detection, as the nature of the paginator and Django's ORM results in the total page count be computed as a `SELECT COUNT(*) FROM (<subquery>)`, which is very slow and gets worse the more data is matched. To resolve this, we're now using dual querysets. One queryset is responsible purely for the filtering. The other builds on this and is responsible for the page data. We use a new `DataGridPaginator` class to handle this, since Django's paginator doesn't give us any way to set the total item count. This class gives us that ability. It's now deprecated for subclasses to return a paginator instance that is not a subclass of `DataGridPaginator`, but for the moment, it's still supported. The old way of augmenting querysets through the datagrid or columns is also deprecated, as it was never built for this dual queryset method in mind. Instead, we have two new augmentation pathways: One for filtering, one for data collection. The filtering path is used for both querysets, and the data collection path is only used for the page data queryset. We'll be able to make further use of this later on when proper support for filtering arrives. And last, but not least, querysets are no longer forced into a `DISTINCT` mode. This is still on by default, but datagrids can turn it off. When on, the database resolves duplicates, but at a higher cost. If a queryset is written well, it can avoid duplicates without the need for this. Future versions of Djblets may switch this off by default.
    c61e887873985749850cff3758aa2875ab282e4c
    Description From Last Updated

    Could add an # Instance Variables # between these.

    maubinmaubin

    Missing docs for state, request and **kwargs.

    maubinmaubin

    Missing docs for state, request and **kwargs.

    maubinmaubin

    Could add a "Version Changed" for the use_distinct arg.

    maubinmaubin

    Missing docs for **kwargs.

    maubinmaubin

    Missing docs for **kwargs.

    maubinmaubin
    david
    1. Ship It!
    2. 
        
    maubin
    1. 
        
    2. djblets/datagrid/grids.py (Diff revision 1)
       
       
       
       
       
       
       
       
       
       
       
       
       
       
       
       
      Show all issues

      Could add an # Instance Variables # between these.

    3. djblets/datagrid/grids.py (Diff revision 1)
       
       
       
       
      Show all issues

      Missing docs for state, request and **kwargs.

    4. djblets/datagrid/grids.py (Diff revision 1)
       
       
       
       
      Show all issues

      Missing docs for state, request and **kwargs.

    5. djblets/datagrid/grids.py (Diff revision 1)
       
       
      Show all issues

      Could add a "Version Changed" for the use_distinct arg.

    6. djblets/datagrid/grids.py (Diff revision 1)
       
       
       
       
      Show all issues

      Missing docs for **kwargs.

    7. djblets/datagrid/grids.py (Diff revision 1)
       
       
       
       
      Show all issues

      Missing docs for **kwargs.

    8. 
        
    chipx86
    maubin
    1. Ship It!
    2. 
        
    chipx86
    Review request changed
    Status:
    Completed
    Change Summary:
    Pushed to release-3.x (8560908)