Add smarter datagrid pagination, query augmentation, and distinct handling.
Review Request #13367 — Created Oct. 22, 2023 and submitted
Datagrids have historically used the same queryset for both page data
and for pagination. This led to expensive pagination detection, as the
nature of the paginator and Django's ORM results in the total page count
be computed as aSELECT COUNT(*) FROM (<subquery>)
, which is very
slow and gets worse the more data is matched.To resolve this, we're now using dual querysets. One queryset is
responsible purely for the filtering. The other builds on this and is
responsible for the page data.We use a new
DataGridPaginator
class to handle this, since Django's
paginator doesn't give us any way to set the total item count. This
class gives us that ability. It's now deprecated for subclasses to
return a paginator instance that is not a subclass of
DataGridPaginator
, but for the moment, it's still supported.The old way of augmenting querysets through the datagrid or columns is
also deprecated, as it was never built for this dual queryset method in
mind. Instead, we have two new augmentation pathways: One for filtering,
one for data collection. The filtering path is used for both querysets,
and the data collection path is only used for the page data queryset.
We'll be able to make further use of this later on when proper support
for filtering arrives.And last, but not least, querysets are no longer forced into a
DISTINCT
mode. This is still on by default, but datagrids can turn it
off. When on, the database resolves duplicates, but at a higher cost. If
a queryset is written well, it can avoid duplicates without the need for
this. Future versions of Djblets may switch this off by default.
Unit tests pass in Djblets and Review Board.
Made use of the new support to improve queryset counts in Review Board.
Summary | ID |
---|---|
c61e887873985749850cff3758aa2875ab282e4c |
Description | From | Last Updated |
---|---|---|
Could add an # Instance Variables # between these. |
maubin | |
Missing docs for state, request and **kwargs. |
maubin | |
Missing docs for state, request and **kwargs. |
maubin | |
Could add a "Version Changed" for the use_distinct arg. |
maubin | |
Missing docs for **kwargs. |
maubin | |
Missing docs for **kwargs. |
maubin |
- Change Summary:
-
- Added a missing
Instance Variables
block. - Added missing argument docs for some functions.
- Added a missing
Version Changed
toDataGrid
.
- Added a missing
- Commits:
-
Summary ID 759e5cec371ab515e34d97e226057b74a396de4b c61e887873985749850cff3758aa2875ab282e4c - Diff:
-
Revision 2 (+836 -44)