Disable search indexing on interdiffs and some datagrids.

Review Request #14363 — Created March 5, 2025 and submitted

Information

Review Board
release-7.1.x

Reviewers

There are a number of pages in Review Board that aren't worth having
search engines crawl. While review requests and diffs can be useful,
interdiffs are not. While the All Review Requests page, Users page, and
Groups page are useful, the lists of review requests from a user or
group are not, since they're in All Review Requests.

The Djblets datagrid code has been updated to enable greater control
over search indexing of datagrids, enabling us to disable some search
indexing.

We now include rel="nofollow noindex" in links to interdiffs for both
diffs and file attachments, and we include a <meta> tag for disabling
indexing on the diff viewer when showing interdiffs and on the User's
and Group's review requests datagrid pages.

This of course is all optional for search engines, and the most
difficult ones (particularly AI training bots) won't respect most or any
of this, but it's at least helpful for the bots who are well-behaved.

Tested the affected pages and links and verified that they contained
the correct nofollow and noindex hints.

Summary ID
Disable search indexing on interdiffs and some datagrids.
There are a number of pages in Review Board that aren't worth having search engines crawl. While review requests and diffs can be useful, interdiffs are not. While the All Review Requests page, Users page, and Groups page are useful, the lists of review requests from a user or group are not, since they're in All Review Requests. The Djblets datagrid code has been updated to enable greater control over search indexing of datagrids, enabling us to disable some search indexing. We now include `rel="nofollow noindex"` in links to interdiffs for both diffs and file attachments, and we include a `<meta>` tag for disabling indexing on the diff viewer when showing interdiffs and on the User's and Group's review requests datagrid pages. This of course is all optional for search engines, and the most difficult ones (particularly AI training bots) won't respect most or any of this, but it's at least helpful for the bots who are well-behaved.
41f42b56bcfca28f894a895e9e6e074c27d6ae50
david
  1. Ship It!
  2. 
      
maubin
  1. Ship It!
  2. 
      
chipx86
Review request changed
Status:
Completed
Change Summary:
Pushed to release-7.1.x (e0861ee)
Loading...