Add support for Elasticsearch 5.x and 7.x.

Review Request #11890 — Created Dec. 14, 2021 and submitted

Information

Review Board
release-4.0.x

Reviewers

Review Board has historically only supported Elasticsearch 1.x and 2.x.
This was due to limitations in Django Haystack, which didn't gain
anything more modern until the more recent 3.x releases. Unfortunately,
we can't depend on these releases, due to lack of Python 2.7 support and
a hard (install-time) requirement on Django 2.2.

In order to support these version, we've backported Haystack's
Elasticsearch backends into the Review Board tree. These are largely
compatible with Haystack 2.8, with the exception of some constants that
were missing (and easily redefined). They were also largely compatible
with Python 2.7, with the exception of some syntax errors (trailing
commas after a **kwargs in a function call).

Our Elasticsearch option will select the right backend based on the
installed version of the elasticsearch module. We display the
installed version on the configuration page, and provide instructions on
how to change the compatible version of Elasticsearch.

(There's also a small fix to the margins for fieldset descriptions, which
was very noticeable on this settings page.)

There's one key compatibility issue introduced by Elasticsearch 7
support, having to do with indexing. We were indexing lists of LocalSite
IDs as numeric, instead of strings. The field type would normally cast
to string, but that was overridden by us. This was fine until
Elasticsearch 7, which enforced stricter types, and wouldn't let us do a
query that factored in LocalSites the way we wanted. The solution is to
cast to a string.

This doesn't appear to regress anything when updating an index that
previously used longs (mixing the two types), and versions of
Elasticsearch prior to 7 don't appear to care. Still, we'll want to
recommend a full reindex on upgrade.

Unit tests pass on Python 2.7 and 3.x.

Manually tested this with the latest Elasticsearch 1.x, 2.x, 5.x, 7.x.
Each test involved:

  1. Installing the appropriate build of elasticsearch (on Python 2.7
    and 3.8).
  2. Reloading the Search Settings page and verifying the version.
  3. Updating the URL to point to an appropriate Elasticsearch server.
  4. Rebuilding indexes with numeric Local Site IDs
  5. Performing search queries
  6. Rebuilding indexes with string Local Site IDs
  7. Performing search queries

The numeric queries failed on Elasticsearch 7.x, as expected, but 7.x
users will be starting with a new index anyway.

Also tested incremental indexing on 1.x and 2.x, mixing numeric and
string Local Site IDs.

Summary ID
Add support for Elasticsearch 5.x and 7.x.
Review Board has historically only supported Elasticsearch 1.x and 2.x. This was due to limitations in Django Haystack, which didn't gain anything more modern until the more recent 3.x releases. Unfortunately, we can't depend on these releases, due to lack of Python 2.7 support and a hard (install-time) requirement on Django 2.2. In order to support these version, we've backported Haystack's Elasticsearch backends into the Review Board tree. These are largely compatible with Haystack 2.8, with the exception of some constants that were missing (and easily redefined). They were also largely compatible with Python 2.7, with the exception of some syntax errors (trailing commas after a `**kwargs` in a function call). Our Elasticsearch option will select the right backend based on the installed version of the `elasticsearch` module. We display the installed version on the configuration page, and provide instructions on how to change the compatible version of Elasticsearch. There's one key compatibility issue introduced by Elasticsearch 7 support, having to do with indexing. We were indexing lists of LocalSite IDs as numeric, instead of strings. The field type would normally cast to string, but that was overridden by us. This was fine until Elasticsearch 7, which enforced stricter types, and wouldn't let us do a query that factored in LocalSites the way we wanted. The solution is to cast to a string. This doesn't appear to regress anything when updating an index that previously used longs (mixing the two types), and versions of Elasticsearch prior to 7 don't appear to care. Still, we'll want to recommend a full reindex on upgrade.
23df9ae04827eacd057836a84cf4299428c1920f

Description From Last Updated

Alphabetize

daviddavid

E501 line too long (87 > 79 characters)

reviewbotreviewbot

E501 line too long (87 > 79 characters)

reviewbotreviewbot

E501 line too long (82 > 79 characters)

reviewbotreviewbot

E501 line too long (93 > 79 characters)

reviewbotreviewbot

E501 line too long (80 > 79 characters)

reviewbotreviewbot

E501 line too long (85 > 79 characters)

reviewbotreviewbot

E501 line too long (83 > 79 characters)

reviewbotreviewbot

E501 line too long (81 > 79 characters)

reviewbotreviewbot

E501 line too long (87 > 79 characters)

reviewbotreviewbot

E501 line too long (87 > 79 characters)

reviewbotreviewbot

E501 line too long (84 > 79 characters)

reviewbotreviewbot

E501 line too long (86 > 79 characters)

reviewbotreviewbot

E501 line too long (81 > 79 characters)

reviewbotreviewbot

E501 line too long (80 > 79 characters)

reviewbotreviewbot

E501 line too long (90 > 79 characters)

reviewbotreviewbot

E501 line too long (83 > 79 characters)

reviewbotreviewbot

E501 line too long (85 > 79 characters)

reviewbotreviewbot

E501 line too long (80 > 79 characters)

reviewbotreviewbot

E501 line too long (87 > 79 characters)

reviewbotreviewbot

E501 line too long (87 > 79 characters)

reviewbotreviewbot

E501 line too long (82 > 79 characters)

reviewbotreviewbot

E501 line too long (102 > 79 characters)

reviewbotreviewbot

E501 line too long (89 > 79 characters)

reviewbotreviewbot

E501 line too long (88 > 79 characters)

reviewbotreviewbot

E501 line too long (95 > 79 characters)

reviewbotreviewbot

E501 line too long (83 > 79 characters)

reviewbotreviewbot

E501 line too long (93 > 79 characters)

reviewbotreviewbot

E501 line too long (80 > 79 characters)

reviewbotreviewbot

E501 line too long (85 > 79 characters)

reviewbotreviewbot

E501 line too long (83 > 79 characters)

reviewbotreviewbot
Checks run (1 failed, 1 succeeded)
flake8 failed.
JSHint passed.

flake8

david
  1. 
      
  2. reviewboard/search/indexes.py (Diff revision 1)
     
     
     
     
    Show all issues

    Alphabetize

  3. 
      
chipx86
Review request changed
Status:
Completed
Change Summary:
Pushed to release-4.0.x (ab2f6db)