Rework our Haystack search support to use a forwarding backend approach.

Review Request #11348 — Created Dec. 23, 2020 and submitted

Information

Review Board
release-4.0.x

Reviewers

Haystack's never been good at dealing with changes to the configured
search backend, but more recent versions have become more difficult,
with more and more objects being initialized and stored on import. This
means that our prior attempts to clear caches when changing backends
just wasn't practical anymore.

This change implements the same approach we use for cache backends. We
now have a single main backend we assign to Haystack, which then
forwards on requests to the configured backend. This gives us a lot more
control and is more future-proof.

The one thing that can't be addressed here is a change they made to
SearchView. They construct a SearchQuerySet at class definition
time, which then pulls the BaseSearchQuery from the backend and stores
it. The only workaround for this is pretty straight-forward: we
construct a new SearchQuerySet at HTTP dispatch time.

With this comes a few additional fixes, primarily in our unit tests. We
sometimes hit random breakages in search tests, due to Whoosh being used
in asynchronous mode when writing to the disk. This sometimes resulted
in newly-indexed data failing to be read ("fixed" by sleeping for .1
second) and sometimes resulted in a prior test clobbering a successive
one (fixed by giving each run its own index directory within a temp
directory).

There's also a couple packaging updates. We now depend on the latest
Haystack series prior to 3.0 (which is Python 3-only), and we depend on
elasticsearch (which is a pretty small module with a very specific
version range we require).

All unit tests pass (except some unrelated breakages I'm sorting outs
separately).

Ran the suite several times and didn't hit any more random breakages,
though I'm not ruling out the possibility.

Tested searching on both Whoosh and Elasticsearch. Switched back-and-forth
between an Elasticsearch configuration with a populated index and a Whoosh
configuration with an empty index. I was able to dynamically switch and use
the proper engine without restarting the web server. Confirmed the right
backend objects were in use via print statements.

Summary ID
Rework our Haystack search support to use a forwarding backend approach.
Haystack's never been good at dealing with changes to the configured search backend, but more recent versions have become more difficult, with more and more objects being initialized and stored on import. This means that our prior attempts to clear caches when changing backends just wasn't practical anymore. This change implements the same approach we use for cache backends. We now have a single main backend we assign to Haystack, which then forwards on requests to the configured backend. This gives us a lot more control and is more future-proof. The one thing that can't be addressed here is a change they made to `SearchView`. They construct a `SearchQuerySet` at class definition time, which then pulls the `BaseSearchQuery` from the backend and stores it. The only workaround for this is pretty straight-forward: we construct a new `SearchQuerySet` at HTTP dispatch time. With this comes a few additional fixes, primarily in our unit tests. We sometimes hit random breakages in search tests, due to Whoosh being used in asynchronous mode when writing to the disk. This sometimes resulted in newly-indexed data failing to be read ("fixed" by sleeping for .1 second) and sometimes resulted in a prior test clobbering a successive one (fixed by giving each run its own index directory within a temp directory). There's also a couple packaging updates. We now depend on the latest Haystack series prior to 3.0 (which is Python 3-only), and we depend on `elasticsearch` (which is a pretty small module with a very specific version range we require).
8e394f1b26e08b96ecbee855666c24df8dd37f78
Description From Last Updated

E501 line too long (80 > 79 characters)

reviewbotreviewbot

E303 too many blank lines (2)

reviewbotreviewbot
Checks run (1 failed, 1 succeeded)
flake8 failed.
JSHint passed.

flake8

chipx86
david
  1. Ship It!
  2. 
      
chipx86
Review request changed

Status: Closed (submitted)

Change Summary:

Pushed to release-4.0.x (3cd0c54)
Loading...