Rework our Haystack search support to use a forwarding backend approach.

Review Request #11348 — Created Dec. 23, 2020 and submitted

Review Board

Haystack's never been good at dealing with changes to the configured
search backend, but more recent versions have become more difficult,
with more and more objects being initialized and stored on import. This
means that our prior attempts to clear caches when changing backends
just wasn't practical anymore.

This change implements the same approach we use for cache backends. We
now have a single main backend we assign to Haystack, which then
forwards on requests to the configured backend. This gives us a lot more
control and is more future-proof.

The one thing that can't be addressed here is a change they made to
SearchView. They construct a SearchQuerySet at class definition
time, which then pulls the BaseSearchQuery from the backend and stores
it. The only workaround for this is pretty straight-forward: we
construct a new SearchQuerySet at HTTP dispatch time.

With this comes a few additional fixes, primarily in our unit tests. We
sometimes hit random breakages in search tests, due to Whoosh being used
in asynchronous mode when writing to the disk. This sometimes resulted
in newly-indexed data failing to be read ("fixed" by sleeping for .1
second) and sometimes resulted in a prior test clobbering a successive
one (fixed by giving each run its own index directory within a temp

There's also a couple packaging updates. We now depend on the latest
Haystack series prior to 3.0 (which is Python 3-only), and we depend on
elasticsearch (which is a pretty small module with a very specific
version range we require).

All unit tests pass (except some unrelated breakages I'm sorting outs

Ran the suite several times and didn't hit any more random breakages,
though I'm not ruling out the possibility.

Tested searching on both Whoosh and Elasticsearch. Switched back-and-forth
between an Elasticsearch configuration with a populated index and a Whoosh
configuration with an empty index. I was able to dynamically switch and use
the proper engine without restarting the web server. Confirmed the right
backend objects were in use via print statements.

Rework our Haystack search support to use a forwarding backend approach.
Description From Last Updated

E501 line too long (80 > 79 characters)


E303 too many blank lines (2)

Checks run (1 failed, 1 succeeded)
flake8 failed.
JSHint passed.


  1. Ship It!
Review request changed

Status: Closed (submitted)

Change Summary:

Pushed to release-4.0.x (3cd0c54)