Avoid crashing when GenerationSynchronizer failed to talk to cache.

Review Request #13319 — Created Oct. 9, 2023 and submitted

Information

Djblets
release-4.x

Reviewers

GenerationSynchronizer is cache-dependent, and when the cache was
down, it would noisily crash. As this was often used in middleware, this
could prevent accessing certain URLs, such as the administration UI and
the new HealthCheckView.

We now protect against this by wrapping all cache-related calls in
exception handlers, logging the errors with some useful contextual
information and informing the admins that the cache server may be down.
This is a strict improvement over the seemingly-random crash behavior
that would manifest before.

Exception handlers are mainly placed in the outer functions where we
know what higher-level operation the synchronizer is trying to perform,
in order to help log better errors.

Tested normal operations, and tested with the cache server down and
in true outage mode. Verified I could still access the health check
endpoint and see useful log entries about the Site Configuration,
extensions, and integrations keys failing to synchronize.

Summary ID
Avoid crashing when GenerationSynchronizer failed to talk to cache.
`GenerationSynchronizer` is cache-dependent, and when the cache was down, it would noisily crash. As this was often used in middleware, this could prevent accessing certain URLs, such as the administration UI and the new `HealthCheckView`. We now protect against this by wrapping all cache-related calls in exception handlers, logging the errors with some useful contextual information and informing the admins that the cache server may be down. This is a strict improvement over the seemingly-random crash behavior that would manifest before. Exception handlers are mainly placed in the outer functions where we know what higher-level operation the synchronizer is trying to perform, in order to help log better errors.
3dbf6e86c0395bb447071403cb9b2e491ec147aa
david
  1. Ship It!
  2. 
      
maubin
  1. Ship It!
  2. 
      
chipx86
Review request changed
Status:
Completed
Change Summary:
Pushed to release-4.x (73b7b62)