Avoid crashing when GenerationSynchronizer failed to talk to cache.
Review Request #13319 — Created Oct. 9, 2023 and submitted
GenerationSynchronizeris cache-dependent, and when the cache was
down, it would noisily crash. As this was often used in middleware, this
could prevent accessing certain URLs, such as the administration UI and
We now protect against this by wrapping all cache-related calls in
exception handlers, logging the errors with some useful contextual
information and informing the admins that the cache server may be down.
This is a strict improvement over the seemingly-random crash behavior
that would manifest before.
Exception handlers are mainly placed in the outer functions where we
know what higher-level operation the synchronizer is trying to perform,
in order to help log better errors.
Tested normal operations, and tested with the cache server down and
in true outage mode. Verified I could still access the health check
endpoint and see useful log entries about the Site Configuration,
extensions, and integrations keys failing to synchronize.