Fix thread safety issues with the forwarding cache backend.

Review Request #6644 — Created Nov. 25, 2014 and submitted

Information

Djblets
release-0.8.x
4e037d0...

Reviewers

The forwarding cache backend had a thread safety problem where a thread
would be resetting the cache while another was attempting to access it,
resulting in the access being performed on a None value. This is a
problem because in Django, a cache backend is shared across all threads.

A couple things were done to improve the thread safety.

First, we no longer reset the backend to None. Instead, we replace the
old backend with a new one. Accesses to the old backend may still occur
without completely breaking.

Second, we reduce the chance of such collisions by preventing several
threads from reloading the backend at the same time. We now keep a
generation number for the cache backend. When we go to load a new
backend, the thread will locally store the last seen generation number
and then attempt to grab a lock.

Whichever thread acquires that lock first will set the new backend and
then update the generation number. Subsequent threads that grab the lock
will check their local copy of the old generation number against the
stored one, and use that to decide whether to load the backend.

I've been dealing with random HTTP 500s in Chrome and sometimes in Firefox
for a while now. Chrome showed no payload, and I thought this was another
case of the browser trying to fetch more requests at once than the server
could handle (an old Chrome bug that was fixed at one point).

Digging into this, I found there were cache backend accesses against None,
and I found that it was due to threads stomping over each other.

This fix solved it. I've reloaded in Chrome and Firefox dozens of times
without a single HTTP 500 error.

Unit tests also pass.

reviewbot
  1. Tool: Pyflakes
    Processed Files:
        djblets/cache/forwarding_backend.py
    
    
    
    Tool: PEP8 Style Checker
    Processed Files:
        djblets/cache/forwarding_backend.py
    
    
  2. 
      
david
  1. Ship It!
  2. 
      
chipx86
Review request changed

Status: Closed (submitted)

Change Summary:

Pushed to release-0.8.x (9ec8fd2)
Loading...