Attempt to handle threading-related issues with extension reloading.

Review Request #5866 — Created May 22, 2014 and submitted — Latest diff uploaded

Information

Djblets
release-0.8.x
8891572...

Reviewers

We observed some issues in a mod_worker environment where attempts to
reload extensions would fail, causing state to become corrupted, leading
to more and more errors. Some of these cases were made more
bullet-proof, but we didn't fix the core issue.

This change attempts to prevent what may be causing the issue by
allowing only one thread to reload the list of extensions at a time.

We have a lock set up for reloading extensions (in case two users try to
reload in the admin UI at the same time).

There's a separate lock in the middleware that's a bit smarter about
calling load(). It checks if the extension state has expired, tries to
grab a lock, and then checks again for expiration. The goal there is to
not have a bunch of threads that are waiting decide they're all going to
trigger their own reloads.

Unit tests have been written that hit issues both during uninit and
during init of extensions. These mirror the errors we've been getting
on reviews.reviewboard.org with Review Bot. With the fix, these tests
pass.

I don't have a real-world repro case yet (we'll see in a couple days on
reviews.reviewboard.org if we start to hit any problems), but I was able
to reproduce some of the errors we through unit tests.

Basically, there's a number of possible failure points, depending on what
instructions are being executed, but the tests manage to reliably trigger
one error during init and one during uninit. Those trigger only if I turn off
locking. Everything passes if locking is on.

    Loading...