Add distributed lock functionality, and locked cache updates.
Review Request #14628 — Created Oct. 8, 2025 and updated
This introduces
djblets.protect, a new module for service protection
capabilities, and specificallydjblets.protect.locks.CacheLock, which
is a simple distributed lock utilizing the cache backend. This can help
avoid cache stampede issues, and overall reduce the work required by a
service.It's important to note that these locks should only be used in cases
where the loss of a lock will not cause corruption or other bad
behavior. As cache backends may expire keys prematurely, and may lack
atomic operations, a lock cannot be guaranteed. These can be thought of
as a soft optimistic lock.Locks have an expiration, and consumers can block waiting on a lock to
be available or return immediately, giving control over how to best
utilize a lock.Locks are set by performing an atomic
add()with a UUID4. If the value
is added, the lock is acquired. If it already exists, the lock has to
either block waiting or return a result. Waiting supports a timeout and
a time between retries.When waiting, the lock will periodically check if it can acquire a new
lock, using the provided timestamp and some random jitter to help avoid
issues with stampedes where too many consumers are trying at the same
times to check and acquire a lock.Locks are released when they expire or (ideally) when
release()is
called. It's also possible they may fall out of cache, at which point
the lock is no longer valid, and suitable logging will occur.Since there aren't atomic operations around deletes, this will try to do
release a lock as safely as possible. If the time spent with the lock is
greater than the expected expiration, it will assume the lock has
expired in cache and won't delete it (it may have been re-acquired
elsewhere). Otherwise, it will attempt to bump the expiration to
keep the key alive long enough to check it, with a worst-case scenario
that the other acquirer may have a new expiration set (likely extending
the lock). This is preferable over deleting another lock.When using a lock as a context manager, both acquiring and releasing the
lock is handled automatically.The interface is designed to be largely API-compatible with
threading.Lockand similar lock interfaces, but with more flexibility
useful for distributed lock behavior.A pattern I expect to be common will be to lock a cache key when
calculating state to store and then writing it, which may be expensive
(for instance, talking to a remote service and storing the result).For this,
cache_memoize()andcache_memoize_iter()have been updated
to work with locks. They now take alock=argument, which accepts a
CacheLockwith the parameters controlling the lock behavior. If
provided, the lock will be acquired if the initial fetch doesn't yield a
value. A second fetch is then attempted (in case it had to wait for
another process to finish), and if it still needs to compute data to
cache, it will do so under the protection of the lock, releasing when
complete.Locks are entirely optional and not enabled by default for any current
caching behavior, but are something we'll likely want to opt into any
time we're working on caching something that's expensive to generate.
Unit tests pass.
| Summary | ID |
|---|---|
| 97a37605a541154ffe6a6e1abcf80fd48a1258a8 |
| Description | From | Last Updated |
|---|---|---|
|
Can we add debug logging for when locks are acquired, extended, and released? This seems like potentially a cause of … |
|
|
|
Remove this blank line? |
|
|
|
typo: as -> was |
|
|
|
Do we want to add any validation to these (ex: no negative numbers, timeout should be longer than retry, etc)? |
|
|
|
These attributes are actually named timeout_secs and retry_secs |
|
|
|
Can we add the lock key in this exception message? AssertionError also seems like maybe not the best type. Perhaps … |
|
|
|
Comparing a float like this isn't reliable. While we could use something like math.isclose(), I think a much better option … |
|
|
|
Seems like we likely want >= instead of > |
|
|
|
The implementation here doesn't return anything |
|
|
|
typo: released -> release |
|
|
|
It seems like there's a potential race here where in between the check and the delete, the existing lock could … |
|
|
|
Should we catch RuntimeError here as well, in case this is called while the lock is already acquired? |
|
|
|
Can we add a __del__ implementation that logs a warning if the object gets garbage collected while the lock is … |
|
- Change Summary:
-
- Moved to a new
djblets.protect, which will be the place for other service protection code, like rate limiting. - Added to the README and codebase docs.
- Moved to a new
- Description:
-
~ This introduces
djblets.cache.locks.CacheLock, which is a simple~ distributed lock utilizing the cache backend. This can help avoid cache ~ stampede issues, and overall reduce the work required by a service. ~ This introduces
djblets.protect, a new module for service protection~ capabilities, and specifically djblets.protect.locks.CacheLock, which~ is a simple distributed lock utilizing the cache backend. This can help + avoid cache stampede issues, and overall reduce the work required by a + service. Locks have an expiration, and consumers can block waiting on a lock to
be available or return immediately, giving control over how to best utilize a lock. Locks are set by performing an atomic
add()with a UUID4. If the valueis added, the lock is acquired. If it already exists, the lock has to either block waiting or return a result. Waiting supports a timeout and a time between retries. Locks are released when they expire or (ideally) when
release()iscalled. When using a lock as a context manager, both acquiring and releasing the
lock is handled automatically. A pattern I expect to be common will be to lock a cache key when
calculating state to store and then writing it, which may be expensive (for instance, talking to a remote service and storing the result). For this,
cache_memoize()andcache_memoize_iter()have been updatedto work with locks. They now take a lock=argument, which accepts aCacheLockwith the parameters controlling the lock behavior. Ifprovided, the lock will be acquired if the initial fetch doesn't yield a value. A second fetch is then attempted (in case it had to wait for another process to finish), and if it still needs to compute data to cache, it will do so under the protection of the lock, releasing when complete. Locks are entirely optional and not enabled by default for any current
caching behavior, but are something we'll likely want to opt into any time we're working on caching something that's expensive to generate. - Commits:
-
Summary ID a0bda35677806af82016bd478fffdf63c61b6267 5ad002a1428c8244e6a2e92d94a3e93f8d952c50
Checks run (2 succeeded)
- Change Summary:
-
Reworked some of the API to be compatible with
threading.Lockand similar. - Description:
-
This introduces
djblets.protect, a new module for service protectioncapabilities, and specifically djblets.protect.locks.CacheLock, whichis a simple distributed lock utilizing the cache backend. This can help avoid cache stampede issues, and overall reduce the work required by a service. Locks have an expiration, and consumers can block waiting on a lock to
be available or return immediately, giving control over how to best utilize a lock. Locks are set by performing an atomic
add()with a UUID4. If the valueis added, the lock is acquired. If it already exists, the lock has to either block waiting or return a result. Waiting supports a timeout and a time between retries. Locks are released when they expire or (ideally) when
release()iscalled. When using a lock as a context manager, both acquiring and releasing the
lock is handled automatically. + The interface is designed to be largely API-compatible with
+ threading.Lockand similar lock interfaces, but with more flexibility+ useful for distributed lock behavior. + A pattern I expect to be common will be to lock a cache key when
calculating state to store and then writing it, which may be expensive (for instance, talking to a remote service and storing the result). For this,
cache_memoize()andcache_memoize_iter()have been updatedto work with locks. They now take a lock=argument, which accepts aCacheLockwith the parameters controlling the lock behavior. Ifprovided, the lock will be acquired if the initial fetch doesn't yield a value. A second fetch is then attempted (in case it had to wait for another process to finish), and if it still needs to compute data to cache, it will do so under the protection of the lock, releasing when complete. Locks are entirely optional and not enabled by default for any current
caching behavior, but are something we'll likely want to opt into any time we're working on caching something that's expensive to generate. - Commits:
-
Summary ID 5ad002a1428c8244e6a2e92d94a3e93f8d952c50 c181dc802fc56790d92b3494a6790ec809bdfce8
Checks run (2 succeeded)
-
-
Can we add debug logging for when locks are acquired, extended, and released? This seems like potentially a cause of difficult bugs.
-
-
-
Do we want to add any validation to these (ex: no negative numbers, timeout should be longer than retry, etc)?
-
-
Can we add the lock key in this exception message?
AssertionErroralso seems like maybe not the best type. Perhaps just aRuntimeError? -
Comparing a float like this isn't reliable. While we could use something like
math.isclose(), I think a much better option than a-1magic value would be to define a sentinel value to use to indicate no timeout. -
-
-
-
It seems like there's a potential race here where in between the check and the delete, the existing lock could timeout and be acquired by another user. I don't suppose there are any atomic operations we could do for this?
- Change Summary:
-
- Added a big note to the docs that these locks can be lossy, listing the limitations and use cases.
- Added safer handling of key loss (from cache purge or race conditions). Now when releasing a lock, we first check if we think we're over the expiration. If so, we don't touch the key. Otherwise we touch the key to extend expiration, and if that succeeds, we check the stored token and delete.
- Added random jitter when retrying lock acquisition, which avoids processes continuously trying to acquire locks at the same time.
- A new token is generated each time
acquire()is called, allowing lock reuse. - Added debug and warning logging.
- Fixed various issues in docs.
- Added some checks and constraints for timeout and retry values.
- Switched to
RuntimeErrorfor exceptions.
- Description:
-
This introduces
djblets.protect, a new module for service protectioncapabilities, and specifically djblets.protect.locks.CacheLock, whichis a simple distributed lock utilizing the cache backend. This can help avoid cache stampede issues, and overall reduce the work required by a service. + It's important to note that these locks should only be used in cases
+ where the loss of a lock will not cause corruption or other bad + behavior. As cache backends may expire keys prematurely, and may lack + atomic operations, a lock cannot be guaranteed. These can be thought of + as a soft optimistic lock. + Locks have an expiration, and consumers can block waiting on a lock to
be available or return immediately, giving control over how to best utilize a lock. Locks are set by performing an atomic
add()with a UUID4. If the valueis added, the lock is acquired. If it already exists, the lock has to either block waiting or return a result. Waiting supports a timeout and a time between retries. + When waiting, the lock will periodically check if it can acquire a new
+ lock, using the provided timestamp and some random jitter to help avoid + issues with stampedes where too many consumers are trying at the same + times to check and acquire a lock. + Locks are released when they expire or (ideally) when
release()is~ called. ~ called. It's also possible they may fall out of cache, at which point + the lock is no longer valid, and suitable logging will occur. + + Since there aren't atomic operations around deletes, this will try to do
+ release a lock as safely as possible. If the time spent with the lock is + greater than the expected expiration, it will assume the lock has + expired in cache and won't delete it (it may have been re-acquired + elsewhere). Otherwise, it will attempt to bump the expiration to + keep the key alive long enough to check it, with a worst-case scenario + that the other acquirer may have a new expiration set (likely extending + the lock). This is preferable over deleting another lock. When using a lock as a context manager, both acquiring and releasing the
lock is handled automatically. The interface is designed to be largely API-compatible with
threading.Lockand similar lock interfaces, but with more flexibilityuseful for distributed lock behavior. A pattern I expect to be common will be to lock a cache key when
calculating state to store and then writing it, which may be expensive (for instance, talking to a remote service and storing the result). For this,
cache_memoize()andcache_memoize_iter()have been updatedto work with locks. They now take a lock=argument, which accepts aCacheLockwith the parameters controlling the lock behavior. Ifprovided, the lock will be acquired if the initial fetch doesn't yield a value. A second fetch is then attempted (in case it had to wait for another process to finish), and if it still needs to compute data to cache, it will do so under the protection of the lock, releasing when complete. Locks are entirely optional and not enabled by default for any current
caching behavior, but are something we'll likely want to opt into any time we're working on caching something that's expensive to generate. - Commits:
-
Summary ID c181dc802fc56790d92b3494a6790ec809bdfce8 97a37605a541154ffe6a6e1abcf80fd48a1258a8