Add distributed lock functionality, and locked cache updates.

Review Request #14628 — Created Oct. 8, 2025 and updated

Information

Djblets
release-5.x

Reviewers

This introduces djblets.protect, a new module for service protection
capabilities, and specifically djblets.protect.locks.CacheLock, which
is a simple distributed lock utilizing the cache backend. This can help
avoid cache stampede issues, and overall reduce the work required by a
service.

Locks have an expiration, and consumers can block waiting on a lock to
be available or return immediately, giving control over how to best
utilize a lock.

Locks are set by performing an atomic add() with a UUID4. If the value
is added, the lock is acquired. If it already exists, the lock has to
either block waiting or return a result. Waiting supports a timeout and
a time between retries.

Locks are released when they expire or (ideally) when release() is
called.

When using a lock as a context manager, both acquiring and releasing the
lock is handled automatically.

The interface is designed to be largely API-compatible with
threading.Lock and similar lock interfaces, but with more flexibility
useful for distributed lock behavior.

A pattern I expect to be common will be to lock a cache key when
calculating state to store and then writing it, which may be expensive
(for instance, talking to a remote service and storing the result).

For this, cache_memoize() and cache_memoize_iter() have been updated
to work with locks. They now take a lock= argument, which accepts a
CacheLock with the parameters controlling the lock behavior. If
provided, the lock will be acquired if the initial fetch doesn't yield a
value. A second fetch is then attempted (in case it had to wait for
another process to finish), and if it still needs to compute data to
cache, it will do so under the protection of the lock, releasing when
complete.

Locks are entirely optional and not enabled by default for any current
caching behavior, but are something we'll likely want to opt into any
time we're working on caching something that's expensive to generate.

Unit tests pass.

Summary ID
Add distributed lock functionality, and locked cache updates.
This introduces `djblets.protect`, a new module for service protection capabilities, and specifically `djblets.protect.locks.CacheLock`, which is a simple distributed lock utilizing the cache backend. This can help avoid cache stampede issues, and overall reduce the work required by a service. Locks have an expiration, and consumers can block waiting on a lock to be available or return immediately, giving control over how to best utilize a lock. Locks are set by performing an atomic `add()` with a UUID4. If the value is added, the lock is acquired. If it already exists, the lock has to either block waiting or return a result. Waiting supports a timeout and a time between retries. Locks are released when they expire or (ideally) when `release()` is called. When using a lock as a context manager, both acquiring and releasing the lock is handled automatically. The interface is designed to be largely API-compatible with `threading.Lock` and similar lock interfaces, but with more flexibility useful for distributed lock behavior. A pattern I expect to be common will be to lock a cache key when calculating state to store and then writing it, which may be expensive (for instance, talking to a remote service and storing the result). For this, `cache_memoize()` and `cache_memoize_iter()` have been updated to work with locks. They now take a `lock=` argument, which accepts a `CacheLock` with the parameters controlling the lock behavior. If provided, the lock will be acquired if the initial fetch doesn't yield a value. A second fetch is then attempted (in case it had to wait for another process to finish), and if it still needs to compute data to cache, it will do so under the protection of the lock, releasing when complete. Locks are entirely optional and not enabled by default for any current caching behavior, but are something we'll likely want to opt into any time we're working on caching something that's expensive to generate.
c181dc802fc56790d92b3494a6790ec809bdfce8
Description From Last Updated

Can we add debug logging for when locks are acquired, extended, and released? This seems like potentially a cause of …

daviddavid

Remove this blank line?

daviddavid

typo: as -> was

daviddavid

Do we want to add any validation to these (ex: no negative numbers, timeout should be longer than retry, etc)?

daviddavid

These attributes are actually named timeout_secs and retry_secs

daviddavid

Can we add the lock key in this exception message? AssertionError also seems like maybe not the best type. Perhaps …

daviddavid

Comparing a float like this isn't reliable. While we could use something like math.isclose(), I think a much better option …

daviddavid

Seems like we likely want >= instead of >

daviddavid

The implementation here doesn't return anything

daviddavid

typo: released -> release

daviddavid

It seems like there's a potential race here where in between the check and the delete, the existing lock could …

daviddavid
chipx86
chipx86
Review request changed
Change Summary:

Reworked some of the API to be compatible with threading.Lock and similar.

Description:
   

This introduces djblets.protect, a new module for service protection

    capabilities, and specifically djblets.protect.locks.CacheLock, which
    is a simple distributed lock utilizing the cache backend. This can help
    avoid cache stampede issues, and overall reduce the work required by a
    service.

   
   

Locks have an expiration, and consumers can block waiting on a lock to

    be available or return immediately, giving control over how to best
    utilize a lock.

   
   

Locks are set by performing an atomic add() with a UUID4. If the value

    is added, the lock is acquired. If it already exists, the lock has to
    either block waiting or return a result. Waiting supports a timeout and
    a time between retries.

   
   

Locks are released when they expire or (ideally) when release() is

    called.

   
   

When using a lock as a context manager, both acquiring and releasing the

    lock is handled automatically.

   
  +

The interface is designed to be largely API-compatible with

  + threading.Lock and similar lock interfaces, but with more flexibility
  + useful for distributed lock behavior.

  +
   

A pattern I expect to be common will be to lock a cache key when

    calculating state to store and then writing it, which may be expensive
    (for instance, talking to a remote service and storing the result).

   
   

For this, cache_memoize() and cache_memoize_iter() have been updated

    to work with locks. They now take a lock= argument, which accepts a
    CacheLock with the parameters controlling the lock behavior. If
    provided, the lock will be acquired if the initial fetch doesn't yield a
    value. A second fetch is then attempted (in case it had to wait for
    another process to finish), and if it still needs to compute data to
    cache, it will do so under the protection of the lock, releasing when
    complete.

   
   

Locks are entirely optional and not enabled by default for any current

    caching behavior, but are something we'll likely want to opt into any
    time we're working on caching something that's expensive to generate.

Commits:
Summary ID
Add distributed lock functionality, and locked cache updates.
This introduces `djblets.protect`, a new module for service protection capabilities, and specifically `djblets.protect.locks.CacheLock`, which is a simple distributed lock utilizing the cache backend. This can help avoid cache stampede issues, and overall reduce the work required by a service. Locks have an expiration, and consumers can block waiting on a lock to be available or return immediately, giving control over how to best utilize a lock. Locks are set by performing an atomic `add()` with a UUID4. If the value is added, the lock is acquired. If it already exists, the lock has to either block waiting or return a result. Waiting supports a timeout and a time between retries. Locks are released when they expire or (ideally) when `release()` is called. When using a lock as a context manager, both acquiring and releasing the lock is handled automatically. A pattern I expect to be common will be to lock a cache key when calculating state to store and then writing it, which may be expensive (for instance, talking to a remote service and storing the result). For this, `cache_memoize()` and `cache_memoize_iter()` have been updated to work with locks. They now take a `lock=` argument, which accepts a `CacheLock` with the parameters controlling the lock behavior. If provided, the lock will be acquired if the initial fetch doesn't yield a value. A second fetch is then attempted (in case it had to wait for another process to finish), and if it still needs to compute data to cache, it will do so under the protection of the lock, releasing when complete. Locks are entirely optional and not enabled by default for any current caching behavior, but are something we'll likely want to opt into any time we're working on caching something that's expensive to generate.
5ad002a1428c8244e6a2e92d94a3e93f8d952c50
Add distributed lock functionality, and locked cache updates.
This introduces `djblets.protect`, a new module for service protection capabilities, and specifically `djblets.protect.locks.CacheLock`, which is a simple distributed lock utilizing the cache backend. This can help avoid cache stampede issues, and overall reduce the work required by a service. Locks have an expiration, and consumers can block waiting on a lock to be available or return immediately, giving control over how to best utilize a lock. Locks are set by performing an atomic `add()` with a UUID4. If the value is added, the lock is acquired. If it already exists, the lock has to either block waiting or return a result. Waiting supports a timeout and a time between retries. Locks are released when they expire or (ideally) when `release()` is called. When using a lock as a context manager, both acquiring and releasing the lock is handled automatically. The interface is designed to be largely API-compatible with `threading.Lock` and similar lock interfaces, but with more flexibility useful for distributed lock behavior. A pattern I expect to be common will be to lock a cache key when calculating state to store and then writing it, which may be expensive (for instance, talking to a remote service and storing the result). For this, `cache_memoize()` and `cache_memoize_iter()` have been updated to work with locks. They now take a `lock=` argument, which accepts a `CacheLock` with the parameters controlling the lock behavior. If provided, the lock will be acquired if the initial fetch doesn't yield a value. A second fetch is then attempted (in case it had to wait for another process to finish), and if it still needs to compute data to cache, it will do so under the protection of the lock, releasing when complete. Locks are entirely optional and not enabled by default for any current caching behavior, but are something we'll likely want to opt into any time we're working on caching something that's expensive to generate.
c181dc802fc56790d92b3494a6790ec809bdfce8

Checks run (2 succeeded)

flake8 passed.
JSHint passed.
david
  1. 
      
  2. Show all issues

    Can we add debug logging for when locks are acquired, extended, and released? This seems like potentially a cause of difficult bugs.

  3. djblets/cache/backend.py (Diff revision 3)
     
     
    Show all issues

    Remove this blank line?

  4. djblets/cache/backend.py (Diff revision 3)
     
     
    Show all issues

    typo: as -> was

  5. djblets/protect/locks.py (Diff revision 3)
     
     
     
    Show all issues

    Do we want to add any validation to these (ex: no negative numbers, timeout should be longer than retry, etc)?

  6. djblets/protect/locks.py (Diff revision 3)
     
     
     
    Show all issues

    These attributes are actually named timeout_secs and retry_secs

  7. djblets/protect/locks.py (Diff revision 3)
     
     
    Show all issues

    Can we add the lock key in this exception message?

    AssertionError also seems like maybe not the best type. Perhaps just a RuntimeError?

  8. djblets/protect/locks.py (Diff revision 3)
     
     
    Show all issues

    Comparing a float like this isn't reliable. While we could use something like math.isclose(), I think a much better option than a -1 magic value would be to define a sentinel value to use to indicate no timeout.

  9. djblets/protect/locks.py (Diff revision 3)
     
     
    Show all issues

    Seems like we likely want >= instead of >

  10. djblets/protect/locks.py (Diff revision 3)
     
     
     
     
     
    Show all issues

    The implementation here doesn't return anything

  11. djblets/protect/locks.py (Diff revision 3)
     
     
    Show all issues

    typo: released -> release

  12. djblets/protect/locks.py (Diff revision 3)
     
     
     
     
    Show all issues

    It seems like there's a potential race here where in between the check and the delete, the existing lock could timeout and be acquired by another user. I don't suppose there are any atomic operations we could do for this?

  13.