Add initial support for message broker backends.
Review Request #15196 — Created July 24, 2026 and updated
This introduces
reviewboard.broker, which provides official support in
Review Board for communicating with message brokers (such as RabbitMQ)
and sending tasks or broadcasts to any workers connected to the broker.This initial change offers base broker backend support, a local
filesystem-based broker, a broker registry, and the beginnings of broker
configuration.A broker backend is responsible for connecting to a broker service,
allowing tasks to be sent or messages to be broadcast to all active
workers, and gathering worker status.We're using Celery for the main broker work, since that's pretty
complete, but the architecture doesn't mandate this.BaseBrokerBackend
doesn't care about the transport, whileBaseCeleryBrokerBackend
manages all the Celery state (and ties it to the instance, rather than
registering a globalCeleryinstance).Subclasses using Celery simply need to inherit from
BaseCeleryBrokerBackendand overrideget_celery_config()to return
the configuration and broker URI needed.There's currently a single built-in filesystem-based broker. It stores
messages and worker registrations in the site's data directory for local
workers to access. This is a default that will be usable with a future
version of Review Bot and with an upcoming local-only companion worker
responsible for background tasks.Future changes will implement worker scanning, more backends,
configuration, and the local background task companion worker.
Unit tests pass.
Tested the basic functionality in combination with other changes and
a modified Review Bot.
| Summary | ID |
|---|---|
| 8cea3ad0b7ef37b01547b9fc82054e2f82591cf3 |
| Description | From | Last Updated |
|---|---|---|
|
There's no reviewboard/broker/base/__init__.py file (exacerbated by us using namespaces = false in pyproject.toml's tool.setuptools.packages.find) |
|
|
|
BaseBrokerBackend creates a self.logger but everything in here uses a module-level logger. |
|
|
|
This logs the message but never resets the value. We'd then hit the assertion below in dev, or just crash … |
|
|
|
We should probably create with a more locked-down permissions mask. |
|
|
|
I don't know if you've had a chance to look at the spec I put together on tasks, but I … |
|
|
|
This doesn't match the implementation/tests. For send_task, the command name is in the headers (message_headers['task']) and the body is [args, … |
|
|
|
This should probably protect with threading.Lock. It would be nice to also listen to the siteconfig sync (or put in … |
|
|
|
There's a lot of duplication in here. It might be nice to have a _get_broker_settings(broker_id) helper which can validate and … |
|
|
|
This is using a module-level logger instead of self.logger. Same for the other one just below. |
|
|
|
Celery's ctor doesn't define a name arg, but it does take in **kwargs and silently ignore anything it doesn't know … |
|
|
|
These should match the same order as they appear in the signature. |
|
|
|
This is passing instance as a positional arg, but the first argument for KeyValueForm.__init__() is data. We need to explicitly … |
|
|
|
Can we format as: from djblets.registries.registry import ( ALREADY_REGISTERED, ... UNREGISTER, ) |
|
|
|
This is defining the entry point but the class doesn't inherit from EntryPointRegistry. If we fix that we also need … |
|
|
|
Our other registries use %(item)s. As-is this will render as: "<class 'LocalBrokerBackend'>" is already a registered broker backend. (with both … |
|
|
|
Same comment about %(item)s vs %(item)r |
|
|
|
Should be reviewboard.broker.base.backends.BaseBrokerBackend |
|
|
|
undefined name 'JSONDict' Column: 33 Error code: F821 |
|
|
|
It looks like this is using an outdated name for get_celery_config. Same for other tests below. |
|
|
|
Any time we open a file in text mode we should pass encoding=. When that's present I prefer to also … |
|
-
-
There's no
reviewboard/broker/base/__init__.pyfile (exacerbated by us usingnamespaces = falsein pyproject.toml's tool.setuptools.packages.find) -
-
This logs the message but never resets the value. We'd then hit the assertion below in dev, or just crash in other ways in prod.
-
-
I don't know if you've had a chance to look at the spec I put together on tasks, but I don't think we should use a result backend.
Instead, what I had planned out was a task model that could store (smaller) results directly, and larger things (like doc conversion or review bot) end up mapping to domain objects like reviews or file attachments.
If we end up actually using a result backend, this would need to be fixed to use "file://localhost{queue_path}" (no extra slash). Celery strips off 16 characters for "file://localhost". For absolute queue paths that currently ends up with a double leading slash, and if somehow the queue path is a relative path, it would end up adding an extra leading slash turning it into an (incorrect) absolute path.
-
This doesn't match the implementation/tests.
For
send_task, the command name is in the headers (message_headers['task']) and the body is[args, kwargs, embed]as per Celery protocol 2.For
broadcast, the body is a{'method': ..., 'arguments': ...}dict. -
This should probably protect with
threading.Lock.It would be nice to also listen to the siteconfig sync (or put in a TODO comment about that).
-
There's a lot of duplication in here.
It might be nice to have a
_get_broker_settings(broker_id)helper which can validate and return the settings. That would be independently testable, and reusable byBaseBrokerSettingsForm.__init__, which currently does the same lookup but with no validation. -
-
Celery's ctor doesn't define a
namearg, but it does take in**kwargsand silently ignore anything it doesn't know about.I think this was intended to be
main='reviewboard' -
-
This is passing
instanceas a positional arg, but the first argument forKeyValueForm.__init__()isdata. We need to explicitly passinstance=instancehere. -
-
This is defining the entry point but the class doesn't inherit from
EntryPointRegistry.If we fix that we also need a
yield from super().get_defaults()inside theget_defaults()implementation. -
Our other registries use
%(item)s. As-is this will render as:"<class 'LocalBrokerBackend'>" is already a registered broker backend.
(with both quotes and repr). Alternatively just remove the quotes.
-
-
-
-
Any time we open a file in text mode we should pass
encoding=. When that's present I prefer to also be explicit withmode='r'.