Give status updates an optional timeout.
Review Request #8512 — Created Oct. 31, 2016 and submitted
Because of the distributed, asynchronous nature of status updates, it's
possible that some client will create a status update and then drop the ball.
This could happen because of any number of issues, but one real-world example
we've hit is that applications which use rabbitmq can block if the disk fills
up too much (in which case the broker will just hold on to messages instead of
passing them).With this change, a client can set a timeout period for the status update,
measured in seconds. If a status update is listed in the "pending" state for
longer than that period, measured from the timestamp, the "effective state"
will returnTIMEOUT
. Clients with legitimately long-running operations can
keep their stuff in the pending state by pinging, either poking the timestamp
field or setting longer timeout values. If a client sets the state to one of
the finished or error states even after the timeout, that new state will take
priority.
- Tested the database evolution.
- Created a status update with a timeout. Saw that before the timeout expired,
the status update was shown in the UI and API as pending, and after it was
shown as timed out.