Supervise the broadcast listener so it recovers from a dropped Redis connection (#76)

The previous fix hardened listen() against a single bad message, but a
connection-level failure (Redis restarting, a network blip) raises from
the async-for iteration itself, outside that guard, and killed the
task permanently and silently -- HTTP/REST and login kept working fine
since neither touches it, but live message delivery was gone for every
room and every user on the affected process until a manual restart.

Confirmed live in production after an OS update restarted Redis:
PUBSUB NUMPAT showed zero pattern subscriptions across all 4 gunicorn
workers, meaning every one of them had lost its listener with nothing
to bring it back.

Broadcaster.run_forever() now supervises listen(), restarting it with
exponential backoff (capped at 30s, reset after a stable 60s+ run) on
any failure, while still letting a genuine shutdown (task
cancellation) propagate immediately instead of being treated as just
another retry.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
ksmithandClaude Sonnet 5 committed 2026-09-26 15:19:59 -06:00
1 parent 86584660fb
commit 850d9ec0fa
3 files changed
+127 -4

No files matched your search

+6 -2
View File
@@ -58,8 +58,12 @@ async def lifespan(app: FastAPI) -> AsyncIterator[None]:
# ASGITransport, used by the plain REST test fixtures -- only
# TestClient's websocket_connect-based tests actually run lifespan).
# The background listener task genuinely needs a running event loop
# though, so that part stays here.
listener_task = asyncio.create_task(app.state.broadcaster.listen())
# though, so that part stays here. run_forever() (not listen() directly)
# -- #76: a bare listen() here died permanently and silently the moment
# its Redis connection ever dropped (e.g. Redis itself restarting),
# with nothing to notice or reconnect; run_forever() supervises it and
# restarts on failure instead.
listener_task = asyncio.create_task(app.state.broadcaster.run_forever())
yield