Private
Public Access
Supervise the broadcast listener so it recovers from a dropped Redis connection (#76)
The previous fix hardened listen() against a single bad message, but a connection-level failure (Redis restarting, a network blip) raises from the async-for iteration itself, outside that guard, and killed the task permanently and silently -- HTTP/REST and login kept working fine since neither touches it, but live message delivery was gone for every room and every user on the affected process until a manual restart. Confirmed live in production after an OS update restarted Redis: PUBSUB NUMPAT showed zero pattern subscriptions across all 4 gunicorn workers, meaning every one of them had lost its listener with nothing to bring it back. Broadcaster.run_forever() now supervises listen(), restarting it with exponential backoff (capped at 30s, reset after a stable 60s+ run) on any failure, while still letting a genuine shutdown (task cancellation) propagate immediately instead of being treated as just another retry. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
1 parent
86584660fb
commit
850d9ec0fa
3 files changed
+127
-4
No files matched your search
+6
-2
@@ -58,8 +58,12 @@ async def lifespan(app: FastAPI) -> AsyncIterator[None]:
|
||||
# ASGITransport, used by the plain REST test fixtures -- only
|
||||
# TestClient's websocket_connect-based tests actually run lifespan).
|
||||
# The background listener task genuinely needs a running event loop
|
||||
# though, so that part stays here.
|
||||
listener_task = asyncio.create_task(app.state.broadcaster.listen())
|
||||
# though, so that part stays here. run_forever() (not listen() directly)
|
||||
# -- #76: a bare listen() here died permanently and silently the moment
|
||||
# its Redis connection ever dropped (e.g. Redis itself restarting),
|
||||
# with nothing to notice or reconnect; run_forever() supervises it and
|
||||
# restarts on failure instead.
|
||||
listener_task = asyncio.create_task(app.state.broadcaster.run_forever())
|
||||
|
||||
yield
|
||||
|
||||
|
||||
Reference in new issue
Block a user