Add WS heartbeat and harden broadcast against dead sockets (#76)

Messages could silently stop arriving until a room switch happened to
paper over it with a REST refetch. Root cause: a connection that dies
without a clean close (laptop sleep, an idle connection recycled by a
proxy/NAT) looks identical to a healthy-but-quiet one -- readyState
stays OPEN, onclose never fires, so the already-correct reconnect and
rejoin logic never runs.

The backend now pings every 30s and the frontend force-closes the
socket if nothing (any frame, not just a pong) arrives within 65s,
routing straight through the existing reconnect/rejoin path instead of
needing separate recovery logic.

Also found and fixed a more severe adjacent bug while tracing this:
ConnectionManager.broadcast()/send_to_user() had no per-socket error
handling, so one dead or racing socket could raise inside the app's
single, never-restarted Redis pubsub listener task and silently kill
live delivery for every room and every user on that process until
restart. Now caught and logged per-socket instead of propagating.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
ksmithandClaude Sonnet 5 committed 2026-09-23 12:37:58 -06:00
1 parent b1c9a85ab3
commit 6f67f36578
5 files changed
+137 -16

No files matched your search

+23 -8
View File
@@ -1,10 +1,13 @@
import json
import logging
import uuid
from redis.asyncio import Redis
from app.ws.connection_manager import ConnectionManager
logger = logging.getLogger(__name__)
ROOM_CHANNEL_PREFIX = "room:"
USER_CHANNEL_PREFIX = "user:"
@@ -43,14 +46,26 @@ class Broadcaster:
async for message in pubsub.listen():
if message["type"] != "pmessage":
continue
channel = message["channel"]
payload = json.loads(message["data"])
if channel.startswith(ROOM_CHANNEL_PREFIX):
room_id = uuid.UUID(channel.removeprefix(ROOM_CHANNEL_PREFIX))
await self._manager.broadcast(room_id, payload)
elif channel.startswith(USER_CHANNEL_PREFIX):
user_id = uuid.UUID(channel.removeprefix(USER_CHANNEL_PREFIX))
await self._manager.send_to_user(user_id, payload)
# #76: this is the one pubsub listener for the whole process
# (see main.py's lifespan) -- ConnectionManager.broadcast/
# send_to_user already guard against one dead socket taking
# the rest of a single fan-out down, but this belt-and-
# suspenders catch is for anything else unexpected (a
# malformed payload, e.g.) doing the same. An uncaught
# exception here previously meant *this whole async for*
# loop died silently -- no more live delivery to any room on
# this instance, for anyone, until the process restarted.
try:
channel = message["channel"]
payload = json.loads(message["data"])
if channel.startswith(ROOM_CHANNEL_PREFIX):
room_id = uuid.UUID(channel.removeprefix(ROOM_CHANNEL_PREFIX))
await self._manager.broadcast(room_id, payload)
elif channel.startswith(USER_CHANNEL_PREFIX):
user_id = uuid.UUID(channel.removeprefix(USER_CHANNEL_PREFIX))
await self._manager.send_to_user(user_id, payload)
except Exception:
logger.error("Error handling pubsub message on channel %s", message.get("channel"), exc_info=True)
finally:
await pubsub.punsubscribe(f"{ROOM_CHANNEL_PREFIX}*", f"{USER_CHANNEL_PREFIX}*")
await pubsub.aclose()