The previous fix hardened listen() against a single bad message, but a
connection-level failure (Redis restarting, a network blip) raises from
the async-for iteration itself, outside that guard, and killed the
task permanently and silently -- HTTP/REST and login kept working fine
since neither touches it, but live message delivery was gone for every
room and every user on the affected process until a manual restart.
Confirmed live in production after an OS update restarted Redis:
PUBSUB NUMPAT showed zero pattern subscriptions across all 4 gunicorn
workers, meaning every one of them had lost its listener with nothing
to bring it back.
Broadcaster.run_forever() now supervises listen(), restarting it with
exponential backoff (capped at 30s, reset after a stable 60s+ run) on
any failure, while still letting a genuine shutdown (task
cancellation) propagate immediately instead of being treated as just
another retry.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Messages could silently stop arriving until a room switch happened to
paper over it with a REST refetch. Root cause: a connection that dies
without a clean close (laptop sleep, an idle connection recycled by a
proxy/NAT) looks identical to a healthy-but-quiet one -- readyState
stays OPEN, onclose never fires, so the already-correct reconnect and
rejoin logic never runs.
The backend now pings every 30s and the frontend force-closes the
socket if nothing (any frame, not just a pong) arrives within 65s,
routing straight through the existing reconnect/rejoin path instead of
needing separate recovery logic.
Also found and fixed a more severe adjacent bug while tracing this:
ConnectionManager.broadcast()/send_to_user() had no per-socket error
handling, so one dead or racing socket could raise inside the app's
single, never-restarted Redis pubsub listener task and silently kill
live delivery for every room and every user on that process until
restart. Now caught and logged per-socket instead of propagating.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Previously GET /api/rooms/mine was only ever fetched once at app mount,
so a room added mid-session stayed invisible until a full page reload
-- add_member had no way to reach an already-open client at all.
Backend: ConnectionManager and Broadcaster (renamed from RoomBroadcaster)
now support per-user channels alongside the existing per-room ones, so a
signal can reach a user's socket even for a room they haven't joined
(and by definition can't have, until this fires). add_member publishes
a room_added event on the target user's channel.
Frontend: the WebSocket connection is no longer scoped to whichever
room is open -- ChatShellPage now owns one persistent connection for
the whole session (including while no room is open, which is exactly
when this bug showed), and ChatPane joins/leaves rooms on top of it.
A room_added event triggers a room-list refetch with no reload needed.
Verified end-to-end in the browser: a user sitting on the empty room
list saw a newly-added room appear live, then chatted in it normally.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Splits the WebSocket layer into three pieces so one app instance and many
behave identically: ConnectionManager stays a purely local socket registry;
RoomBroadcaster publishes chat messages to a per-room Redis channel and
every instance (including the publisher) forwards received messages to its
own local sockets via a single psubscribe("room:*") listener started in
main.py's lifespan; Presence is a Redis-backed refcounted hash per room
tracking who's connected across all instances.
Presence replaces the old process-local connected_user_ids check that
Phase 4's offline-push logic used -- without it, a user connected on a
different instance would look offline and get a redundant push. Fixing
this was scoped in beyond the issue's literal ask (message fan-out only)
since it's a real correctness gap in a phase specifically about running
more than one instance; a known limitation (no heartbeat/TTL, so a hard
crash leaks a presence increment) is documented in the README instead of
solved here.
New tests/test_broadcast.py spins up two independent app instances sharing
one Postgres + Redis to prove delivery and presence both actually cross
the Redis boundary, not just work in-process. Manually verified the same
thing against two real uvicorn processes on different ports.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
ksmithandClaude Sonnet 5