Phase 5: Redis pub/sub for horizontal scaling

Splits the WebSocket layer into three pieces so one app instance and many
behave identically: ConnectionManager stays a purely local socket registry;
RoomBroadcaster publishes chat messages to a per-room Redis channel and
every instance (including the publisher) forwards received messages to its
own local sockets via a single psubscribe("room:*") listener started in
main.py's lifespan; Presence is a Redis-backed refcounted hash per room
tracking who's connected across all instances.

Presence replaces the old process-local connected_user_ids check that
Phase 4's offline-push logic used -- without it, a user connected on a
different instance would look offline and get a redundant push. Fixing
this was scoped in beyond the issue's literal ask (message fan-out only)
since it's a real correctness gap in a phase specifically about running
more than one instance; a known limitation (no heartbeat/TTL, so a hard
crash leaks a presence increment) is documented in the README instead of
solved here.

New tests/test_broadcast.py spins up two independent app instances sharing
one Postgres + Redis to prove delivery and presence both actually cross
the Redis boundary, not just work in-process. Manually verified the same
thing against two real uvicorn processes on different ports.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
2026-08-14 07:13:06 -06:00
co-authored by Claude Sonnet 5
parent d09bf4a30a
commit 0b995ef75f
11 changed files with 328 additions and 57 deletions
+58 -17
View File
@@ -1,10 +1,10 @@
# KeepItTalking backend (Phase 1 + 2 + 4)
# KeepItTalking backend (Phase 1 + 2 + 4 + 5)
FastAPI + SQLAlchemy 2.0 (async) + PostgreSQL. Implements auth, room CRUD
(open and private), room roles (owner/admin/member) and invites, a
single-instance WebSocket chat endpoint, and Web Push notifications for
offline room members. See `../ARCHITECTURE.md` for the full system design
and the phased build plan.
FastAPI + SQLAlchemy 2.0 (async) + PostgreSQL + Redis. Implements auth, room
CRUD (open and private), room roles (owner/admin/member) and invites, a
WebSocket chat endpoint that fans out across multiple app-server instances
via Redis pub/sub, and Web Push notifications for offline room members. See
`../ARCHITECTURE.md` for the full system design and the phased build plan.
This is an **invite-only site**: there is no public registration endpoint.
Accounts are created by an operator on the app server — see step 4 below.
@@ -30,7 +30,16 @@ docker exec chatapp-postgres psql -U chatapp -d chatapp -c "CREATE DATABASE chat
(Docker here is purely a local-dev convenience for standing up Postgres quickly —
the actual deployment target has no containers at all, see `ARCHITECTURE.md` §9.)
### 2. Python environment
### 2. Redis
Used for cross-instance WebSocket fan-out and presence (see the section
below). Required — there's no in-memory fallback.
```bash
docker run -d --name chatapp-redis -p 6379:6379 redis:7-alpine
```
### 3. Python environment
```bash
cd backend
@@ -41,13 +50,13 @@ cp .env.example .env
# python3 -c "import secrets; print(secrets.token_urlsafe(32))"
```
### 3. Migrations
### 4. Migrations
```bash
.venv/bin/alembic upgrade head
```
### 4. Create a user
### 5. Create a user
There's no public sign-up. Create accounts directly with the CLI (add
`--admin` to grant `is_site_admin`, useful ahead of the phase-6 admin portal):
@@ -56,7 +65,7 @@ There's no public sign-up. Create accounts directly with the CLI (add
.venv/bin/python -m app.cli create-user alice alice@example.com "some-password"
```
### 5. (Optional) Set up push notifications
### 6. (Optional) Set up push notifications
Push works without any setup — `VAPID_PUBLIC_KEY`/`VAPID_PRIVATE_KEY` are
unset by default and push delivery is silently skipped. To enable it:
@@ -66,7 +75,7 @@ unset by default and push delivery is silently skipped. To enable it:
# paste the three printed lines into backend/.env
```
### 6. Run the dev server
### 7. Run the dev server
```bash
.venv/bin/uvicorn app.main:app --reload
@@ -74,10 +83,16 @@ unset by default and push delivery is silently skipped. To enable it:
API docs: http://localhost:8000/docs. WebSocket chat endpoint: `ws://localhost:8000/ws/chat`.
### 7. Run tests
To try horizontal scaling locally, run a second instance on another port
against the same Postgres + Redis (`.venv/bin/uvicorn app.main:app --port 8001`)
— a message sent through one instance's WebSocket is delivered to clients
connected to the other, purely via Redis.
### 8. Run tests
Tests run against a real Postgres database (`chatapp_test` by default — native
`ENUM`/`UUID` types aren't faithfully reproduced by SQLite), with each test
`ENUM`/`UUID` types aren't faithfully reproduced by SQLite) and a real Redis
(db 15 by default, kept separate from dev use of db 0), with each test
wrapped in a transaction that's rolled back afterward:
```bash
@@ -99,20 +114,46 @@ app/
schemas/ Pydantic request/response models
routers/ auth, rooms, invites, push, health
services/ business logic called by routers
ws/ WebSocket connection manager + /ws/chat handler
ws/ connection_manager (local sockets), presence +
broadcaster (Redis), /ws/chat handler
alembic/ migrations
tests/ pytest + httpx/TestClient tests
```
## Cross-instance broadcast (Phase 5)
The WebSocket layer is split into three pieces so that running one app
instance and running many behave identically:
- `app/ws/connection_manager.py` — purely local: which sockets on *this*
process are in which room, used only to actually `send_json` to them.
- `app/ws/broadcaster.py` (`RoomBroadcaster`) — on a chat message,
`publish()`s it to a Redis channel scoped to the room (`room:{id}`).
Every app instance, including the publisher, runs a single background
`listen()` task (started in `app/main.py`'s lifespan) pattern-subscribed
to `room:*`; each message it receives is handed to its own local
`ConnectionManager.broadcast()`. One instance just talks to itself
through Redis, so there's no separate single-instance code path.
- `app/ws/presence.py` (`Presence`) — a Redis hash per room
(`presence:{room_id}`, field = user ID, value = connection refcount) is
the cross-instance answer to "is this member connected *anywhere* right
now," which is what the Phase 4 offline-push check uses instead of the
local `ConnectionManager`. Refcounted so a user connected from two tabs
(or two instances) isn't marked offline until every connection closes.
Known limitation: `Presence` has no heartbeat/TTL, so a hard process crash
(not a clean disconnect) leaks that connection's increment forever — same
category of simplification as the "no server-side session revocation" note
below.
## Push notifications (Phase 4)
`POST /api/push/subscribe` (upserts by `endpoint`) / `DELETE /api/push/subscribe`
manage a user's `push_subscriptions` rows; `GET /api/push/vapid-public-key` gives
the frontend the key it needs for `PushManager.subscribe()`. On every chat
message, `app/ws/chat.py` computes `room members - ConnectionManager.
message, `app/ws/chat.py` computes `room members - Presence.
connected_user_ids(room_id)` (who's actually connected to *that room* right
now, tracked alongside the existing WebSocket registry) and sends each
offline member a push via `pywebpush`, awaited inline against the same
now, across every app instance — see Phase 5 below) and sends each offline
member a push via `pywebpush`, awaited inline against the same
request-scoped session rather than fired as a background task — the
broadcast to online members already happened by that point, so nothing
online-facing is delayed, and it sidesteps `asyncio.create_task()`s outliving