Phase 8: Production deployment (Debian 13, Nginx Proxy Manager)

Deployment artifacts for the two-server architecture from ARCHITECTURE.md
§9, grounded in verified Debian 13 (trixie) package facts (Python 3.13,
PostgreSQL 17, Node.js 20, redis-server 8.0, certbot 4.0, ufw --
confirmed rather than guessed) rather than a generic "modern Linux" guide:
deploy/systemd/chatapp.service, deploy/chatapp.env.example,
deploy/backup-postgres.sh, deploy/upgrade.sh, and DEPLOYMENT.md as the
actual numbered runbook.

Revised mid-implementation once the user clarified the app sits behind an
existing, separate Nginx Proxy Manager rather than local Nginx+certbot:
dropped the local Nginx config entirely, gunicorn now binds a TCP port
instead of a Unix socket, and app/main.py gained a static-file mount + SPA
fallback route so gunicorn alone serves the built frontend, /api, and /ws
on one port -- what lets NPM's simple one-upstream-per-domain mode work
with zero custom path routing. Path-traversal-guarded (full_path comes
straight from the URL) and cache-header-differentiated (far-future
immutable on Vite's content-hashed assets, no-cache on index.html/sw.js/
manifest so a deploy actually propagates instead of leaving clients on a
stale service worker) -- verified locally against a real gunicorn process
serving a real frontend build, not just eyeballed.

Two real gaps found and fixed alongside the docs, not just noted: gunicorn
wasn't a dependency anywhere despite being the whole app-server design, and
there was no WebSocket reconnect logic on the client -- a reverse proxy's
idle-connection timeout (NPM's or otherwise) would have silently killed a
quiet chat connection with nothing to recover it. Added exponential-backoff
reconnect to useChatSocket.ts, verified by hand (killed and restarted the
local dev backend mid-session, confirmed auto-reconnect and that a message
sends successfully afterward with no page reload).

Every command in DEPLOYMENT.md that could be verified locally, was: the
exact systemd ExecStart line run against local dev Postgres/Redis with
clean SIGTERM shutdown, the static-file serving behavior against a real
build, both shell scripts syntax-checked. What couldn't be verified from
this sandbox (actual Debian 13 hardware, Nginx Proxy Manager itself) is
flagged explicitly in the plan rather than claimed.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

DEPLOYMENT.md

@@ -0,0 +1,326 @@
+# Deploying KeepItTalking
+
+Two Debian 13 servers, no containers, matching [ARCHITECTURE.md §9](ARCHITECTURE.md#9-deployment-architecture--two-linux-servers-no-docker):
+
+```
+                    Nginx Proxy Manager (elsewhere in your infra --
+                    terminates TLS, reverse-proxies to the app server)
+                                |
+                                v
+        App server                          Data server
+   Gunicorn + Uvicorn workers  <-------->  PostgreSQL + Redis
+   (systemd, port 8000)         private     (private interface only)
+   serves the built frontend    network
+   + /api + /ws on one port
+```
+
+TLS termination and public-facing reverse proxying are **not** handled on
+the app server — they're handled by an existing, separate Nginx Proxy
+Manager (NPM) instance elsewhere in your infrastructure. The app server just
+needs to be reachable on one TCP port by NPM; §5 below covers what to
+configure in NPM's own UI.
+
+This assumes you already have SSH access (with sudo) to two Debian 13
+machines — no OS-bootstrap/hardening steps here, just app-specific setup.
+Package versions referenced below (Python 3.13, PostgreSQL 17, Node.js 20,
+`redis-server` 8.0) are what Debian 13's own repos ship as of this writing —
+no third-party apt sources needed anywhere in this guide.
+
+Replace every `<PLACEHOLDER>` below with your actual values before running
+a command.
+
+## 1. Prerequisites
+
+- A domain (e.g. `chat.example.com`) — DNS and TLS are handled entirely by
+  Nginx Proxy Manager, so just make sure NPM itself can already reach the
+  app server's address before starting §5.
+- The data server and app server can reach each other over your hosting
+  provider's private network. Find each box's private IP with `ip addr` —
+  ask your provider's docs which interface is the private one if it's not
+  obvious (`eth1`, `ens19`, etc. are common).
+
+## 2. Data server: PostgreSQL + Redis
+
+```bash
+sudo apt update
+sudo apt install -y postgresql redis-server
+```
+
+**PostgreSQL** — create the role and database:
+
+```bash
+sudo -u postgres psql -c "CREATE ROLE chatapp WITH LOGIN PASSWORD '<DB_PASSWORD>';"
+sudo -u postgres psql -c "CREATE DATABASE chatapp OWNER chatapp;"
+```
+
+Bind it to the private interface only (find the exact config path with
+`sudo -u postgres psql -c 'SHOW config_file;'` if 17 isn't your version):
+
+```bash
+sudo sed -i "s/^#\?listen_addresses.*/listen_addresses = 'localhost,<DATA_SERVER_PRIVATE_IP>'/" \
+  /etc/postgresql/17/main/postgresql.conf
+```
+
+Allow the app server in over the private network — Postgres matches
+`pg_hba.conf` rules top-to-bottom, but this one's address is specific
+enough (a single `/32`) that it won't collide with Debian's default
+`127.0.0.1`/`::1`-only entries, so appending is fine:
+
+```bash
+echo "host    chatapp    chatapp    <APP_SERVER_PRIVATE_IP>/32    scram-sha-256" \
+  | sudo tee -a /etc/postgresql/17/main/pg_hba.conf
+sudo systemctl restart postgresql
+```
+
+**Redis** — bind to the private interface and require a password
+(`/etc/redis/redis.conf`):
+
+```bash
+sudo sed -i "s/^bind .*/bind 127.0.0.1 <DATA_SERVER_PRIVATE_IP>/" /etc/redis/redis.conf
+sudo sed -i "s/^# requirepass .*/requirepass <REDIS_PASSWORD>/" /etc/redis/redis.conf
+sudo systemctl restart redis-server
+```
+
+**Firewall** — only the app server's private IP may reach either service:
+
+```bash
+sudo apt install -y ufw
+sudo ufw allow OpenSSH
+sudo ufw allow from <APP_SERVER_PRIVATE_IP> to any port 5432 proto tcp
+sudo ufw allow from <APP_SERVER_PRIVATE_IP> to any port 6379 proto tcp
+sudo ufw enable
+```
+
+**Nightly backups** — see `deploy/backup-postgres.sh`'s own header for the
+install steps (copy it to `/usr/local/bin/`, cron entry). Off-box shipping
+is left as a placeholder in that script — see §8 below.
+
+## 3. App server: Python, Node.js, the `chatapp` user, and the app itself
+
+```bash
+sudo apt update
+sudo apt install -y python3 python3-venv nodejs npm git
+```
+
+### 3a. The `chatapp` system user and directory
+
+```bash
+sudo useradd --system --shell /usr/sbin/nologin --home-dir /srv/chatapp --create-home chatapp
+sudo chown chatapp:chatapp /srv/chatapp
+```
+
+### 3b. Clone the repo (deploy key, not a personal token)
+
+```bash
+sudo -u chatapp mkdir -p /srv/chatapp/.ssh
+sudo -u chatapp ssh-keygen -t ed25519 -f /srv/chatapp/.ssh/id_ed25519 -N ""
+sudo cat /srv/chatapp/.ssh/id_ed25519.pub
+```
+
+Add that public key as a **read-only deploy key** on the Gitea repo
+(Settings → Deploy Keys), then:
+
+```bash
+sudo -u chatapp ssh-keyscan git.darksingularity.org >> /srv/chatapp/.ssh/known_hosts
+sudo -u chatapp git clone git@git.darksingularity.org:DarkSingularity/KeepItTalking.git /srv/chatapp
+```
+
+(If your Gitea's SSH is on a non-default port, adjust the clone URL and
+`ssh-keyscan -p <port>` accordingly.)
+
+### 3c. Backend: venv, env file, migrations, first admin
+
+```bash
+sudo -u chatapp python3 -m venv /srv/chatapp/backend/.venv
+sudo -u chatapp /srv/chatapp/backend/.venv/bin/pip install -e /srv/chatapp/backend
+```
+
+```bash
+sudo mkdir -p /etc/chatapp
+sudo cp /srv/chatapp/deploy/chatapp.env.example /etc/chatapp/env
+sudo chown root:chatapp /etc/chatapp/env
+sudo chmod 0640 /etc/chatapp/env
+sudo -e /etc/chatapp/env   # fill in DATABASE_URL, REDIS_URL, SESSION_SECRET (see below)
+```
+
+Generate `SESSION_SECRET`:
+
+```bash
+python3 -c "import secrets; print(secrets.token_urlsafe(32))"
+```
+
+Run migrations and create the first admin account (as `chatapp`, with the
+env file sourced so `DATABASE_URL` is set):
+
+```bash
+sudo -u chatapp bash -c 'set -a; source /etc/chatapp/env; set +a; \
+  cd /srv/chatapp/backend && .venv/bin/alembic upgrade head'
+
+sudo -u chatapp bash -c 'set -a; source /etc/chatapp/env; set +a; \
+  cd /srv/chatapp/backend && .venv/bin/python -m app.cli create-user <ADMIN_USERNAME> <ADMIN_EMAIL> "<ADMIN_PASSWORD>" --admin'
+```
+
+Optional: push notifications. Skipped silently if `VAPID_PUBLIC_KEY`/
+`VAPID_PRIVATE_KEY` are left unset in `/etc/chatapp/env`. To enable:
+
+```bash
+sudo -u chatapp /srv/chatapp/backend/.venv/bin/python -m app.cli generate-vapid-keys
+# paste the three printed lines into /etc/chatapp/env
+```
+
+### 3d. Frontend build
+
+`backend/app/main.py` serves `frontend/dist` directly (alongside `/api` and
+`/ws`) whenever that directory exists — that's what lets Nginx Proxy
+Manager forward the whole domain to one port with no custom path routing.
+
+```bash
+sudo -u chatapp bash -c 'cd /srv/chatapp/frontend && npm ci && npm run build'
+```
+
+### 3e. systemd unit
+
+```bash
+sudo cp /srv/chatapp/deploy/systemd/chatapp.service /etc/systemd/system/
+sudo systemctl daemon-reload
+sudo systemctl enable --now chatapp
+sudo systemctl status chatapp --no-pager
+```
+
+Confirm it's actually up before continuing:
+
+```bash
+curl -s http://127.0.0.1:8000/api/health   # expect {"status":"ok"}
+```
+
+### 3f. Let `chatapp` restart its own service (needed for `deploy/upgrade.sh`)
+
+```bash
+echo 'chatapp ALL=(root) NOPASSWD: /usr/bin/systemctl restart chatapp, /usr/bin/systemctl status chatapp' \
+  | sudo tee /etc/sudoers.d/chatapp
+sudo chmod 0440 /etc/sudoers.d/chatapp
+sudo visudo -cf /etc/sudoers.d/chatapp   # validates syntax before it's live
+```
+
+### 3g. Firewall
+
+Only Nginx Proxy Manager's address may reach port 8000:
+
+```bash
+sudo apt install -y ufw
+sudo ufw allow OpenSSH
+sudo ufw allow from <NPM_IP> to any port 8000 proto tcp
+sudo ufw enable
+```
+
+If NPM reaches this box over the same private network the data server
+uses, bind gunicorn to that private IP instead of `0.0.0.0` in
+`deploy/systemd/chatapp.service` for defense in depth on top of the
+firewall rule (edit `--bind`, then `daemon-reload` + `restart`).
+
+## 4. Configuring Nginx Proxy Manager
+
+This is config in NPM's own UI/database, not a file this repo ships:
+
+1. **Proxy Hosts → Add Proxy Host**
+2. Domain Names: `chat.example.com`
+3. Scheme: `http`, Forward Hostname/IP: the app server's address (private
+   IP if reachable from NPM, otherwise its public IP — matches whatever you
+   firewalled to NPM's IP in §3g), Forward Port: `8000`
+4. **Websockets Support: ON** — without this, `/ws/chat` won't upgrade and
+   chat won't work at all. This is the one setting that actually matters
+   beyond the basics.
+5. **SSL tab**: request a new Let's Encrypt certificate, enable "Force SSL".
+   The app has no WS-level ping/pong keepalive, so if NPM's own idle-connection
+   timeout ever recycles a quiet chat connection, the client reconnects
+   automatically within a few seconds (`frontend/src/ws/useChatSocket.ts`) —
+   nothing further to tune here unless you want to avoid that churn entirely,
+   in which case raise NPM's proxy read/send timeout in its Advanced tab.
+6. Save.
+
+## 5. First-deploy verification
+
+- `curl -s https://chat.example.com/api/health` → `{"status":"ok"}`
+- Open `https://chat.example.com` in a browser, log in with the admin
+  account from §3c, create a room, send a message, confirm it appears
+  live (WebSocket working).
+- `sudo journalctl -u chatapp -f` on the app server while doing the above —
+  should show request logs, no tracebacks.
+
+## 6. Upgrades
+
+```bash
+sudo -u chatapp /srv/chatapp/deploy/upgrade.sh
+```
+
+Pulls latest `main`, reinstalls backend deps, runs `alembic upgrade head`,
+rebuilds the frontend, restarts `chatapp`, and curls `/api/health` to
+confirm it came back up. Fails loudly (`set -euo pipefail`) and stops
+before restarting anything if an earlier step — most importantly a failed
+migration — errors out, so a bad deploy doesn't take down the previously
+working one.
+
+Active users get disconnected for a few seconds during the restart and
+reconnect automatically (same reconnect logic as §4's NPM-timeout note) —
+expected, not a bug.
+
+**Rollback**: if a deploy goes bad, `git log` to find the last-good commit,
+`git checkout <commit>` on the app server, then re-run the relevant parts of
+`deploy/upgrade.sh` manually (skip the migration step if the bad deploy's
+migration needs to stay applied — there's no automated downgrade story here,
+matching how Alembic is used everywhere else in this project: forward-only
+in practice, downgrades written and tested by hand if one is ever needed).
+
+## 7. Backups
+
+`deploy/backup-postgres.sh` (installed in §2) runs nightly via cron,
+producing a gzipped `pg_dump` in `/var/backups/chatapp/` with 14-day local
+rotation. Off-box shipping is a placeholder in that script (commented-out
+rsync/S3 examples) — decide where those need to go and fill it in.
+
+**Test a restore** (against a scratch database, never directly onto
+`chatapp`):
+
+```bash
+sudo -u postgres createdb chatapp_restore_test
+gunzip -c /var/backups/chatapp/chatapp-<TIMESTAMP>.sql.gz | sudo -u postgres psql chatapp_restore_test
+sudo -u postgres dropdb chatapp_restore_test
+```
+
+## 8. Troubleshooting
+
+- **`chatapp` service won't start**: `sudo journalctl -u chatapp -n 50`.
+  Common causes: `/etc/chatapp/env` missing/malformed (gunicorn workers
+  crash-loop on `pydantic-settings` validation errors), or Postgres/Redis
+  unreachable (check the data-server firewall rules in §2 actually match
+  the app server's real private IP).
+- **502/connection refused from NPM**: confirm `curl
+  http://127.0.0.1:8000/api/health` works *on the app server itself* first
+  (isolates "app is down" from "NPM can't reach it") — then check §3g's
+  `ufw` rule matches NPM's actual source IP.
+- **Migration fails mid-`upgrade.sh`**: the script stops before restarting
+  `chatapp`, so the previous (still-migrated-to-its-old-schema) code keeps
+  running. Fix the migration, re-run the script.
+- **Chat works but disconnects after ~a minute of inactivity, then
+  reconnects**: expected under the current design (§4's NPM timeout note) —
+  not a bug unless it happens mid-typing, in which case raise NPM's proxy
+  timeouts.
+- **Cert renewal**: handled entirely by NPM's own Let's Encrypt integration
+  (not by anything on the app server) — check NPM's own logs if a cert
+  expires unexpectedly.
+
+## 9. Known gaps
+
+Carried forward from earlier phases (see `backend/README.md`'s own "Notes /
+scope decisions" for the full detail on each):
+- No rate limiting on human or bot API traffic.
+- No CSRF token (relies on `SameSite=Lax` cookies).
+- No server-side session revocation (signed cookies only).
+- SSRF protection on outgoing webhooks is creation-time only, not
+  re-validated per delivery (DNS-rebinding gap).
+- Backup off-box shipping is a placeholder — decide a destination and fill
+  in `deploy/backup-postgres.sh`.
+
+None of these are new to this phase — deploying doesn't change any of them,
+just makes them reachable from the internet instead of localhost, which is
+exactly why they're listed here again rather than only in `backend/README.md`.

README.md

@@ -50,6 +50,6 @@ CORS configuration is needed in development.
 
 ## Deployment
 
-Not part of Phase 1. The target is two plain Linux servers with no
+See [DEPLOYMENT.md](DEPLOYMENT.md) for the full production runbook — two
-containers — see [ARCHITECTURE.md §9](ARCHITECTURE.md#9-deployment-architecture--two-linux-servers-no-docker)
+Debian 13 servers, no containers, matching
-and the corresponding "Production deployment" issue in the tracker.
+[ARCHITECTURE.md §9](ARCHITECTURE.md#9-deployment-architecture--two-linux-servers-no-docker).

backend/README.md

@@ -1,4 +1,4 @@
-# KeepItTalking backend (Phase 1 + 2 + 4 + 5 + 6 + 7)
+# KeepItTalking backend (Phase 1 + 2 + 4 + 5 + 6 + 7 + 8)
 
 FastAPI + SQLAlchemy 2.0 (async) + PostgreSQL + Redis. Implements auth, room
 CRUD (open and private), room roles (owner/admin/member) and invites, a
@@ -107,7 +107,8 @@ DATABASE_URL=postgresql+asyncpg://chatapp:chatapp@localhost:5432/chatapp_test .v
 
 ```
 app/
-  main.py            create_app(), session middleware, router/WS mounting
+  main.py            create_app(), session middleware, router/WS mounting,
+                        serves frontend/dist if it exists (see below)
   config.py           environment-driven settings (pydantic-settings)
   database.py          async engine/session, get_db() dependency
   dependencies.py       get_current_user (session cookie or Bearer token),
@@ -128,6 +129,24 @@ alembic/                      migrations
 tests/                         pytest + httpx/TestClient tests
 ```
 
+## Production deployment (Phase 8)
+
+See [`../DEPLOYMENT.md`](../DEPLOYMENT.md) for the full runbook. The one
+piece that lives in this backend's own code: `app/main.py` serves the built
+frontend directly (mounts `frontend/dist/assets` with far-future
+`Cache-Control` on Vite's content-hashed filenames, and a catch-all route
+that serves any other real file under `frontend/dist` or falls back to
+`index.html` for client-side routes like `/rooms/<id>` — `index.html`/
+`sw.js`/`manifest.webmanifest` always get `Cache-Control: no-cache` instead,
+since caching any of those is exactly how a client ends up stuck on a stale
+app version after a deploy) — but only if `frontend/dist` exists at
+startup. It never does in local dev (the Vite dev server handles the
+frontend there instead), so this is fully inert until someone actually runs
+`npm run build`. The point: one Gunicorn port ends up serving the frontend
+*and* `/api` *and* `/ws`, which is what lets a reverse proxy (Nginx Proxy
+Manager, in the deployment this was built for) forward a whole domain to a
+single upstream with no custom per-path routing.
+
 ## Admin portal (Phase 6)
 
 Every `/api/admin/*` route (`app/routers/admin.py`) requires

backend/app/main.py

@@ -1,9 +1,12 @@
 import asyncio
 import contextlib
+import pathlib
 from collections.abc import AsyncIterator
 from contextlib import asynccontextmanager
 
-from fastapi import FastAPI
+from fastapi import FastAPI, HTTPException
+from fastapi.responses import FileResponse, Response
+from fastapi.staticfiles import StaticFiles
 from redis.asyncio import Redis
 from starlette.middleware.sessions import SessionMiddleware
 
@@ -14,6 +17,22 @@ from app.ws.chat import router as ws_router
 from app.ws.connection_manager import ConnectionManager
 from app.ws.presence import Presence
 
+# backend/app/main.py -> backend/ -> repo root -- matches both the local
+# monorepo layout and the production layout (/srv/chatapp/backend,
+# /srv/chatapp/frontend/dist), which is the same relative shape.
+FRONTEND_DIST = pathlib.Path(__file__).resolve().parent.parent.parent / "frontend" / "dist"
+
+
+class ImmutableStaticFiles(StaticFiles):
+    """Vite fingerprints these filenames by content hash, so once served a
+    given path never changes -- safe to cache aggressively and skip
+    revalidation entirely, unlike index.html/sw.js below."""
+
+    async def get_response(self, path: str, scope) -> Response:
+        response = await super().get_response(path, scope)
+        response.headers["Cache-Control"] = "public, max-age=31536000, immutable"
+        return response
+
 
 @asynccontextmanager
 async def lifespan(app: FastAPI) -> AsyncIterator[None]:
@@ -61,6 +80,43 @@ def create_app() -> FastAPI:
     app.include_router(webhooks.router)
     app.include_router(ws_router)
 
+    # Serves the built frontend so a single gunicorn port is enough for a
+    # reverse proxy (e.g. Nginx Proxy Manager) to forward the whole domain
+    # to -- no separate static-file host or per-path proxy routing needed
+    # in front of it. Conditional on frontend/dist existing so local dev
+    # (Vite's own dev server handles the frontend; frontend/dist is never
+    # built there) is unaffected.
+    if FRONTEND_DIST.is_dir():
+        app.mount(
+            "/assets", ImmutableStaticFiles(directory=FRONTEND_DIST / "assets"), name="frontend-assets"
+        )
+
+        # Must always be revalidated -- caching any of these is exactly how
+        # a client ends up stuck on a stale app version after a deploy.
+        # Vite's hashed /assets/ files (ImmutableStaticFiles above) are the
+        # opposite case on purpose: their filename changes when their
+        # content does, so there's nothing to revalidate.
+        NO_CACHE_FILES = {"index.html", "sw.js", "registerSW.js", "manifest.webmanifest"}
+
+        @app.get("/{full_path:path}", include_in_schema=False)
+        async def serve_frontend(full_path: str) -> FileResponse:
+            if full_path.startswith("api/") or full_path.startswith("ws/"):
+                raise HTTPException(status_code=404)
+
+            # .resolve() + is_relative_to() guards against path traversal
+            # (e.g. full_path="../../etc/passwd") -- full_path comes
+            # straight from the URL, and plain Path./ doesn't stop ".."
+            # segments from escaping FRONTEND_DIST on its own.
+            candidate = (FRONTEND_DIST / full_path).resolve()
+            headers = {"Cache-Control": "no-cache"} if full_path in NO_CACHE_FILES else None
+            if full_path and candidate.is_relative_to(FRONTEND_DIST) and candidate.is_file():
+                return FileResponse(candidate, headers=headers)
+
+            # Anything else is a client-side route (e.g. /rooms/<id>) --
+            # fall back to the SPA shell, same as Nginx's `try_files $uri
+            # /index.html` would have done.
+            return FileResponse(FRONTEND_DIST / "index.html", headers={"Cache-Control": "no-cache"})
+
     return app
 
 

backend/pyproject.toml

@@ -17,6 +17,7 @@ dependencies = [
     "pywebpush>=2.0",
     "redis>=5.0",
     "httpx>=0.27",
+    "gunicorn>=23.0",
 ]
 
 [project.scripts]

deploy/backup-postgres.sh

@@ -0,0 +1,48 @@
+#!/usr/bin/env bash
+# Nightly Postgres backup for the KeepItTalking data server.
+#
+# Install (as root, on the data server):
+#   sudo cp deploy/backup-postgres.sh /usr/local/bin/chatapp-backup-postgres.sh
+#   sudo chmod 0700 /usr/local/bin/chatapp-backup-postgres.sh
+#   sudo crontab -e
+#     # add:
+#     0 3 * * * /usr/local/bin/chatapp-backup-postgres.sh
+#
+# See ../DEPLOYMENT.md for the full data-server setup this fits into.
+
+set -euo pipefail
+
+DB_NAME="chatapp"
+DB_USER="chatapp"
+BACKUP_DIR="/var/backups/chatapp"
+RETENTION_DAYS=14
+TIMESTAMP="$(date +%F-%H%M%S)"
+DEST="${BACKUP_DIR}/chatapp-${TIMESTAMP}.sql.gz"
+
+mkdir -p "$BACKUP_DIR"
+
+# Runs as the postgres OS user (peer auth) so no password handling here --
+# see DEPLOYMENT.md for why the crontab entry above is on root's crontab
+# calling `sudo -u postgres` implicitly via pg_dump's own permission model.
+sudo -u postgres pg_dump --format=plain --no-owner --dbname="$DB_NAME" \
+  | gzip > "$DEST"
+
+echo "Backed up ${DB_NAME} to ${DEST}"
+
+# Local rotation -- keep RETENTION_DAYS days on this box regardless of
+# whether off-box shipping (below) is configured yet.
+find "$BACKUP_DIR" -name 'chatapp-*.sql.gz' -mtime "+${RETENTION_DAYS}" -delete
+
+# --- Off-box shipping -------------------------------------------------
+# Not configured yet -- destination wasn't decided as of this script being
+# written. Uncomment and fill in ONE of these once you have somewhere to
+# send it; a local-only backup doesn't survive losing this machine.
+#
+# rsync (to a second host reachable by the data server, e.g. over the same
+# private network / a WireGuard tunnel used for anything else):
+#   rsync -a "$DEST" backup-user@backup-host:/path/to/chatapp-backups/
+#
+# S3-compatible object storage (needs `aws configure` or rclone set up
+# separately first):
+#   aws s3 cp "$DEST" s3://your-bucket/chatapp-backups/
+#   # or: rclone copy "$DEST" remote:chatapp-backups/

deploy/chatapp.env.example

@@ -0,0 +1,37 @@
+# /etc/chatapp/env  (production)
+#
+# This file is loaded by systemd's EnvironmentFile= (see
+# deploy/systemd/chatapp.service) directly into the app process's
+# environment -- it is NOT a dotenv file Python reads from a working
+# directory, and it must never be committed to the repository.
+#
+# Install:
+#   sudo mkdir -p /etc/chatapp
+#   sudo cp deploy/chatapp.env.example /etc/chatapp/env
+#   sudo chown root:chatapp /etc/chatapp/env
+#   sudo chmod 0640 /etc/chatapp/env
+#   # then edit in the real values below
+#
+# See ../DEPLOYMENT.md for how each value is generated.
+
+# Points at the data server's PRIVATE address -- never the public one.
+# The role/password here are whatever you created on the data server in
+# DEPLOYMENT.md step 2.
+DATABASE_URL=postgresql+asyncpg://chatapp:REPLACE_ME@<DATA_SERVER_PRIVATE_IP>:5432/chatapp
+
+# Generate with: python3 -c "import secrets; print(secrets.token_urlsafe(32))"
+SESSION_SECRET=REPLACE_ME
+
+# true in production -- cookies are only sent over HTTPS. The local dev
+# default (backend/.env.example) is false because dev runs over plain HTTP.
+SESSION_HTTPS_ONLY=true
+
+# Matches the requirepass set in /etc/redis/redis.conf on the data server
+# (see DEPLOYMENT.md step 2). Same private-address rule as DATABASE_URL.
+REDIS_URL=redis://:REPLACE_ME@<DATA_SERVER_PRIVATE_IP>:6379/0
+
+# Optional: push notifications are silently skipped if these are unset.
+# Generate with: .venv/bin/python -m app.cli generate-vapid-keys
+VAPID_PUBLIC_KEY=
+VAPID_PRIVATE_KEY=
+VAPID_SUBJECT=mailto:you@example.com

deploy/systemd/chatapp.service

@@ -0,0 +1,54 @@
+# /etc/systemd/system/chatapp.service
+#
+# Install: sudo cp deploy/systemd/chatapp.service /etc/systemd/system/
+#          sudo systemctl daemon-reload
+#          sudo systemctl enable --now chatapp
+#
+# See ../../DEPLOYMENT.md for the full app-server setup this fits into.
+# TLS termination and public-facing reverse proxying are handled by an
+# external Nginx Proxy Manager instance, not anything on this box -- this
+# unit just needs to be reachable on the TCP port below.
+
+[Unit]
+Description=KeepItTalking chat service app server
+After=network.target
+
+[Service]
+# No Type= override -- defaults to "simple", which is correct here since
+# gunicorn runs in the foreground (no --daemon flag below) and doesn't send
+# systemd's sd_notify readiness protocol.
+User=chatapp
+Group=chatapp
+WorkingDirectory=/srv/chatapp/backend
+EnvironmentFile=/etc/chatapp/env
+Environment=PYTHONUNBUFFERED=1
+
+# 0.0.0.0 because Nginx Proxy Manager runs on a separate host -- the actual
+# security boundary is the `ufw` rule in DEPLOYMENT.md restricting this
+# port to NPM's IP specifically, not the bind address. If NPM reaches this
+# box over a private network interface, bind to that private IP instead
+# for defense in depth (belt-and-suspenders on top of the firewall rule).
+ExecStart=/srv/chatapp/backend/.venv/bin/gunicorn app.main:app \
+  -k uvicorn.workers.UvicornWorker \
+  --workers 4 \
+  --bind 0.0.0.0:8000 \
+  --timeout 30
+
+# alembic upgrade head deliberately does NOT run here -- with --workers 4,
+# every restart would race multiple processes trying to migrate at once.
+# It's an explicit step in deploy/upgrade.sh instead, run once before the
+# restart that picks up the new code.
+
+Restart=on-failure
+RestartSec=2
+
+# Baseline hardening -- not a full systemd sandboxing pass, just the
+# well-understood safe defaults for a service that doesn't need to write
+# anywhere outside its own working directory.
+NoNewPrivileges=true
+PrivateTmp=true
+ProtectSystem=full
+ProtectHome=true
+
+[Install]
+WantedBy=multi-user.target

deploy/upgrade.sh

@@ -0,0 +1,59 @@
+#!/usr/bin/env bash
+# Day-2 deploy/upgrade script for the KeepItTalking app server. Run by hand
+# over SSH as the `chatapp` user (or via sudo -u chatapp):
+#
+#   sudo -u chatapp /srv/chatapp/deploy/upgrade.sh
+#
+# Fails loudly and stops before touching the running service if any step
+# fails -- the previous deploy keeps running rather than being torn down
+# mid-upgrade. See ../DEPLOYMENT.md for what each step assumes is already
+# in place (venv, /etc/chatapp/env, the systemd unit, Node.js).
+
+set -euo pipefail
+
+REPO_DIR="/srv/chatapp"
+BACKEND_DIR="${REPO_DIR}/backend"
+FRONTEND_DIR="${REPO_DIR}/frontend"
+ENV_FILE="/etc/chatapp/env"
+
+echo "==> Pulling latest code"
+cd "$REPO_DIR"
+git pull --ff-only
+
+echo "==> Installing backend dependencies"
+cd "$BACKEND_DIR"
+.venv/bin/pip install -e . --quiet
+
+echo "==> Running database migrations"
+# alembic reads DATABASE_URL from the environment (backend/alembic/env.py),
+# so the env file has to actually be sourced into this shell first -- it's
+# not read automatically just because systemd's EnvironmentFile= points at
+# it (that only applies to the chatapp.service process, not this script).
+set -a
+# shellcheck disable=SC1090
+source "$ENV_FILE"
+set +a
+.venv/bin/alembic upgrade head
+
+echo "==> Building frontend"
+cd "$FRONTEND_DIR"
+npm ci --silent
+npm run build --silent
+
+echo "==> Restarting chatapp"
+# Active WebSocket connections drop here and reconnect automatically within
+# a few seconds (frontend/src/ws/useChatSocket.ts's exponential-backoff
+# reconnect) -- expected, not a bug, and not worth a blue-green setup for.
+sudo systemctl restart chatapp
+
+echo "==> Verifying"
+sleep 2
+if curl -sf http://127.0.0.1:8000/api/health >/dev/null; then
+  echo "Health check OK"
+else
+  echo "Health check FAILED -- check: sudo journalctl -u chatapp -n 50" >&2
+  exit 1
+fi
+sudo systemctl status chatapp --no-pager -l | head -10
+
+echo "==> Done. journalctl -u chatapp -f to watch logs."

frontend/src/ws/useChatSocket.ts

@@ -7,6 +7,9 @@ interface UseChatSocketOptions {
   onUnauthenticated: () => void
 }
 
+const RECONNECT_BASE_DELAY_MS = 1000
+const RECONNECT_MAX_DELAY_MS = 30000
+
 export function useChatSocket({ roomId, onMessage, onUnauthenticated }: UseChatSocketOptions) {
   const socketRef = useRef<WebSocket | null>(null)
   const [connected, setConnected] = useState(false)
@@ -16,11 +19,17 @@ export function useChatSocket({ roomId, onMessage, onUnauthenticated }: UseChatS
   onUnauthenticatedRef.current = onUnauthenticated
 
   useEffect(() => {
+    let stopped = false
+    let reconnectDelay = RECONNECT_BASE_DELAY_MS
+    let reconnectTimer: ReturnType<typeof setTimeout> | null = null
+
+    function connect() {
       const protocol = location.protocol === 'https:' ? 'wss' : 'ws'
       const ws = new WebSocket(`${protocol}://${location.host}/ws/chat`)
       socketRef.current = ws
 
       ws.onopen = () => {
+        reconnectDelay = RECONNECT_BASE_DELAY_MS
         setConnected(true)
         ws.send(JSON.stringify({ type: 'join', room_id: roomId }))
       }
@@ -31,13 +40,30 @@ export function useChatSocket({ roomId, onMessage, onUnauthenticated }: UseChatS
 
       ws.onclose = (event) => {
         setConnected(false)
+        socketRef.current = null
+
         if (event.code === 4401) {
           onUnauthenticatedRef.current()
+          return
+        }
+        if (stopped) return
+
+        // Unexpected close -- a deploy restarting the app server, a brief
+        // network blip, or (absent any app-level ping/pong) an idle
+        // connection getting recycled by a reverse proxy in front of it.
+        // Retry with exponential backoff instead of leaving the chat
+        // silently dead until the user manually reloads.
+        reconnectTimer = setTimeout(connect, reconnectDelay)
+        reconnectDelay = Math.min(reconnectDelay * 2, RECONNECT_MAX_DELAY_MS)
       }
     }
 
+    connect()
+
     return () => {
-      ws.close()
+      stopped = true
+      if (reconnectTimer) clearTimeout(reconnectTimer)
+      socketRef.current?.close()
       socketRef.current = null
     }
   }, [roomId])