Phase 8: Production deployment (Debian 13, Nginx Proxy Manager)

Deployment artifacts for the two-server architecture from ARCHITECTURE.md
§9, grounded in verified Debian 13 (trixie) package facts (Python 3.13,
PostgreSQL 17, Node.js 20, redis-server 8.0, certbot 4.0, ufw --
confirmed rather than guessed) rather than a generic "modern Linux" guide:
deploy/systemd/chatapp.service, deploy/chatapp.env.example,
deploy/backup-postgres.sh, deploy/upgrade.sh, and DEPLOYMENT.md as the
actual numbered runbook.

Revised mid-implementation once the user clarified the app sits behind an
existing, separate Nginx Proxy Manager rather than local Nginx+certbot:
dropped the local Nginx config entirely, gunicorn now binds a TCP port
instead of a Unix socket, and app/main.py gained a static-file mount + SPA
fallback route so gunicorn alone serves the built frontend, /api, and /ws
on one port -- what lets NPM's simple one-upstream-per-domain mode work
with zero custom path routing. Path-traversal-guarded (full_path comes
straight from the URL) and cache-header-differentiated (far-future
immutable on Vite's content-hashed assets, no-cache on index.html/sw.js/
manifest so a deploy actually propagates instead of leaving clients on a
stale service worker) -- verified locally against a real gunicorn process
serving a real frontend build, not just eyeballed.

Two real gaps found and fixed alongside the docs, not just noted: gunicorn
wasn't a dependency anywhere despite being the whole app-server design, and
there was no WebSocket reconnect logic on the client -- a reverse proxy's
idle-connection timeout (NPM's or otherwise) would have silently killed a
quiet chat connection with nothing to recover it. Added exponential-backoff
reconnect to useChatSocket.ts, verified by hand (killed and restarted the
local dev backend mid-session, confirmed auto-reconnect and that a message
sends successfully afterward with no page reload).

Every command in DEPLOYMENT.md that could be verified locally, was: the
exact systemd ExecStart line run against local dev Postgres/Redis with
clean SIGTERM shutdown, the static-file serving behavior against a real
build, both shell scripts syntax-checked. What couldn't be verified from
this sandbox (actual Debian 13 hardware, Nginx Proxy Manager itself) is
flagged explicitly in the plan rather than claimed.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

deploy/backup-postgres.sh

@@ -0,0 +1,48 @@
+#!/usr/bin/env bash
+# Nightly Postgres backup for the KeepItTalking data server.
+#
+# Install (as root, on the data server):
+#   sudo cp deploy/backup-postgres.sh /usr/local/bin/chatapp-backup-postgres.sh
+#   sudo chmod 0700 /usr/local/bin/chatapp-backup-postgres.sh
+#   sudo crontab -e
+#     # add:
+#     0 3 * * * /usr/local/bin/chatapp-backup-postgres.sh
+#
+# See ../DEPLOYMENT.md for the full data-server setup this fits into.
+
+set -euo pipefail
+
+DB_NAME="chatapp"
+DB_USER="chatapp"
+BACKUP_DIR="/var/backups/chatapp"
+RETENTION_DAYS=14
+TIMESTAMP="$(date +%F-%H%M%S)"
+DEST="${BACKUP_DIR}/chatapp-${TIMESTAMP}.sql.gz"
+
+mkdir -p "$BACKUP_DIR"
+
+# Runs as the postgres OS user (peer auth) so no password handling here --
+# see DEPLOYMENT.md for why the crontab entry above is on root's crontab
+# calling `sudo -u postgres` implicitly via pg_dump's own permission model.
+sudo -u postgres pg_dump --format=plain --no-owner --dbname="$DB_NAME" \
+  | gzip > "$DEST"
+
+echo "Backed up ${DB_NAME} to ${DEST}"
+
+# Local rotation -- keep RETENTION_DAYS days on this box regardless of
+# whether off-box shipping (below) is configured yet.
+find "$BACKUP_DIR" -name 'chatapp-*.sql.gz' -mtime "+${RETENTION_DAYS}" -delete
+
+# --- Off-box shipping -------------------------------------------------
+# Not configured yet -- destination wasn't decided as of this script being
+# written. Uncomment and fill in ONE of these once you have somewhere to
+# send it; a local-only backup doesn't survive losing this machine.
+#
+# rsync (to a second host reachable by the data server, e.g. over the same
+# private network / a WireGuard tunnel used for anything else):
+#   rsync -a "$DEST" backup-user@backup-host:/path/to/chatapp-backups/
+#
+# S3-compatible object storage (needs `aws configure` or rclone set up
+# separately first):
+#   aws s3 cp "$DEST" s3://your-bucket/chatapp-backups/
+#   # or: rclone copy "$DEST" remote:chatapp-backups/

deploy/chatapp.env.example

@@ -0,0 +1,37 @@
+# /etc/chatapp/env  (production)
+#
+# This file is loaded by systemd's EnvironmentFile= (see
+# deploy/systemd/chatapp.service) directly into the app process's
+# environment -- it is NOT a dotenv file Python reads from a working
+# directory, and it must never be committed to the repository.
+#
+# Install:
+#   sudo mkdir -p /etc/chatapp
+#   sudo cp deploy/chatapp.env.example /etc/chatapp/env
+#   sudo chown root:chatapp /etc/chatapp/env
+#   sudo chmod 0640 /etc/chatapp/env
+#   # then edit in the real values below
+#
+# See ../DEPLOYMENT.md for how each value is generated.
+
+# Points at the data server's PRIVATE address -- never the public one.
+# The role/password here are whatever you created on the data server in
+# DEPLOYMENT.md step 2.
+DATABASE_URL=postgresql+asyncpg://chatapp:REPLACE_ME@<DATA_SERVER_PRIVATE_IP>:5432/chatapp
+
+# Generate with: python3 -c "import secrets; print(secrets.token_urlsafe(32))"
+SESSION_SECRET=REPLACE_ME
+
+# true in production -- cookies are only sent over HTTPS. The local dev
+# default (backend/.env.example) is false because dev runs over plain HTTP.
+SESSION_HTTPS_ONLY=true
+
+# Matches the requirepass set in /etc/redis/redis.conf on the data server
+# (see DEPLOYMENT.md step 2). Same private-address rule as DATABASE_URL.
+REDIS_URL=redis://:REPLACE_ME@<DATA_SERVER_PRIVATE_IP>:6379/0
+
+# Optional: push notifications are silently skipped if these are unset.
+# Generate with: .venv/bin/python -m app.cli generate-vapid-keys
+VAPID_PUBLIC_KEY=
+VAPID_PRIVATE_KEY=
+VAPID_SUBJECT=mailto:you@example.com

deploy/systemd/chatapp.service

@@ -0,0 +1,54 @@
+# /etc/systemd/system/chatapp.service
+#
+# Install: sudo cp deploy/systemd/chatapp.service /etc/systemd/system/
+#          sudo systemctl daemon-reload
+#          sudo systemctl enable --now chatapp
+#
+# See ../../DEPLOYMENT.md for the full app-server setup this fits into.
+# TLS termination and public-facing reverse proxying are handled by an
+# external Nginx Proxy Manager instance, not anything on this box -- this
+# unit just needs to be reachable on the TCP port below.
+
+[Unit]
+Description=KeepItTalking chat service app server
+After=network.target
+
+[Service]
+# No Type= override -- defaults to "simple", which is correct here since
+# gunicorn runs in the foreground (no --daemon flag below) and doesn't send
+# systemd's sd_notify readiness protocol.
+User=chatapp
+Group=chatapp
+WorkingDirectory=/srv/chatapp/backend
+EnvironmentFile=/etc/chatapp/env
+Environment=PYTHONUNBUFFERED=1
+
+# 0.0.0.0 because Nginx Proxy Manager runs on a separate host -- the actual
+# security boundary is the `ufw` rule in DEPLOYMENT.md restricting this
+# port to NPM's IP specifically, not the bind address. If NPM reaches this
+# box over a private network interface, bind to that private IP instead
+# for defense in depth (belt-and-suspenders on top of the firewall rule).
+ExecStart=/srv/chatapp/backend/.venv/bin/gunicorn app.main:app \
+  -k uvicorn.workers.UvicornWorker \
+  --workers 4 \
+  --bind 0.0.0.0:8000 \
+  --timeout 30
+
+# alembic upgrade head deliberately does NOT run here -- with --workers 4,
+# every restart would race multiple processes trying to migrate at once.
+# It's an explicit step in deploy/upgrade.sh instead, run once before the
+# restart that picks up the new code.
+
+Restart=on-failure
+RestartSec=2
+
+# Baseline hardening -- not a full systemd sandboxing pass, just the
+# well-understood safe defaults for a service that doesn't need to write
+# anywhere outside its own working directory.
+NoNewPrivileges=true
+PrivateTmp=true
+ProtectSystem=full
+ProtectHome=true
+
+[Install]
+WantedBy=multi-user.target

deploy/upgrade.sh

@@ -0,0 +1,59 @@
+#!/usr/bin/env bash
+# Day-2 deploy/upgrade script for the KeepItTalking app server. Run by hand
+# over SSH as the `chatapp` user (or via sudo -u chatapp):
+#
+#   sudo -u chatapp /srv/chatapp/deploy/upgrade.sh
+#
+# Fails loudly and stops before touching the running service if any step
+# fails -- the previous deploy keeps running rather than being torn down
+# mid-upgrade. See ../DEPLOYMENT.md for what each step assumes is already
+# in place (venv, /etc/chatapp/env, the systemd unit, Node.js).
+
+set -euo pipefail
+
+REPO_DIR="/srv/chatapp"
+BACKEND_DIR="${REPO_DIR}/backend"
+FRONTEND_DIR="${REPO_DIR}/frontend"
+ENV_FILE="/etc/chatapp/env"
+
+echo "==> Pulling latest code"
+cd "$REPO_DIR"
+git pull --ff-only
+
+echo "==> Installing backend dependencies"
+cd "$BACKEND_DIR"
+.venv/bin/pip install -e . --quiet
+
+echo "==> Running database migrations"
+# alembic reads DATABASE_URL from the environment (backend/alembic/env.py),
+# so the env file has to actually be sourced into this shell first -- it's
+# not read automatically just because systemd's EnvironmentFile= points at
+# it (that only applies to the chatapp.service process, not this script).
+set -a
+# shellcheck disable=SC1090
+source "$ENV_FILE"
+set +a
+.venv/bin/alembic upgrade head
+
+echo "==> Building frontend"
+cd "$FRONTEND_DIR"
+npm ci --silent
+npm run build --silent
+
+echo "==> Restarting chatapp"
+# Active WebSocket connections drop here and reconnect automatically within
+# a few seconds (frontend/src/ws/useChatSocket.ts's exponential-backoff
+# reconnect) -- expected, not a bug, and not worth a blue-green setup for.
+sudo systemctl restart chatapp
+
+echo "==> Verifying"
+sleep 2
+if curl -sf http://127.0.0.1:8000/api/health >/dev/null; then
+  echo "Health check OK"
+else
+  echo "Health check FAILED -- check: sudo journalctl -u chatapp -n 50" >&2
+  exit 1
+fi
+sudo systemctl status chatapp --no-pager -l | head -10
+
+echo "==> Done. journalctl -u chatapp -f to watch logs."