ds-chat/DEPLOYMENT.md

Deploying DS Chat

Two Debian 13 servers, no containers, matching ARCHITECTURE.md §9:

                    Nginx Proxy Manager (elsewhere in your infra --
                    terminates TLS, reverse-proxies to the app server)
                                |
                                v
        App server                          Data server
   Gunicorn + Uvicorn workers  <-------->  PostgreSQL + Redis
   (systemd, port 8000)         private     (private interface only)
   serves the built frontend    network
   + /api + /ws on one port

TLS termination and public-facing reverse proxying are not handled on the app server — they're handled by an existing, separate Nginx Proxy Manager (NPM) instance elsewhere in your infrastructure. The app server just needs to be reachable on one TCP port by NPM; §5 below covers what to configure in NPM's own UI.

This assumes you already have SSH access (with sudo) to two Debian 13 machines — no OS-bootstrap/hardening steps here, just app-specific setup. Package versions referenced below (Python 3.13, PostgreSQL 17, Node.js 20, redis-server 8.0) are what Debian 13's own repos ship as of this writing — no third-party apt sources needed anywhere in this guide.

Replace every <PLACEHOLDER> below with your actual values before running a command.

1. Prerequisites

• A domain (e.g. chat.example.com) — DNS and TLS are handled entirely by Nginx Proxy Manager, so just make sure NPM itself can already reach the app server's address before starting §5.
• The data server and app server can reach each other over your hosting provider's private network. Find each box's private IP with ip addr — ask your provider's docs which interface is the private one if it's not obvious (eth1, ens19, etc. are common).

2. Data server: PostgreSQL + Redis

sudo apt update
sudo apt install -y postgresql redis-server

PostgreSQL — create the role and database:

sudo -u postgres psql -c "CREATE ROLE ds_chat WITH LOGIN PASSWORD '<DB_PASSWORD>';"
sudo -u postgres psql -c "CREATE DATABASE ds_chat OWNER ds_chat;"

Bind it to the private interface only (find the exact config path with sudo -u postgres psql -c 'SHOW config_file;' if 17 isn't your version):

sudo sed -i "s/^#\?listen_addresses.*/listen_addresses = 'localhost,<DATA_SERVER_PRIVATE_IP>'/" \
  /etc/postgresql/17/main/postgresql.conf

Allow the app server in over the private network — Postgres matches pg_hba.conf rules top-to-bottom, but this one's address is specific enough (a single /32) that it won't collide with Debian's default 127.0.0.1/::1-only entries, so appending is fine:

echo "host    ds_chat    ds_chat    <APP_SERVER_PRIVATE_IP>/32    scram-sha-256" \
  | sudo tee -a /etc/postgresql/17/main/pg_hba.conf
sudo systemctl restart postgresql

Redis — bind to the private interface and require a password (/etc/redis/redis.conf):

sudo sed -i "s/^bind .*/bind 127.0.0.1 <DATA_SERVER_PRIVATE_IP>/" /etc/redis/redis.conf
sudo sed -i "s/^# requirepass .*/requirepass <REDIS_PASSWORD>/" /etc/redis/redis.conf
sudo systemctl restart redis-server

Firewall — only the app server's private IP may reach either service:

sudo apt install -y ufw
sudo ufw allow OpenSSH
sudo ufw allow from <APP_SERVER_PRIVATE_IP> to any port 5432 proto tcp
sudo ufw allow from <APP_SERVER_PRIVATE_IP> to any port 6379 proto tcp
sudo ufw enable

Nightly backups — see deploy/backup-postgres.sh's own header for the install steps (copy it to /usr/local/bin/, cron entry). Off-box shipping is left as a placeholder in that script — see §8 below.

3. App server: Python, Node.js, the ds-chat user, and the app itself

sudo apt update
sudo apt install -y python3 python3-venv nodejs npm git

3a. The ds-chat system user and directory

sudo useradd --system --shell /usr/sbin/nologin --home-dir /srv/ds-chat --create-home ds-chat
sudo chown ds-chat:ds-chat /srv/ds-chat
sudo -u ds-chat mkdir -p /srv/ds-chat/uploads

3b. Clone the repo (deploy key, not a personal token)

sudo -u ds-chat mkdir -p /srv/ds-chat/.ssh
sudo -u ds-chat ssh-keygen -t ed25519 -f /srv/ds-chat/.ssh/id_ed25519 -N ""
sudo cat /srv/ds-chat/.ssh/id_ed25519.pub

Add that public key as a read-only deploy key on the Gitea repo (Settings → Deploy Keys), then:

sudo -u ds-chat ssh-keyscan git.darksingularity.org >> /srv/ds-chat/.ssh/known_hosts
sudo -u ds-chat git clone git@git.darksingularity.org:DarkSingularity/ds-chat.git /srv/ds-chat

(If your Gitea's SSH is on a non-default port, adjust the clone URL and ssh-keyscan -p <port> accordingly.)

3c. Backend: venv, env file, migrations, first admin

sudo -u ds-chat python3 -m venv /srv/ds-chat/backend/.venv
sudo -u ds-chat /srv/ds-chat/backend/.venv/bin/pip install -e /srv/ds-chat/backend

sudo mkdir -p /etc/ds-chat
sudo cp /srv/ds-chat/deploy/ds-chat.env.example /etc/ds-chat/env
sudo chown root:ds-chat /etc/ds-chat/env
sudo chmod 0640 /etc/ds-chat/env
sudo -e /etc/ds-chat/env   # fill in DATABASE_URL, REDIS_URL, SESSION_SECRET (see below)

Generate SESSION_SECRET:

python3 -c "import secrets; print(secrets.token_urlsafe(32))"

Run migrations and create the first admin account (as ds-chat, with the env file sourced so DATABASE_URL is set):

sudo -u ds-chat bash -c 'set -a; source /etc/ds-chat/env; set +a; \
  cd /srv/ds-chat/backend && .venv/bin/alembic upgrade head'

sudo -u ds-chat bash -c 'set -a; source /etc/ds-chat/env; set +a; \
  cd /srv/ds-chat/backend && .venv/bin/python -m app.cli create-user <ADMIN_USERNAME> <ADMIN_EMAIL> "<ADMIN_PASSWORD>" --admin'

Optional: push notifications. Skipped silently if VAPID_PUBLIC_KEY/ VAPID_PRIVATE_KEY are left unset in /etc/ds-chat/env. To enable:

sudo -u ds-chat /srv/ds-chat/backend/.venv/bin/python -m app.cli generate-vapid-keys
# paste the three printed lines into /etc/ds-chat/env

Optional: outgoing email (admin-invited signups, room membership notifications). Unlike everything else on this page, SMTP is not configured here — it's set through the Admin portal's Settings tab at runtime, no redeploy or env file edit needed. Skipped silently (logged, not an error) until an admin sets it up.

3d. Frontend build

backend/app/main.py serves frontend/dist directly (alongside /api and /ws) whenever that directory exists — that's what lets Nginx Proxy Manager forward the whole domain to one port with no custom path routing.

sudo -u ds-chat bash -c 'cd /srv/ds-chat/frontend && npm ci && npm run build'

3e. systemd unit

sudo cp /srv/ds-chat/deploy/systemd/ds-chat.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now ds-chat
sudo systemctl status ds-chat --no-pager

Confirm it's actually up before continuing:

curl -s http://127.0.0.1:8000/api/health   # expect {"status":"ok"}

3f. Let ds-chat restart its own service (needed for deploy/upgrade.sh)

echo 'ds-chat ALL=(root) NOPASSWD: /usr/bin/systemctl restart ds-chat, /usr/bin/systemctl status ds-chat' \
  | sudo tee /etc/sudoers.d/ds-chat
sudo chmod 0440 /etc/sudoers.d/ds-chat
sudo visudo -cf /etc/sudoers.d/ds-chat   # validates syntax before it's live

3g. Firewall

Only Nginx Proxy Manager's address may reach port 8000:

sudo apt install -y ufw
sudo ufw allow OpenSSH
sudo ufw allow from <NPM_IP> to any port 8000 proto tcp
sudo ufw enable

If NPM reaches this box over the same private network the data server uses, bind gunicorn to that private IP instead of 0.0.0.0 in deploy/systemd/ds-chat.service for defense in depth on top of the firewall rule (edit --bind, then daemon-reload + restart).

4. Configuring Nginx Proxy Manager

This is config in NPM's own UI/database, not a file this repo ships:

1. Proxy Hosts → Add Proxy Host
2. Domain Names: chat.example.com
3. Scheme: http, Forward Hostname/IP: the app server's address (private IP if reachable from NPM, otherwise its public IP — matches whatever you firewalled to NPM's IP in §3g), Forward Port: 8000
4. Websockets Support: ON — without this, /ws/chat won't upgrade and chat won't work at all. This is the one setting that actually matters beyond the basics.
5. SSL tab: request a new Let's Encrypt certificate, enable "Force SSL". The app has no WS-level ping/pong keepalive, so if NPM's own idle-connection timeout ever recycles a quiet chat connection, the client reconnects automatically within a few seconds (frontend/src/ws/useChatSocket.ts) — nothing further to tune here unless you want to avoid that churn entirely, in which case raise NPM's proxy read/send timeout in its Advanced tab.
6. Save.

5. First-deploy verification

• curl -s https://chat.example.com/api/health → {"status":"ok"}
• Open https://chat.example.com in a browser, log in with the admin account from §3c, create a room, send a message, confirm it appears live (WebSocket working).
• sudo journalctl -u ds-chat -f on the app server while doing the above — should show request logs, no tracebacks.

6. Upgrades

sudo -u ds-chat /srv/ds-chat/deploy/upgrade.sh

Pulls latest main, reinstalls backend deps, runs alembic upgrade head, rebuilds the frontend, restarts ds-chat, and curls /api/health to confirm it came back up. Fails loudly (set -euo pipefail) and stops before restarting anything if an earlier step — most importantly a failed migration — errors out, so a bad deploy doesn't take down the previously working one.

Active users get disconnected for a few seconds during the restart and reconnect automatically (same reconnect logic as §4's NPM-timeout note) — expected, not a bug.

Rollback: if a deploy goes bad, git log to find the last-good commit, git checkout <commit> on the app server, then re-run the relevant parts of deploy/upgrade.sh manually (skip the migration step if the bad deploy's migration needs to stay applied — there's no automated downgrade story here, matching how Alembic is used everywhere else in this project: forward-only in practice, downgrades written and tested by hand if one is ever needed).

7. Backups

deploy/backup-postgres.sh (installed in §2) runs nightly via cron, producing a gzipped pg_dump in /var/backups/ds-chat/ with 14-day local rotation. Off-box shipping is a placeholder in that script (commented-out rsync/S3 examples) — decide where those need to go and fill it in.

That script covers Postgres only. Uploaded chat images and file attachments both live on the app server's disk (/srv/ds-chat/uploads, created in §3a) — a separate machine from the data server this script runs on — and currently have no backup mechanism at all. Whatever off-box destination you pick above, include /srv/ds-chat/uploads in it too (e.g. a second rsync line run from the app server).

Test a restore (against a scratch database, never directly onto ds_chat):

sudo -u postgres createdb ds_chat_restore_test
gunzip -c /var/backups/ds-chat/ds-chat-<TIMESTAMP>.sql.gz | sudo -u postgres psql ds_chat_restore_test
sudo -u postgres dropdb ds_chat_restore_test

8. Troubleshooting

• ds-chat service won't start: sudo journalctl -u ds-chat -n 50. Common causes: /etc/ds-chat/env missing/malformed (gunicorn workers crash-loop on pydantic-settings validation errors), or Postgres/Redis unreachable (check the data-server firewall rules in §2 actually match the app server's real private IP).
• 502/connection refused from NPM: confirm curl http://127.0.0.1:8000/api/health works on the app server itself first (isolates "app is down" from "NPM can't reach it") — then check §3g's ufw rule matches NPM's actual source IP.
• Migration fails mid-upgrade.sh: the script stops before restarting ds-chat, so the previous (still-migrated-to-its-old-schema) code keeps running. Fix the migration, re-run the script.
• Chat works but disconnects after ~a minute of inactivity, then reconnects: expected under the current design (§4's NPM timeout note) — not a bug unless it happens mid-typing, in which case raise NPM's proxy timeouts.
• Cert renewal: handled entirely by NPM's own Let's Encrypt integration (not by anything on the app server) — check NPM's own logs if a cert expires unexpectedly.

9. Known gaps

Carried forward from earlier phases (see backend/README.md's own "Notes / scope decisions" for the full detail on each):

• No rate limiting on human or bot API traffic.
• No CSRF token (relies on SameSite=Lax cookies).
• No server-side session revocation (signed cookies only).
• SSRF protection on outgoing webhooks is creation-time only, not re-validated per delivery (DNS-rebinding gap).
• Backup off-box shipping is a placeholder — decide a destination and fill in deploy/backup-postgres.sh.
• Uploaded chat images and file attachments (/srv/ds-chat/uploads on the app server) have no backup coverage at all yet, on-box or off — see §7.
• Uploaded-but-never-sent images or files (a user attaches one, then never hits Send) leak an orphaned file on disk — no cleanup job for this yet. Not a security issue (still gated by room membership to view), just an eventual disk-space housekeeping item.

None of these are new to this phase — deploying doesn't change any of them, just makes them reachable from the internet instead of localhost, which is exactly why they're listed here again rather than only in backend/README.md.