ds-chat/DEPLOYMENT.md

Deploying DS Chat

Two Debian 13 servers, no containers, matching ARCHITECTURE.md §9:

                    Nginx Proxy Manager (elsewhere in your infra --
                    terminates TLS, reverse-proxies to the app server)
                                |
                                v
        App server                          Data server
   Gunicorn + Uvicorn workers  <-------->  PostgreSQL + Redis
   (systemd, port 8000)         private     (private interface only)
   serves the built frontend    network
   + /api + /ws on one port

TLS termination and public-facing reverse proxying are not handled on the app server — they're handled by an existing, separate Nginx Proxy Manager (NPM) instance elsewhere in your infrastructure. The app server just needs to be reachable on one TCP port by NPM; §5 below covers what to configure in NPM's own UI.

This assumes you already have SSH access (with sudo) to two Debian 13 machines — no OS-bootstrap/hardening steps here, just app-specific setup. Package versions referenced below (Python 3.13, PostgreSQL 17, Node.js 20, redis-server 8.0) are what Debian 13's own repos ship as of this writing — no third-party apt sources needed anywhere in this guide.

Replace every <PLACEHOLDER> below with your actual values before running a command.

1. Prerequisites

• A domain (e.g. chat.example.com) — DNS and TLS are handled entirely by Nginx Proxy Manager, so just make sure NPM itself can already reach the app server's address before starting §5.
• The data server and app server can reach each other over your hosting provider's private network. Find each box's private IP with ip addr — ask your provider's docs which interface is the private one if it's not obvious (eth1, ens19, etc. are common).

2. Data server: PostgreSQL + Redis

sudo apt update
sudo apt install -y postgresql redis-server

PostgreSQL — create the role and database:

sudo -u postgres psql -c "CREATE ROLE ds_chat WITH LOGIN PASSWORD '<DB_PASSWORD>';"
sudo -u postgres psql -c "CREATE DATABASE ds_chat OWNER ds_chat;"

Bind it to the private interface only (find the exact config path with sudo -u postgres psql -c 'SHOW config_file;' if 17 isn't your version):

sudo sed -i "s/^#\?listen_addresses.*/listen_addresses = 'localhost,<DATA_SERVER_PRIVATE_IP>'/" \
  /etc/postgresql/17/main/postgresql.conf

Allow the app server in over the private network — Postgres matches pg_hba.conf rules top-to-bottom, but this one's address is specific enough (a single /32) that it won't collide with Debian's default 127.0.0.1/::1-only entries, so appending is fine:

echo "host    ds_chat    ds_chat    <APP_SERVER_PRIVATE_IP>/32    scram-sha-256" \
  | sudo tee -a /etc/postgresql/17/main/pg_hba.conf
sudo systemctl restart postgresql

Redis — bind to the private interface and require a password (/etc/redis/redis.conf):

sudo sed -i "s/^bind .*/bind 127.0.0.1 <DATA_SERVER_PRIVATE_IP>/" /etc/redis/redis.conf
sudo sed -i "s/^# requirepass .*/requirepass <REDIS_PASSWORD>/" /etc/redis/redis.conf
sudo systemctl restart redis-server

Firewall — only the app server's private IP may reach either service:

sudo apt install -y ufw
sudo ufw allow OpenSSH
sudo ufw allow from <APP_SERVER_PRIVATE_IP> to any port 5432 proto tcp
sudo ufw allow from <APP_SERVER_PRIVATE_IP> to any port 6379 proto tcp
sudo ufw enable

Nightly backups — see deploy/backup-postgres.sh's own header for the install steps (copy it to /usr/local/bin/, cron entry). Off-box shipping is left as a placeholder in that script — see §8 below.

3. App server: Python, Node.js, the ds-chat user, and the app itself

sudo apt update
sudo apt install -y python3 python3-venv nodejs npm git

3a. The ds-chat system user and directory

sudo useradd --system --shell /usr/sbin/nologin --home-dir /srv/ds-chat ds-chat
# Deliberately no --create-home: on Debian, --system + --create-home is
# unreliable in *both* directions -- sometimes it silently skips creating
# the directory at all, sometimes it creates it and populates it from
# /etc/skel (.bashrc, .profile, .bash_logout), which then makes the git
# clone in the next step fail since it refuses to clone into a non-empty
# directory. Creating the directory ourselves, with no skel copying,
# sidesteps both failure modes.
sudo mkdir -p /srv/ds-chat
sudo chown ds-chat:ds-chat /srv/ds-chat

Deliberately not creating /srv/ds-chat/uploads here -- git clone in the next step refuses to clone into a non-empty directory, so anything created inside /srv/ds-chat has to wait until after the clone. (The app would also create this directory itself on first upload if it were missing -- see app/storage.py -- so this step is for clarity/permissions, not a strict requirement.)

3b. Clone the repo (deploy key or access token)

sudo -u ds-chat mkdir -p /srv/ds-chat/.ssh
sudo -u ds-chat ssh-keygen -t ed25519 -f /srv/ds-chat/.ssh/id_ed25519 -N ""
sudo cat /srv/ds-chat/.ssh/id_ed25519.pub

Add that public key as a read-only deploy key on the Gitea repo (Settings → Deploy Keys), then:

sudo -u ds-chat ssh-keyscan git.darksingularity.org >> /srv/ds-chat/.ssh/known_hosts
sudo -u ds-chat git clone git@git.darksingularity.org:DarkSingularity/ds-chat.git /srv/ds-chat

(If your Gitea's SSH is on a non-default port, adjust the clone URL and ssh-keyscan -p <port> accordingly.)

Alternative: a personal/deployment-user access token instead of a deploy key — skip the .ssh/ssh-keygen/ssh-keyscan commands above entirely and clone over HTTPS with the token embedded in the URL:

sudo -u ds-chat git clone https://<TOKEN>@git.darksingularity.org/DarkSingularity/ds-chat.git /srv/ds-chat

The token then lives in plaintext in /srv/ds-chat/.git/config (git remote -v shows it) — readable by root and the ds-chat user, not by anyone else under normal file permissions. deploy/upgrade.sh's later git pulls reuse this same authenticated URL automatically, no extra setup needed. Fine as long as the token is scoped to read-only access on just this repo.

Either way, now that the repo is cloned:

sudo -u ds-chat mkdir -p /srv/ds-chat/uploads

3c. Backend: venv, env file, migrations, first admin

sudo -u ds-chat python3 -m venv /srv/ds-chat/backend/.venv
sudo -u ds-chat /srv/ds-chat/backend/.venv/bin/pip install -e /srv/ds-chat/backend

sudo mkdir -p /etc/ds-chat
sudo cp /srv/ds-chat/deploy/ds-chat.env.example /etc/ds-chat/env
sudo chown root:ds-chat /etc/ds-chat/env
sudo chmod 0640 /etc/ds-chat/env
sudo -e /etc/ds-chat/env   # fill in DATABASE_URL, REDIS_URL, SESSION_SECRET (see below)

Generate SESSION_SECRET:

python3 -c "import secrets; print(secrets.token_urlsafe(32))"

Run migrations (as ds-chat, with the env file sourced so DATABASE_URL is set) — safe as a single non-interactive command since nothing here is a user-chosen value that could contain quotes/spaces/special characters:

sudo -u ds-chat bash -c 'set -a; source /etc/ds-chat/env; set +a; cd /srv/ds-chat/backend && .venv/bin/alembic upgrade head'

Creating the first admin account is different — the password is arbitrary user input, and wrapping it in a second layer of quoting inside the bash -c '...' above is fragile (a space drops the rest of the password as "unrecognized arguments"; a literal ' in the password breaks the outer quoting entirely). Drop into an authenticated shell first instead, so there's only one layer of quoting to get right, at an actual prompt:

sudo -u ds-chat bash -c 'set -a; source /etc/ds-chat/env; set +a; exec bash'

Then, from inside that shell:

cd /srv/ds-chat/backend
.venv/bin/python -m app.cli create-user <ADMIN_USERNAME> <ADMIN_EMAIL> "<ADMIN_PASSWORD>" --admin
exit

Optional: push notifications. Skipped silently if VAPID_PUBLIC_KEY/ VAPID_PRIVATE_KEY are left unset in /etc/ds-chat/env. To enable:

sudo -u ds-chat /srv/ds-chat/backend/.venv/bin/python -m app.cli generate-vapid-keys
# paste the three printed lines into /etc/ds-chat/env

Optional: outgoing email (admin-invited signups, room membership notifications). Unlike everything else on this page, SMTP is not configured here — it's set through the Admin portal's Settings tab at runtime, no redeploy or env file edit needed. Skipped silently (logged, not an error) until an admin sets it up.

3d. Frontend build

backend/app/main.py serves frontend/dist directly (alongside /api and /ws) whenever that directory exists — that's what lets Nginx Proxy Manager forward the whole domain to one port with no custom path routing.

sudo -u ds-chat bash -c 'cd /srv/ds-chat/frontend && npm ci && npm run build'

3e. systemd unit

sudo cp /srv/ds-chat/deploy/systemd/ds-chat.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now ds-chat
sudo systemctl status ds-chat --no-pager

Confirm it's actually up before continuing:

curl -s http://127.0.0.1:8000/api/health   # expect {"status":"ok"}

3f. Let ds-chat restart its own service (needed for deploy/upgrade.sh)

echo 'ds-chat ALL=(root) NOPASSWD: /usr/bin/systemctl restart ds-chat, /usr/bin/systemctl status ds-chat, /usr/bin/systemctl status ds-chat *' \
  | sudo tee /etc/sudoers.d/ds-chat
sudo chmod 0440 /etc/sudoers.d/ds-chat
sudo visudo -cf /etc/sudoers.d/ds-chat   # validates syntax before it's live

The third pattern (... status ds-chat *) matters, not just the bare one: deploy/upgrade.sh actually calls systemctl status ds-chat --no-pager -l, and sudoers matches commands on the exact argument string unless a wildcard is present — the bare status ds-chat entry alone doesn't cover those extra flags, so without this it silently falls back to a password prompt on every upgrade. Since ds-chat has no password (correctly — it's a nologin system account), that prompt can never actually be satisfied, only worked around with Ctrl+C after the (already-succeeded) upgrade.

3g. Firewall

Only Nginx Proxy Manager's address may reach port 8000:

sudo apt install -y ufw
sudo ufw allow OpenSSH
sudo ufw allow from <NPM_IP> to any port 8000 proto tcp
sudo ufw enable

If NPM reaches this box over the same private network the data server uses, bind gunicorn to that private IP instead of 0.0.0.0 in deploy/systemd/ds-chat.service for defense in depth on top of the firewall rule (edit --bind, then daemon-reload + restart).

4. Configuring Nginx Proxy Manager

This is config in NPM's own UI/database, not a file this repo ships:

1. Proxy Hosts → Add Proxy Host
2. Domain Names: chat.example.com
3. Scheme: http, Forward Hostname/IP: the app server's address (private IP if reachable from NPM, otherwise its public IP — matches whatever you firewalled to NPM's IP in §3g), Forward Port: 8000
4. Websockets Support: ON — without this, /ws/chat won't upgrade and chat won't work at all. This is the one setting that actually matters beyond the basics.
5. SSL tab: request a new Let's Encrypt certificate, enable "Force SSL". The app has no WS-level ping/pong keepalive, so if NPM's own idle-connection timeout ever recycles a quiet chat connection, the client reconnects automatically within a few seconds (frontend/src/ws/useChatSocket.ts) — nothing further to tune here unless you want to avoid that churn entirely, in which case raise NPM's proxy read/send timeout in its Advanced tab.
6. Save.

5. First-deploy verification

• curl -s https://chat.example.com/api/health → {"status":"ok"}
• Open https://chat.example.com in a browser, log in with the admin account from §3c, create a room, send a message, confirm it appears live (WebSocket working).
• sudo journalctl -u ds-chat -f on the app server while doing the above — should show request logs, no tracebacks.

6. Upgrades

sudo -u ds-chat /srv/ds-chat/deploy/upgrade.sh

Pulls latest main, reinstalls backend deps, runs alembic upgrade head, rebuilds the frontend, restarts ds-chat, and curls /api/health to confirm it came back up. Fails loudly (set -euo pipefail) and stops before restarting anything if an earlier step — most importantly a failed migration — errors out, so a bad deploy doesn't take down the previously working one.

Active users get disconnected for a few seconds during the restart and reconnect automatically (same reconnect logic as §4's NPM-timeout note) — expected, not a bug.

Rollback: if a deploy goes bad, git log to find the last-good commit, git checkout <commit> on the app server, then re-run the relevant parts of deploy/upgrade.sh manually (skip the migration step if the bad deploy's migration needs to stay applied — there's no automated downgrade story here, matching how Alembic is used everywhere else in this project: forward-only in practice, downgrades written and tested by hand if one is ever needed).

7. Backups

deploy/backup-postgres.sh (installed in §2) runs nightly via cron, producing a gzipped pg_dump in /var/backups/ds-chat/ with 14-day local rotation. Off-box shipping is a placeholder in that script (commented-out rsync/S3 examples) — decide where those need to go and fill it in.

That script covers Postgres only. Uploaded chat images and file attachments both live on the app server's disk (/srv/ds-chat/uploads, created in §3a) — a separate machine from the data server this script runs on — and currently have no backup mechanism at all. Whatever off-box destination you pick above, include /srv/ds-chat/uploads in it too (e.g. a second rsync line run from the app server).

Test a restore (against a scratch database, never directly onto ds_chat):

sudo -u postgres createdb ds_chat_restore_test
gunzip -c /var/backups/ds-chat/ds-chat-<TIMESTAMP>.sql.gz | sudo -u postgres psql ds_chat_restore_test
sudo -u postgres dropdb ds_chat_restore_test

8. Troubleshooting

• ds-chat service won't start: sudo journalctl -u ds-chat -n 50. Common causes: /etc/ds-chat/env missing/malformed (gunicorn workers crash-loop on pydantic-settings validation errors), or Postgres/Redis unreachable (check the data-server firewall rules in §2 actually match the app server's real private IP).
• 502/connection refused from NPM: confirm curl http://127.0.0.1:8000/api/health works on the app server itself first (isolates "app is down" from "NPM can't reach it") — then check §3g's ufw rule matches NPM's actual source IP.
• Migration fails mid-upgrade.sh: the script stops before restarting ds-chat, so the previous (still-migrated-to-its-old-schema) code keeps running. Fix the migration, re-run the script.
• Chat works but disconnects after ~a minute of inactivity, then reconnects: expected under the current design (§4's NPM timeout note) — not a bug unless it happens mid-typing, in which case raise NPM's proxy timeouts.
• Cert renewal: handled entirely by NPM's own Let's Encrypt integration (not by anything on the app server) — check NPM's own logs if a cert expires unexpectedly.

9. Known gaps

Carried forward from earlier phases (see backend/README.md's own "Notes / scope decisions" for the full detail on each):

• No rate limiting on human or bot API traffic.
• No CSRF token (relies on SameSite=Lax cookies).
• No server-side session revocation (signed cookies only).
• SSRF protection on outgoing webhooks is creation-time only, not re-validated per delivery (DNS-rebinding gap).
• Backup off-box shipping is a placeholder — decide a destination and fill in deploy/backup-postgres.sh.
• Uploaded chat images and file attachments (/srv/ds-chat/uploads on the app server) have no backup coverage at all yet, on-box or off — see §7.
• Uploaded-but-never-sent images or files (a user attaches one, then never hits Send) leak an orphaned file on disk — no cleanup job for this yet. Not a security issue (still gated by room membership to view), just an eventual disk-space housekeeping item.

None of these are new to this phase — deploying doesn't change any of them, just makes them reachable from the internet instead of localhost, which is exactly why they're listed here again rather than only in backend/README.md.