From f6c71753b522d6818173f71a59aa3f847e15067e Mon Sep 17 00:00:00 2001 From: Keith Smith Date: Fri, 28 Aug 2026 21:27:52 -0600 Subject: [PATCH] Bump to v1.1.0 and bring all documentation current Version bump in both pyproject.toml and package.json. Documentation update covers everything shipped since v1.0.0 (direct messages, message deletion, custom emoji, video attachments, DM/room email notifications, active sessions, and more), and corrects claims that had gone stale -- backend/README.md and DEPLOYMENT.md both still said "no server-side session revocation" and backend/README.md said "no custom/uploaded emoji," both now false. Co-Authored-By: Claude Sonnet 5 --- ARCHITECTURE.md | 37 +++++- DEPLOYMENT.md | 17 +-- README.md | 31 +++-- USER_GUIDE.md | 68 ++++++++-- backend/README.md | 288 ++++++++++++++++++++++++++++++++++------- backend/pyproject.toml | 2 +- frontend/README.md | 66 ++++++---- frontend/package.json | 2 +- 8 files changed, 401 insertions(+), 110 deletions(-) diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index f0e273a..4aa43ed 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -28,7 +28,7 @@ database and Redis; the other runs the application and serves the frontend. | Frontend | React + Vite, vite-plugin-pwa | Generates the manifest and service worker for install + push | | Reverse proxy / TLS | External Nginx Proxy Manager (pre-existing infra, not deployed by this project) | Terminates TLS, forwards REST + WebSocket traffic to the app server's one port. The app itself serves the built static frontend directly — no separate static-asset server needed | | Process management | systemd | No Docker — native to the target Linux distro, no extra install | -| Auth | Session cookies (httpOnly, secure) | Simplest to carry through a WebSocket handshake automatically | +| Auth | Session cookies (httpOnly, secure), backed by a server-side session table | Cookie carries only an opaque session id — simplest way to carry auth through a WebSocket handshake automatically, while still allowing server-side revocation (see §4/§10) | ## 3. System architecture @@ -85,17 +85,31 @@ users id, username, email, password_hash, is_bot, is_site_admin, created_at rooms - id, name, description, is_private, owner_id, created_at + id, name, description, is_private, is_dm, is_archived, owner_id, created_at room_memberships - room_id, user_id, role (owner | admin | member), joined_at + room_id, user_id, role (owner | admin | member), joined_at, last_read_at, + hidden_at (DM-only: hides it from one participant's sidebar), + email_notifications (opt-in, non-DM rooms only) room_invites id, room_id, invited_by, token, target_user_id or target_email, expires_at, status (pending | accepted | revoked) messages - id, room_id, user_id, content, created_at, edited_at, deleted_at + id, room_id, user_id, content, image_id, file_id, created_at, edited_at, + deleted_at + +message_reactions + id, message_id, user_id, emoji (unicode glyph or a custom emoji's + `:shortcode:`) + +custom_emoji + id, shortcode (unique, site-wide), storage_filename, content_type, + uploaded_by, created_at + +sessions + id, user_id, ip_address, user_agent, created_at, last_seen_at, revoked_at push_subscriptions id, user_id, endpoint, p256dh_key, auth_key, created_at @@ -114,6 +128,13 @@ admin_audit_log id, actor_id, action, target_type, target_id, metadata, created_at ``` +Not shown above (present in the implementation, omitted here for brevity): +`message_images`, `message_files`, `message_mentions`, +`message_room_references`, `link_previews`, `custom_themes`, +`upload_settings`, `smtp_settings`, `site_invites`, `password_resets` — see +`backend/README.md` for the full, current model list and per-feature +detail on all of these. + ## 5. Permission model - **Room visibility**: `open` (any authenticated user can find and join) or @@ -133,7 +154,13 @@ admin_audit_log 4. Every app instance subscribed to that channel forwards it to its own connected members over WebSocket. 5. For members with no active connection, the server looks up - `push_subscriptions` and sends a Web Push notification via `pywebpush`. + `push_subscriptions` and sends a Web Push notification via `pywebpush`, + and a native desktop notification over the socket if the client is DS + Chat Desktop. +6. Separately, a genuinely-offline DM recipient always gets an email; a + regular room's member gets one too if they've opted in for that room + (first unread message, or any `@mention` regardless of debounce) — see + `backend/README.md`'s "Email notifications for missed messages". ## 7. Extension system: bots and AI agents diff --git a/DEPLOYMENT.md b/DEPLOYMENT.md index 0262ad5..209a4b8 100644 --- a/DEPLOYMENT.md +++ b/DEPLOYMENT.md @@ -345,11 +345,12 @@ producing a gzipped `pg_dump` in `/var/backups/ds-chat/` with 14-day local rotation. Off-box shipping is a placeholder in that script (commented-out rsync/S3 examples) — decide where those need to go and fill it in. -That script covers Postgres only. Uploaded chat images and file attachments -both live on the **app** server's disk (`/srv/ds-chat/uploads`, created in -§3a) — a separate machine from the data server this script runs on — and -currently have no backup mechanism at all. Whatever off-box destination you -pick above, include `/srv/ds-chat/uploads` in it too (e.g. a second `rsync` +That script covers Postgres only. Uploaded chat images, file/video +attachments, avatars, and custom emoji all live on the **app** server's +disk (`/srv/ds-chat/uploads`, created in §3a) — a separate machine from +the data server this script runs on — and currently have no backup +mechanism at all. Whatever off-box destination you pick above, include +`/srv/ds-chat/uploads` in it too (e.g. a second `rsync` line run from the app server). **Test a restore** (against a scratch database, never directly onto @@ -389,13 +390,13 @@ Carried forward from earlier phases (see `backend/README.md`'s own "Notes / scope decisions" for the full detail on each): - No rate limiting on human or bot API traffic. - No CSRF token (relies on `SameSite=Lax` cookies). -- No server-side session revocation (signed cookies only). - SSRF protection on outgoing webhooks is creation-time only, not re-validated per delivery (DNS-rebinding gap). - Backup off-box shipping is a placeholder — decide a destination and fill in `deploy/backup-postgres.sh`. -- Uploaded chat images and file attachments (`/srv/ds-chat/uploads` on the - app server) have no backup coverage at all yet, on-box or off — see §7. +- Uploaded chat images, file/video attachments, and custom emoji + (`/srv/ds-chat/uploads` on the app server) have no backup coverage at + all yet, on-box or off — see §7. - Uploaded-but-never-sent images or files (a user attaches one, then never hits Send) leak an orphaned file on disk — no cleanup job for this yet. Not a security issue (still gated by room membership to view), just an diff --git a/README.md b/README.md index bf0db63..110a771 100644 --- a/README.md +++ b/README.md @@ -18,21 +18,26 @@ build plan. ## Features -- **Auth & accounts** — session-based auth, invite-only signup (admin-issued - site invites or room invites, both delivered by email), password reset, - per-user light/dark/midnight/sunset presets plus a live theme builder for - fully custom, named, savable color themes. -- **Rooms** — open and private rooms, owner/admin/member roles, invites, - room browsing/search, file/image galleries per room. +- **Auth & accounts** — server-side, revocable sessions (see every device + you're logged in from and sign one out remotely), invite-only signup + (admin-issued site invites or room invites, both delivered by email), + password reset, per-user light/dark/midnight/sunset presets plus a live + theme builder for fully custom, named, savable color themes. +- **Rooms & direct messages** — open and private rooms, owner/admin/member + roles, invites (with resend), room browsing/search, file/image galleries + per room, plus 1:1 direct messages with a collapsible sidebar section. - **Real-time chat** — WebSocket-based messaging with automatic reconnect - and backoff, Markdown rendering, @mentions with autocomplete and inline - highlighting, emoji reactions, message editing, image and file - attachments (drag-and-drop, paste, or picker) with inline previews for - images/PDFs/text/Markdown, unread indicators, and presence (online/away/ + and backoff, Markdown rendering (headings with custom anchors, sub/ + superscript, tables, and more), @mentions and #room-reference links with + autocomplete and inline highlighting, emoji reactions and shortcodes plus + site-wide custom/uploaded emoji, message editing and deletion, image/ + video/file attachments (drag-and-drop, paste, or picker) with inline + previews and playback, unread indicators, and presence (online/away/ offline, with a manual "appear offline" override). -- **Notifications** — Web Push for offline/backgrounded members, with - per-type opt-in/out (mentions vs. all messages), plus in-app unread - badges. +- **Notifications** — Web Push and native desktop notifications for + offline/backgrounded members, plus email: always-on for direct messages, + opt-in per room (first unread message and every mention), all alongside + in-app unread badges. - **PWA** — installable, offline-capable (cached room/message data, a dedicated offline banner), with automatic update detection that prompts a reload as soon as a new deploy goes live. diff --git a/USER_GUIDE.md b/USER_GUIDE.md index 6180a26..3fb66aa 100644 --- a/USER_GUIDE.md +++ b/USER_GUIDE.md @@ -7,15 +7,17 @@ attachments, notifications, and personalizing your account. - [Getting started](#getting-started) - [Rooms](#rooms) +- [Direct messages](#direct-messages) - [Sending messages](#sending-messages) - [Formatting](#formatting) - [Mentions and room links](#mentions-and-room-links) - [Attachments](#attachments) -- [Reactions](#reactions) -- [Editing a message](#editing-a-message) +- [Reactions and custom emoji](#reactions-and-custom-emoji) +- [Editing and deleting a message](#editing-and-deleting-a-message) - [Presence and notifications](#presence-and-notifications) - [Your profile](#your-profile) - [Room details](#room-details) +- [Active sessions](#active-sessions) - [Staying up to date](#staying-up-to-date) ## Getting started @@ -45,6 +47,21 @@ search box at the top to filter by name. - **Unread indicators**: a room with new activity shows a dot next to its name — a plain dot for unread messages, a highlighted dot if you were specifically @mentioned. +- **Collapsible sections**: the **Direct Messages** and **Rooms** headers in + the sidebar can be collapsed to hide their contents — click the header to + toggle. Your choice is remembered. + +## Direct messages + +Click **People** to see everyone on the site and start a 1:1 conversation +with someone — it opens (or reopens, if you've messaged them before) +under **Direct Messages** in the sidebar. There's only ever one +conversation per pair of people, however many times you start it. + +You can hide a conversation you're done with (its own menu, or from its +**Room details** panel) without deleting anything — it drops out of your +sidebar but reappears automatically the moment the other person sends a +new message, or if you message them again yourself. ## Sending messages @@ -61,7 +78,9 @@ Messages support Markdown: - `-` or `1.` for bulleted/numbered lists - `[link text](https://example.com)` — or just paste a bare URL and it becomes clickable automatically -- `#`, `##`, `###` for headings +- `#`, `##`, `###` for headings — add `{#custom-id}` at the end of a + heading line to control its link anchor instead of the auto-generated one +- `~sub~` and `^sup^` for subscript and superscript Pasting a link on its own often also generates a preview card underneath your message, pulled from that page's title/description/image, when the @@ -69,7 +88,9 @@ page provides one. Emoji: click the 🙂 button in the composer to open the emoji picker, or type a shortcode like `:tada:` and it's converted automatically once you -send. The picker remembers your recently-used emoji and has a search box. +send. The picker remembers your recently-used emoji and has a search box — +see [Reactions and custom emoji](#reactions-and-custom-emoji) for uploading +your own. ## Mentions and room links @@ -87,26 +108,39 @@ can see the message (and is a member of that room) straight to it. Click the paperclip icon to attach a file, or just drag a file onto the message box and drop it. Images show as an inline thumbnail — click one -to view it full-size. Other files show as a small card with the -filename and size; `.txt`, `.md`, and `.pdf` files open in a preview -without leaving the room, everything else downloads when clicked. +to view it full-size. Common video formats (MP4, WebM, Ogg) play inline +too, with a button to expand to a larger view; other files show as a +small card with the filename and size — `.txt`, `.md`, and `.pdf` files +open in a preview without leaving the room, everything else downloads +when clicked. There's a server-configured maximum file size — if a file is too large, you'll see an error before it uploads. -## Reactions +## Reactions and custom emoji Hover over a message and click the 🙂 icon in its action row to react with an emoji. Reactions from everyone appear as small pills under the message with a count; click an existing pill to add or remove your own reaction to it. Hovering a pill shows who reacted. -## Editing a message +Anyone can add a custom emoji: open the emoji picker (the 🙂 button, either +in the composer or on a message) and click **+ Add** in the **Custom** +section. Give it a short name and an image — it's then usable by everyone, +both as a reaction and inline in message text via `:your-name:`, right +alongside the built-in picker. You can remove a custom emoji you uploaded +(or any of them, if you're a site admin) from the same picker. + +## Editing and deleting a message You can edit any message you sent: hover it and click **Edit**, make your changes, then press **Enter** to save or **Escape** to cancel. Clicking away also saves. Edited messages are marked *(edited)*. +To delete a message you sent, hover it and click **Delete**. It's replaced +with a "message deleted" placeholder rather than disappearing outright, so +the conversation doesn't visibly shift for anyone else reading it. + ## Presence and notifications Everyone's avatar shows a small status dot — green for online, grey for @@ -124,6 +158,12 @@ The same menu has a notifications toggle: get a notification whenever the app is minimized *or* simply not the focused window, even if it's still open somewhere on screen. +You'll also get an email if someone messages you in a direct conversation +while you're genuinely offline — no setup needed. For regular rooms, +email is opt-in per room: open a room's **Room details** panel and turn +on **Email notifications** to get emailed on that room's first unread +message and on every `@mention`, while you're offline. + ## Your profile Open your account menu and choose **Profile settings** to: @@ -135,12 +175,22 @@ Open your account menu and choose **Profile settings** to: live) - Change your password +## Active sessions + +Profile settings also lists **Active sessions** — every device/browser +currently logged into your account, with its approximate location (IP +address) and when it was last active. If you see one you don't +recognize, click **Revoke** to sign it out immediately. Revoking your own +current device signs you out too. + ## Room details Click the info icon in a room's header to open its details panel, where you can: - See who else is in the room and their role (member/admin/owner) +- Turn on **Email notifications** for that room (see + [Presence and notifications](#presence-and-notifications)) - Browse and re-download every file and image ever shared in the room, without scrolling back through history - Leave the room — unless you're the owner, in which case ownership has diff --git a/backend/README.md b/backend/README.md index 3e14322..f97a7ab 100644 --- a/backend/README.md +++ b/backend/README.md @@ -1,18 +1,22 @@ -# DS Chat backend (Phase 1 + 2 + 4 + 5 + 6 + 7 + 8, image uploads, file attachments, admin-configurable upload size limits, emoji & reactions, user profiles, site invites & email, password reset) +# DS Chat backend (Phase 1 + 2 + 4 + 5 + 6 + 7 + 8, image uploads, file attachments, admin-configurable upload size limits, emoji & reactions, user profiles, site invites & email, password reset, direct messages, active sessions) -FastAPI + SQLAlchemy 2.0 (async) + PostgreSQL + Redis. Implements auth, room -CRUD (open and private), room roles (owner/admin/member) and direct -membership management, a WebSocket chat endpoint that fans out across -multiple app-server instances via Redis pub/sub, Web Push notifications for -offline room members, a site-admin portal (user/room/bot management + an -audit log), a bot/extension layer (scoped API tokens, live bot WebSocket -access, incoming and outgoing webhooks, message editing), image uploads and -generic file attachments in chat messages, emoji reactions on messages, -self-service user profiles -(display name, avatar), self-service password change and a token-based -forgot-password flow, and admin-issued email invites for new accounts -plus email notifications when a user is added to a room. See -`../ARCHITECTURE.md` for the full system design and the phased build plan. +FastAPI + SQLAlchemy 2.0 (async) + PostgreSQL + Redis. Implements auth +(server-side, revocable sessions — see Active sessions below), room CRUD +(open and private), room roles (owner/admin/member) and direct membership +management, direct messages, a WebSocket chat endpoint that fans out across +multiple app-server instances via Redis pub/sub, Web Push and email +notifications for offline room members (plus a native desktop-notification +bridge for DS Chat Desktop), a site-admin portal (user/room/bot management + +an audit log), a bot/extension layer (scoped API tokens, live bot WebSocket +access, incoming and outgoing webhooks, message editing), image uploads, +inline-playable video attachments, and generic file attachments in chat +messages, message deletion, emoji reactions (built-in Unicode plus +site-wide custom/uploaded emoji, both usable in reactions and inline in +message text), self-service user profiles (display name, avatar), +self-service password change and a token-based forgot-password flow, and +admin-issued email invites for new accounts plus email notifications when a +user is added to a room. See `../ARCHITECTURE.md` for the full system +design and the phased build plan. This is an **invite-only site**: there is no public registration endpoint. Accounts are created by an operator on the app server — see step 4 below. @@ -129,10 +133,14 @@ app/ on-disk save/read -- see Image uploads below cli.py `python -m app.cli create-user` / `generate-vapid-keys` models/ SQLAlchemy models (users, rooms, room_memberships, - messages, message_images, message_reactions, + messages, message_images, message_files, + message_reactions, message_mentions, + message_room_references, link_previews, + custom_emoji, custom_themes, sessions, site_invites, password_resets, smtp_settings, - push_subscriptions, admin_audit_log, api_tokens, - webhooks_incoming, event_subscriptions) + upload_settings, push_subscriptions, + admin_audit_log, api_tokens, webhooks_incoming, + event_subscriptions) schemas/ Pydantic request/response models routers/ auth, rooms, users, signup, push, admin, bots, webhooks, health @@ -345,6 +353,36 @@ No `User`/`PushSubscription` schema change was needed for this feature — the only backend change is the new `desktop_notification` envelope type, covered by `backend/tests/test_desktop_notifications.py`. +## Email notifications for missed messages + +Two related, separately-scoped email triggers layered on top of Web Push/ +desktop notifications, both in `app/services/message_events.py`: + +**DMs** (`_maybe_email_dm_notification`) — always on, no opt-in toggle. +Emails a DM's other participant when they're *genuinely* offline +(`GlobalPresence.is_online`, not just "not connected to this room's own +channel" the way the Web Push/desktop-notification audience is computed — +someone actively using the app in a different room shouldn't get emailed +for a DM) or have `appear_offline` set. Debounced to the first unread +message in the conversation, not one email per message in a burst, by +checking whether any other unread message already exists in the room +since the recipient's `last_read_at`. + +**Regular rooms** (`_maybe_email_room_notifications`) — opt-in per +member, per room (`RoomMembership.email_notifications`, toggled via `PATCH +/api/rooms/{id}/notifications`; rejected for a DM — `CannotModifyDmError` +→ 400 — since DMs already get the always-on behavior above). Two triggers, +not one: the room's first unread message debounces the same way DMs do, +*but* a message that `@mentions` the subscriber always emails regardless +of that debounce — a mention is a stronger, individually-addressed signal +that shouldn't get silently absorbed by an earlier plain message in the +same burst already having used up the "first unread" email. + +Both paths go through the same `email_service.send_email` used by +invites/room-membership notifications above, so they're silently skipped +if SMTP isn't configured and never block message delivery on an SMTP +outage. + ## Room roles and membership (Phase 2) Rooms can be `open` (anyone can join via `POST /api/rooms/{id}/join`) or @@ -365,6 +403,47 @@ since nothing meaningful was gained by making the target confirm first. `GET /api/rooms/mine` lists every room (open + private) the current user belongs to, alongside their role. +## Direct messages + +A DM is a `Room` with `is_dm=True` and a deterministic, never-shown +internal `name` (`dm_room_name(user_a, user_b)` — the two user ids sorted +and joined, so it's the same string regardless of who initiates), not a +separate model — `POST /api/rooms/dm` (`find_or_create_dm`) looks up an +existing DM by that name and creates one (`is_private=True`, both +participants as plain `member`) only if none exists yet, so starting a DM +with the same person twice always resolves to the one conversation. The +frontend never renders `Room.name` for a DM; `MyRoomItem.dm_partner` +(`DmPartnerInfo`: the *other* participant's id/username/display +name/avatar/status) is precomputed server-side instead, batched per +request rather than N+1. + +**Hiding a DM**: `RoomMembership.hidden_at` lets one participant remove a +DM from their own sidebar without touching the other participant's copy or +deleting anything — a DM has no sensible "leave" (it would violate +`find_or_create_dm`'s exactly-two-members assumption). `POST +/api/rooms/{id}/hide` sets it; it's cleared automatically (un-hiding the +DM) whenever a new message arrives in it or `find_or_create_dm` resolves +back to an already-hidden one — both count as the conversation being +active again, matching how a re-opened DM in Slack/Discord reappears on +its own rather than needing an explicit "unhide." + +## Message deletion + +`DELETE`-shaped over WS (`{"type": "delete", "room_id", "message_id"}`, +`message_service.delete_message`) — author-only (`NotMessageAuthorError` → +error frame otherwise, no admin/moderator override yet). A real delete of +content, not a UI-only hide: `content`, `image_id`, `file_id`, and +`preview_url` are all cleared and any attached `MessageImage`/`MessageFile` +row (plus its on-disk file) is actually removed, only `deleted_at` (and +`id`/`room_id`/`user_id`/`created_at`, so the tombstone still occupies its +place in history) survives. The attachment's storage filename is read and +the DB row/file only unlinked *after* a successful commit — same ordering +`delete_room` already uses, so a rolled-back transaction never leaves an +already-destroyed file with no way back. Broadcasts +`{"type": "message_deleted", "id", "room_id"}`; the frontend renders a +"message deleted" placeholder rather than removing the row, so the +conversation doesn't visibly shift when someone deletes something above. + ## Image uploads A message can carry an image (`Message.image_id`, nullable), a caption @@ -432,6 +511,18 @@ share them; `ImageTooLargeError` was likewise renamed to Same orphaned-upload disk-space caveat as images applies here too. +**Inline video playback**: a browser-natively-playable video attachment +(`INLINE_SAFE_VIDEO_CONTENT_TYPES` in `app/storage.py` — a strict allowlist, +`video/mp4`/`video/webm`/`video/ogg`, deliberately not "every `video/*` +type") is served *without* the `filename=` param above, so it plays inline +in a `