diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index f0e273a..4aa43ed 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -28,7 +28,7 @@ database and Redis; the other runs the application and serves the frontend. | Frontend | React + Vite, vite-plugin-pwa | Generates the manifest and service worker for install + push | | Reverse proxy / TLS | External Nginx Proxy Manager (pre-existing infra, not deployed by this project) | Terminates TLS, forwards REST + WebSocket traffic to the app server's one port. The app itself serves the built static frontend directly — no separate static-asset server needed | | Process management | systemd | No Docker — native to the target Linux distro, no extra install | -| Auth | Session cookies (httpOnly, secure) | Simplest to carry through a WebSocket handshake automatically | +| Auth | Session cookies (httpOnly, secure), backed by a server-side session table | Cookie carries only an opaque session id — simplest way to carry auth through a WebSocket handshake automatically, while still allowing server-side revocation (see §4/§10) | ## 3. System architecture @@ -85,17 +85,31 @@ users id, username, email, password_hash, is_bot, is_site_admin, created_at rooms - id, name, description, is_private, owner_id, created_at + id, name, description, is_private, is_dm, is_archived, owner_id, created_at room_memberships - room_id, user_id, role (owner | admin | member), joined_at + room_id, user_id, role (owner | admin | member), joined_at, last_read_at, + hidden_at (DM-only: hides it from one participant's sidebar), + email_notifications (opt-in, non-DM rooms only) room_invites id, room_id, invited_by, token, target_user_id or target_email, expires_at, status (pending | accepted | revoked) messages - id, room_id, user_id, content, created_at, edited_at, deleted_at + id, room_id, user_id, content, image_id, file_id, created_at, edited_at, + deleted_at + +message_reactions + id, message_id, user_id, emoji (unicode glyph or a custom emoji's + `:shortcode:`) + +custom_emoji + id, shortcode (unique, site-wide), storage_filename, content_type, + uploaded_by, created_at + +sessions + id, user_id, ip_address, user_agent, created_at, last_seen_at, revoked_at push_subscriptions id, user_id, endpoint, p256dh_key, auth_key, created_at @@ -114,6 +128,13 @@ admin_audit_log id, actor_id, action, target_type, target_id, metadata, created_at ``` +Not shown above (present in the implementation, omitted here for brevity): +`message_images`, `message_files`, `message_mentions`, +`message_room_references`, `link_previews`, `custom_themes`, +`upload_settings`, `smtp_settings`, `site_invites`, `password_resets` — see +`backend/README.md` for the full, current model list and per-feature +detail on all of these. + ## 5. Permission model - **Room visibility**: `open` (any authenticated user can find and join) or @@ -133,7 +154,13 @@ admin_audit_log 4. Every app instance subscribed to that channel forwards it to its own connected members over WebSocket. 5. For members with no active connection, the server looks up - `push_subscriptions` and sends a Web Push notification via `pywebpush`. + `push_subscriptions` and sends a Web Push notification via `pywebpush`, + and a native desktop notification over the socket if the client is DS + Chat Desktop. +6. Separately, a genuinely-offline DM recipient always gets an email; a + regular room's member gets one too if they've opted in for that room + (first unread message, or any `@mention` regardless of debounce) — see + `backend/README.md`'s "Email notifications for missed messages". ## 7. Extension system: bots and AI agents diff --git a/DEPLOYMENT.md b/DEPLOYMENT.md index 0262ad5..209a4b8 100644 --- a/DEPLOYMENT.md +++ b/DEPLOYMENT.md @@ -345,11 +345,12 @@ producing a gzipped `pg_dump` in `/var/backups/ds-chat/` with 14-day local rotation. Off-box shipping is a placeholder in that script (commented-out rsync/S3 examples) — decide where those need to go and fill it in. -That script covers Postgres only. Uploaded chat images and file attachments -both live on the **app** server's disk (`/srv/ds-chat/uploads`, created in -§3a) — a separate machine from the data server this script runs on — and -currently have no backup mechanism at all. Whatever off-box destination you -pick above, include `/srv/ds-chat/uploads` in it too (e.g. a second `rsync` +That script covers Postgres only. Uploaded chat images, file/video +attachments, avatars, and custom emoji all live on the **app** server's +disk (`/srv/ds-chat/uploads`, created in §3a) — a separate machine from +the data server this script runs on — and currently have no backup +mechanism at all. Whatever off-box destination you pick above, include +`/srv/ds-chat/uploads` in it too (e.g. a second `rsync` line run from the app server). **Test a restore** (against a scratch database, never directly onto @@ -389,13 +390,13 @@ Carried forward from earlier phases (see `backend/README.md`'s own "Notes / scope decisions" for the full detail on each): - No rate limiting on human or bot API traffic. - No CSRF token (relies on `SameSite=Lax` cookies). -- No server-side session revocation (signed cookies only). - SSRF protection on outgoing webhooks is creation-time only, not re-validated per delivery (DNS-rebinding gap). - Backup off-box shipping is a placeholder — decide a destination and fill in `deploy/backup-postgres.sh`. -- Uploaded chat images and file attachments (`/srv/ds-chat/uploads` on the - app server) have no backup coverage at all yet, on-box or off — see §7. +- Uploaded chat images, file/video attachments, and custom emoji + (`/srv/ds-chat/uploads` on the app server) have no backup coverage at + all yet, on-box or off — see §7. - Uploaded-but-never-sent images or files (a user attaches one, then never hits Send) leak an orphaned file on disk — no cleanup job for this yet. Not a security issue (still gated by room membership to view), just an diff --git a/README.md b/README.md index bf0db63..110a771 100644 --- a/README.md +++ b/README.md @@ -18,21 +18,26 @@ build plan. ## Features -- **Auth & accounts** — session-based auth, invite-only signup (admin-issued - site invites or room invites, both delivered by email), password reset, - per-user light/dark/midnight/sunset presets plus a live theme builder for - fully custom, named, savable color themes. -- **Rooms** — open and private rooms, owner/admin/member roles, invites, - room browsing/search, file/image galleries per room. +- **Auth & accounts** — server-side, revocable sessions (see every device + you're logged in from and sign one out remotely), invite-only signup + (admin-issued site invites or room invites, both delivered by email), + password reset, per-user light/dark/midnight/sunset presets plus a live + theme builder for fully custom, named, savable color themes. +- **Rooms & direct messages** — open and private rooms, owner/admin/member + roles, invites (with resend), room browsing/search, file/image galleries + per room, plus 1:1 direct messages with a collapsible sidebar section. - **Real-time chat** — WebSocket-based messaging with automatic reconnect - and backoff, Markdown rendering, @mentions with autocomplete and inline - highlighting, emoji reactions, message editing, image and file - attachments (drag-and-drop, paste, or picker) with inline previews for - images/PDFs/text/Markdown, unread indicators, and presence (online/away/ + and backoff, Markdown rendering (headings with custom anchors, sub/ + superscript, tables, and more), @mentions and #room-reference links with + autocomplete and inline highlighting, emoji reactions and shortcodes plus + site-wide custom/uploaded emoji, message editing and deletion, image/ + video/file attachments (drag-and-drop, paste, or picker) with inline + previews and playback, unread indicators, and presence (online/away/ offline, with a manual "appear offline" override). -- **Notifications** — Web Push for offline/backgrounded members, with - per-type opt-in/out (mentions vs. all messages), plus in-app unread - badges. +- **Notifications** — Web Push and native desktop notifications for + offline/backgrounded members, plus email: always-on for direct messages, + opt-in per room (first unread message and every mention), all alongside + in-app unread badges. - **PWA** — installable, offline-capable (cached room/message data, a dedicated offline banner), with automatic update detection that prompts a reload as soon as a new deploy goes live. diff --git a/USER_GUIDE.md b/USER_GUIDE.md index 6180a26..3fb66aa 100644 --- a/USER_GUIDE.md +++ b/USER_GUIDE.md @@ -7,15 +7,17 @@ attachments, notifications, and personalizing your account. - [Getting started](#getting-started) - [Rooms](#rooms) +- [Direct messages](#direct-messages) - [Sending messages](#sending-messages) - [Formatting](#formatting) - [Mentions and room links](#mentions-and-room-links) - [Attachments](#attachments) -- [Reactions](#reactions) -- [Editing a message](#editing-a-message) +- [Reactions and custom emoji](#reactions-and-custom-emoji) +- [Editing and deleting a message](#editing-and-deleting-a-message) - [Presence and notifications](#presence-and-notifications) - [Your profile](#your-profile) - [Room details](#room-details) +- [Active sessions](#active-sessions) - [Staying up to date](#staying-up-to-date) ## Getting started @@ -45,6 +47,21 @@ search box at the top to filter by name. - **Unread indicators**: a room with new activity shows a dot next to its name — a plain dot for unread messages, a highlighted dot if you were specifically @mentioned. +- **Collapsible sections**: the **Direct Messages** and **Rooms** headers in + the sidebar can be collapsed to hide their contents — click the header to + toggle. Your choice is remembered. + +## Direct messages + +Click **People** to see everyone on the site and start a 1:1 conversation +with someone — it opens (or reopens, if you've messaged them before) +under **Direct Messages** in the sidebar. There's only ever one +conversation per pair of people, however many times you start it. + +You can hide a conversation you're done with (its own menu, or from its +**Room details** panel) without deleting anything — it drops out of your +sidebar but reappears automatically the moment the other person sends a +new message, or if you message them again yourself. ## Sending messages @@ -61,7 +78,9 @@ Messages support Markdown: - `-` or `1.` for bulleted/numbered lists - `[link text](https://example.com)` — or just paste a bare URL and it becomes clickable automatically -- `#`, `##`, `###` for headings +- `#`, `##`, `###` for headings — add `{#custom-id}` at the end of a + heading line to control its link anchor instead of the auto-generated one +- `~sub~` and `^sup^` for subscript and superscript Pasting a link on its own often also generates a preview card underneath your message, pulled from that page's title/description/image, when the @@ -69,7 +88,9 @@ page provides one. Emoji: click the 🙂 button in the composer to open the emoji picker, or type a shortcode like `:tada:` and it's converted automatically once you -send. The picker remembers your recently-used emoji and has a search box. +send. The picker remembers your recently-used emoji and has a search box — +see [Reactions and custom emoji](#reactions-and-custom-emoji) for uploading +your own. ## Mentions and room links @@ -87,26 +108,39 @@ can see the message (and is a member of that room) straight to it. Click the paperclip icon to attach a file, or just drag a file onto the message box and drop it. Images show as an inline thumbnail — click one -to view it full-size. Other files show as a small card with the -filename and size; `.txt`, `.md`, and `.pdf` files open in a preview -without leaving the room, everything else downloads when clicked. +to view it full-size. Common video formats (MP4, WebM, Ogg) play inline +too, with a button to expand to a larger view; other files show as a +small card with the filename and size — `.txt`, `.md`, and `.pdf` files +open in a preview without leaving the room, everything else downloads +when clicked. There's a server-configured maximum file size — if a file is too large, you'll see an error before it uploads. -## Reactions +## Reactions and custom emoji Hover over a message and click the 🙂 icon in its action row to react with an emoji. Reactions from everyone appear as small pills under the message with a count; click an existing pill to add or remove your own reaction to it. Hovering a pill shows who reacted. -## Editing a message +Anyone can add a custom emoji: open the emoji picker (the 🙂 button, either +in the composer or on a message) and click **+ Add** in the **Custom** +section. Give it a short name and an image — it's then usable by everyone, +both as a reaction and inline in message text via `:your-name:`, right +alongside the built-in picker. You can remove a custom emoji you uploaded +(or any of them, if you're a site admin) from the same picker. + +## Editing and deleting a message You can edit any message you sent: hover it and click **Edit**, make your changes, then press **Enter** to save or **Escape** to cancel. Clicking away also saves. Edited messages are marked *(edited)*. +To delete a message you sent, hover it and click **Delete**. It's replaced +with a "message deleted" placeholder rather than disappearing outright, so +the conversation doesn't visibly shift for anyone else reading it. + ## Presence and notifications Everyone's avatar shows a small status dot — green for online, grey for @@ -124,6 +158,12 @@ The same menu has a notifications toggle: get a notification whenever the app is minimized *or* simply not the focused window, even if it's still open somewhere on screen. +You'll also get an email if someone messages you in a direct conversation +while you're genuinely offline — no setup needed. For regular rooms, +email is opt-in per room: open a room's **Room details** panel and turn +on **Email notifications** to get emailed on that room's first unread +message and on every `@mention`, while you're offline. + ## Your profile Open your account menu and choose **Profile settings** to: @@ -135,12 +175,22 @@ Open your account menu and choose **Profile settings** to: live) - Change your password +## Active sessions + +Profile settings also lists **Active sessions** — every device/browser +currently logged into your account, with its approximate location (IP +address) and when it was last active. If you see one you don't +recognize, click **Revoke** to sign it out immediately. Revoking your own +current device signs you out too. + ## Room details Click the info icon in a room's header to open its details panel, where you can: - See who else is in the room and their role (member/admin/owner) +- Turn on **Email notifications** for that room (see + [Presence and notifications](#presence-and-notifications)) - Browse and re-download every file and image ever shared in the room, without scrolling back through history - Leave the room — unless you're the owner, in which case ownership has diff --git a/backend/README.md b/backend/README.md index 3e14322..f97a7ab 100644 --- a/backend/README.md +++ b/backend/README.md @@ -1,18 +1,22 @@ -# DS Chat backend (Phase 1 + 2 + 4 + 5 + 6 + 7 + 8, image uploads, file attachments, admin-configurable upload size limits, emoji & reactions, user profiles, site invites & email, password reset) +# DS Chat backend (Phase 1 + 2 + 4 + 5 + 6 + 7 + 8, image uploads, file attachments, admin-configurable upload size limits, emoji & reactions, user profiles, site invites & email, password reset, direct messages, active sessions) -FastAPI + SQLAlchemy 2.0 (async) + PostgreSQL + Redis. Implements auth, room -CRUD (open and private), room roles (owner/admin/member) and direct -membership management, a WebSocket chat endpoint that fans out across -multiple app-server instances via Redis pub/sub, Web Push notifications for -offline room members, a site-admin portal (user/room/bot management + an -audit log), a bot/extension layer (scoped API tokens, live bot WebSocket -access, incoming and outgoing webhooks, message editing), image uploads and -generic file attachments in chat messages, emoji reactions on messages, -self-service user profiles -(display name, avatar), self-service password change and a token-based -forgot-password flow, and admin-issued email invites for new accounts -plus email notifications when a user is added to a room. See -`../ARCHITECTURE.md` for the full system design and the phased build plan. +FastAPI + SQLAlchemy 2.0 (async) + PostgreSQL + Redis. Implements auth +(server-side, revocable sessions — see Active sessions below), room CRUD +(open and private), room roles (owner/admin/member) and direct membership +management, direct messages, a WebSocket chat endpoint that fans out across +multiple app-server instances via Redis pub/sub, Web Push and email +notifications for offline room members (plus a native desktop-notification +bridge for DS Chat Desktop), a site-admin portal (user/room/bot management + +an audit log), a bot/extension layer (scoped API tokens, live bot WebSocket +access, incoming and outgoing webhooks, message editing), image uploads, +inline-playable video attachments, and generic file attachments in chat +messages, message deletion, emoji reactions (built-in Unicode plus +site-wide custom/uploaded emoji, both usable in reactions and inline in +message text), self-service user profiles (display name, avatar), +self-service password change and a token-based forgot-password flow, and +admin-issued email invites for new accounts plus email notifications when a +user is added to a room. See `../ARCHITECTURE.md` for the full system +design and the phased build plan. This is an **invite-only site**: there is no public registration endpoint. Accounts are created by an operator on the app server — see step 4 below. @@ -129,10 +133,14 @@ app/ on-disk save/read -- see Image uploads below cli.py `python -m app.cli create-user` / `generate-vapid-keys` models/ SQLAlchemy models (users, rooms, room_memberships, - messages, message_images, message_reactions, + messages, message_images, message_files, + message_reactions, message_mentions, + message_room_references, link_previews, + custom_emoji, custom_themes, sessions, site_invites, password_resets, smtp_settings, - push_subscriptions, admin_audit_log, api_tokens, - webhooks_incoming, event_subscriptions) + upload_settings, push_subscriptions, + admin_audit_log, api_tokens, webhooks_incoming, + event_subscriptions) schemas/ Pydantic request/response models routers/ auth, rooms, users, signup, push, admin, bots, webhooks, health @@ -345,6 +353,36 @@ No `User`/`PushSubscription` schema change was needed for this feature — the only backend change is the new `desktop_notification` envelope type, covered by `backend/tests/test_desktop_notifications.py`. +## Email notifications for missed messages + +Two related, separately-scoped email triggers layered on top of Web Push/ +desktop notifications, both in `app/services/message_events.py`: + +**DMs** (`_maybe_email_dm_notification`) — always on, no opt-in toggle. +Emails a DM's other participant when they're *genuinely* offline +(`GlobalPresence.is_online`, not just "not connected to this room's own +channel" the way the Web Push/desktop-notification audience is computed — +someone actively using the app in a different room shouldn't get emailed +for a DM) or have `appear_offline` set. Debounced to the first unread +message in the conversation, not one email per message in a burst, by +checking whether any other unread message already exists in the room +since the recipient's `last_read_at`. + +**Regular rooms** (`_maybe_email_room_notifications`) — opt-in per +member, per room (`RoomMembership.email_notifications`, toggled via `PATCH +/api/rooms/{id}/notifications`; rejected for a DM — `CannotModifyDmError` +→ 400 — since DMs already get the always-on behavior above). Two triggers, +not one: the room's first unread message debounces the same way DMs do, +*but* a message that `@mentions` the subscriber always emails regardless +of that debounce — a mention is a stronger, individually-addressed signal +that shouldn't get silently absorbed by an earlier plain message in the +same burst already having used up the "first unread" email. + +Both paths go through the same `email_service.send_email` used by +invites/room-membership notifications above, so they're silently skipped +if SMTP isn't configured and never block message delivery on an SMTP +outage. + ## Room roles and membership (Phase 2) Rooms can be `open` (anyone can join via `POST /api/rooms/{id}/join`) or @@ -365,6 +403,47 @@ since nothing meaningful was gained by making the target confirm first. `GET /api/rooms/mine` lists every room (open + private) the current user belongs to, alongside their role. +## Direct messages + +A DM is a `Room` with `is_dm=True` and a deterministic, never-shown +internal `name` (`dm_room_name(user_a, user_b)` — the two user ids sorted +and joined, so it's the same string regardless of who initiates), not a +separate model — `POST /api/rooms/dm` (`find_or_create_dm`) looks up an +existing DM by that name and creates one (`is_private=True`, both +participants as plain `member`) only if none exists yet, so starting a DM +with the same person twice always resolves to the one conversation. The +frontend never renders `Room.name` for a DM; `MyRoomItem.dm_partner` +(`DmPartnerInfo`: the *other* participant's id/username/display +name/avatar/status) is precomputed server-side instead, batched per +request rather than N+1. + +**Hiding a DM**: `RoomMembership.hidden_at` lets one participant remove a +DM from their own sidebar without touching the other participant's copy or +deleting anything — a DM has no sensible "leave" (it would violate +`find_or_create_dm`'s exactly-two-members assumption). `POST +/api/rooms/{id}/hide` sets it; it's cleared automatically (un-hiding the +DM) whenever a new message arrives in it or `find_or_create_dm` resolves +back to an already-hidden one — both count as the conversation being +active again, matching how a re-opened DM in Slack/Discord reappears on +its own rather than needing an explicit "unhide." + +## Message deletion + +`DELETE`-shaped over WS (`{"type": "delete", "room_id", "message_id"}`, +`message_service.delete_message`) — author-only (`NotMessageAuthorError` → +error frame otherwise, no admin/moderator override yet). A real delete of +content, not a UI-only hide: `content`, `image_id`, `file_id`, and +`preview_url` are all cleared and any attached `MessageImage`/`MessageFile` +row (plus its on-disk file) is actually removed, only `deleted_at` (and +`id`/`room_id`/`user_id`/`created_at`, so the tombstone still occupies its +place in history) survives. The attachment's storage filename is read and +the DB row/file only unlinked *after* a successful commit — same ordering +`delete_room` already uses, so a rolled-back transaction never leaves an +already-destroyed file with no way back. Broadcasts +`{"type": "message_deleted", "id", "room_id"}`; the frontend renders a +"message deleted" placeholder rather than removing the row, so the +conversation doesn't visibly shift when someone deletes something above. + ## Image uploads A message can carry an image (`Message.image_id`, nullable), a caption @@ -432,6 +511,18 @@ share them; `ImageTooLargeError` was likewise renamed to Same orphaned-upload disk-space caveat as images applies here too. +**Inline video playback**: a browser-natively-playable video attachment +(`INLINE_SAFE_VIDEO_CONTENT_TYPES` in `app/storage.py` — a strict allowlist, +`video/mp4`/`video/webm`/`video/ogg`, deliberately not "every `video/*` +type") is served *without* the `filename=` param above, so it plays inline +in a `