Private
Public Access
Add URL previews for chat messages (#43)
Slack/Discord-style link unfurling: the first http(s) URL in a message's content gets a small preview card (title/description/image/site name) fetched from the page's Open Graph tags. Backend: - Message.preview_url (extracted at create/edit time, cheap regex, no I/O) points at a link_previews cache row keyed by URL -- the same URL posted in different messages/rooms fetches once, and a failed fetch is cached too so a dead URL isn't retried on every reference. - The actual fetch runs in a background asyncio.create_task from broadcast_new_message/broadcast_message_update, on its own DB session, so a slow third-party site never delays message delivery. A separate "link_preview" WS envelope carries the result once it resolves. - SSRF protection reuses app/services/ssrf.py's validate_target_url (renamed from UnsafeWebhookUrlError to UnsafeUrlError now that it's shared with webhooks), but re-validates before every hop of a redirect chain rather than once up front -- redirects are followed manually so each intermediate URL is checked before it's ever connected to. - Parsed with stdlib html.parser -- no new dependency. Frontend: a LinkPreviewCard rendered under message content when present, patched into state live via the new WS envelope and included in message history for reloads. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
@@ -572,6 +572,52 @@ unguessable expiring token is the actual protection once a request is made),
|
|||||||
no cleanup job for expired/used `password_resets` rows (same as
|
no cleanup job for expired/used `password_resets` rows (same as
|
||||||
`site_invites`, which has never had one either).
|
`site_invites`, which has never had one either).
|
||||||
|
|
||||||
|
## Link previews
|
||||||
|
|
||||||
|
Slack/Discord-style unfurling: the first `http(s)://` URL found in a
|
||||||
|
message's `content` (`link_preview_service.extract_first_url`) gets a small
|
||||||
|
preview card fetched from that page's Open Graph tags (`og:title`,
|
||||||
|
`og:description`, `og:image`, `og:site_name`, falling back to `<title>`).
|
||||||
|
|
||||||
|
- **Never blocks the send.** `create_message` extracts and stores the URL
|
||||||
|
on `Message.preview_url` synchronously (cheap, no I/O), but the actual
|
||||||
|
fetch runs in a background `asyncio.create_task` from
|
||||||
|
`message_events.broadcast_new_message`/`broadcast_message_update`, on its
|
||||||
|
own DB session (`async_session_factory()`, never the caller's session —
|
||||||
|
see `push_service.send_push_to_user`'s docstring for why sharing a
|
||||||
|
session across a fire-and-forget task is unsafe). Once it resolves, a
|
||||||
|
separate `"link_preview"` WS envelope carries the result to the room;
|
||||||
|
the initial `"message"`/`"message_update"` broadcast always has
|
||||||
|
`link_preview: null`.
|
||||||
|
- **SSRF protection is the real security boundary here**, more so than for
|
||||||
|
outgoing webhooks — a webhook's `target_url` is admin-configured, but a
|
||||||
|
link-preview URL comes from *any* room member's message content. Reuses
|
||||||
|
`app/services/ssrf.py`'s `validate_target_url` (originally webhook-only,
|
||||||
|
the exception renamed from `UnsafeWebhookUrlError` to `UnsafeUrlError`
|
||||||
|
now that it's shared), but re-validates before **every hop** of a
|
||||||
|
redirect chain instead of once up front — redirects are followed
|
||||||
|
manually (`httpx.AsyncClient(follow_redirects=False)`) specifically so
|
||||||
|
each intermediate URL is checked before it's ever connected to, not
|
||||||
|
after. Same accepted DNS-rebinding gap as the webhook case (see that
|
||||||
|
module's docstring); response body capped at 512 KB and only fetched if
|
||||||
|
`Content-Type` is `text/html`.
|
||||||
|
- **Cached by URL, not by message** (`link_previews` table, unique on
|
||||||
|
`url`) — a URL posted by five different people in five different rooms
|
||||||
|
fetches once. A row also gets written on a *failed* fetch
|
||||||
|
(`fetch_failed=True`) so a URL that genuinely doesn't unfurl (SSRF
|
||||||
|
rejection, timeout, no usable title) isn't re-attempted on every message
|
||||||
|
that references it; both kinds expire after 7 days (`_CACHE_TTL`).
|
||||||
|
- Parsed with stdlib `html.parser.HTMLParser`, not a new dependency — only
|
||||||
|
meta-tag scraping is needed, not general HTML parsing.
|
||||||
|
- Editing a message re-extracts the URL; if it changed or was removed, the
|
||||||
|
frontend clears the now-stale preview immediately (`preview_url` on the
|
||||||
|
`message_update` envelope) rather than leaving the old one showing while
|
||||||
|
a new fetch (if any) is in flight.
|
||||||
|
- Scope cuts: one preview per message (the first URL only, matching the
|
||||||
|
issue's "a small preview" framing), no way to dismiss/suppress a preview
|
||||||
|
before sending, no re-fetch-on-demand if a cached preview goes stale
|
||||||
|
mid-TTL.
|
||||||
|
|
||||||
## Notes / scope decisions
|
## Notes / scope decisions
|
||||||
|
|
||||||
- Invite-only site registration: no `POST /api/auth/register`. Accounts are
|
- Invite-only site registration: no `POST /api/auth/register`. Accounts are
|
||||||
|
|||||||
@@ -0,0 +1,53 @@
|
|||||||
|
"""link previews for URL unfurling in chat messages
|
||||||
|
|
||||||
|
Revision ID: d00f93766fa5
|
||||||
|
Revises: ffdd409227bb
|
||||||
|
Create Date: 2026-08-17 17:31:13.598917
|
||||||
|
|
||||||
|
"""
|
||||||
|
from typing import Sequence, Union
|
||||||
|
|
||||||
|
from alembic import op
|
||||||
|
import sqlalchemy as sa
|
||||||
|
|
||||||
|
|
||||||
|
# revision identifiers, used by Alembic.
|
||||||
|
revision: str = 'd00f93766fa5'
|
||||||
|
down_revision: Union[str, Sequence[str], None] = 'ffdd409227bb'
|
||||||
|
branch_labels: Union[str, Sequence[str], None] = None
|
||||||
|
depends_on: Union[str, Sequence[str], None] = None
|
||||||
|
|
||||||
|
|
||||||
|
def upgrade() -> None:
|
||||||
|
"""Upgrade schema."""
|
||||||
|
op.create_table(
|
||||||
|
"link_previews",
|
||||||
|
sa.Column("id", sa.Uuid(), nullable=False),
|
||||||
|
sa.Column("url", sa.String(length=2048), nullable=False),
|
||||||
|
sa.Column("title", sa.String(length=500), nullable=True),
|
||||||
|
sa.Column("description", sa.String(length=1000), nullable=True),
|
||||||
|
sa.Column("image_url", sa.String(length=2048), nullable=True),
|
||||||
|
sa.Column("site_name", sa.String(length=200), nullable=True),
|
||||||
|
sa.Column("fetch_failed", sa.Boolean(), nullable=False),
|
||||||
|
sa.Column(
|
||||||
|
"fetched_at",
|
||||||
|
sa.DateTime(timezone=True),
|
||||||
|
server_default=sa.text("clock_timestamp()"),
|
||||||
|
nullable=False,
|
||||||
|
),
|
||||||
|
sa.PrimaryKeyConstraint("id"),
|
||||||
|
)
|
||||||
|
op.create_index(
|
||||||
|
op.f("ix_link_previews_url"), "link_previews", ["url"], unique=True
|
||||||
|
)
|
||||||
|
# Nullable, no default -- just adds a column to the catalog, no table
|
||||||
|
# rewrite, no lock beyond the instant one ALTER TABLE ADD COLUMN always
|
||||||
|
# takes for a nullable column with no default.
|
||||||
|
op.add_column("messages", sa.Column("preview_url", sa.String(length=2048), nullable=True))
|
||||||
|
|
||||||
|
|
||||||
|
def downgrade() -> None:
|
||||||
|
"""Downgrade schema."""
|
||||||
|
op.drop_column("messages", "preview_url")
|
||||||
|
op.drop_index(op.f("ix_link_previews_url"), table_name="link_previews")
|
||||||
|
op.drop_table("link_previews")
|
||||||
@@ -4,6 +4,7 @@ from app.models.base import Base
|
|||||||
from app.models.custom_theme import CustomTheme
|
from app.models.custom_theme import CustomTheme
|
||||||
from app.models.event_subscription import EventSubscription
|
from app.models.event_subscription import EventSubscription
|
||||||
from app.models.invite import InviteStatus
|
from app.models.invite import InviteStatus
|
||||||
|
from app.models.link_preview import LinkPreview
|
||||||
from app.models.membership import RoomMembership, RoomRole
|
from app.models.membership import RoomMembership, RoomRole
|
||||||
from app.models.message import Message
|
from app.models.message import Message
|
||||||
from app.models.message_file import MessageFile
|
from app.models.message_file import MessageFile
|
||||||
@@ -41,4 +42,5 @@ __all__ = [
|
|||||||
"WebhookIncoming",
|
"WebhookIncoming",
|
||||||
"EventSubscription",
|
"EventSubscription",
|
||||||
"CustomTheme",
|
"CustomTheme",
|
||||||
|
"LinkPreview",
|
||||||
]
|
]
|
||||||
|
|||||||
@@ -0,0 +1,32 @@
|
|||||||
|
import uuid
|
||||||
|
from datetime import datetime
|
||||||
|
|
||||||
|
from sqlalchemy import Boolean, DateTime, String, func
|
||||||
|
from sqlalchemy.orm import Mapped, mapped_column
|
||||||
|
|
||||||
|
from app.models.base import Base
|
||||||
|
|
||||||
|
|
||||||
|
class LinkPreview(Base):
|
||||||
|
"""A cache keyed by URL, not by message -- multiple messages (in the
|
||||||
|
same room or different ones) linking the same URL share one fetch
|
||||||
|
instead of each triggering their own. link_preview_service is the only
|
||||||
|
writer; Message.preview_url (the first URL found in a message's
|
||||||
|
content, set at creation time) is the join key a caller looks this up
|
||||||
|
by."""
|
||||||
|
|
||||||
|
__tablename__ = "link_previews"
|
||||||
|
|
||||||
|
id: Mapped[uuid.UUID] = mapped_column(primary_key=True, default=uuid.uuid4)
|
||||||
|
url: Mapped[str] = mapped_column(String(2048), unique=True, index=True, nullable=False)
|
||||||
|
title: Mapped[str | None] = mapped_column(String(500))
|
||||||
|
description: Mapped[str | None] = mapped_column(String(1000))
|
||||||
|
image_url: Mapped[str | None] = mapped_column(String(2048))
|
||||||
|
site_name: Mapped[str | None] = mapped_column(String(200))
|
||||||
|
# Cached separately from a "no row yet" state so a URL that genuinely
|
||||||
|
# doesn't unfurl (no title, fetch error, blocked by SSRF checks) isn't
|
||||||
|
# re-fetched on every message that references it within the TTL.
|
||||||
|
fetch_failed: Mapped[bool] = mapped_column(Boolean, default=False, nullable=False)
|
||||||
|
fetched_at: Mapped[datetime] = mapped_column(
|
||||||
|
DateTime(timezone=True), server_default=func.clock_timestamp(), nullable=False
|
||||||
|
)
|
||||||
@@ -1,7 +1,7 @@
|
|||||||
import uuid
|
import uuid
|
||||||
from datetime import datetime
|
from datetime import datetime
|
||||||
|
|
||||||
from sqlalchemy import CheckConstraint, DateTime, ForeignKey, Text, func
|
from sqlalchemy import CheckConstraint, DateTime, ForeignKey, String, Text, func
|
||||||
from sqlalchemy.orm import Mapped, mapped_column, relationship
|
from sqlalchemy.orm import Mapped, mapped_column, relationship
|
||||||
|
|
||||||
from app.models.base import Base
|
from app.models.base import Base
|
||||||
@@ -25,6 +25,12 @@ class Message(Base):
|
|||||||
content: Mapped[str | None] = mapped_column(Text)
|
content: Mapped[str | None] = mapped_column(Text)
|
||||||
image_id: Mapped[uuid.UUID | None] = mapped_column(ForeignKey("message_images.id"))
|
image_id: Mapped[uuid.UUID | None] = mapped_column(ForeignKey("message_images.id"))
|
||||||
file_id: Mapped[uuid.UUID | None] = mapped_column(ForeignKey("message_files.id"))
|
file_id: Mapped[uuid.UUID | None] = mapped_column(ForeignKey("message_files.id"))
|
||||||
|
# First http(s) URL found in content at creation time (see
|
||||||
|
# link_preview_service.extract_first_url), if any -- a plain string, not
|
||||||
|
# a FK, since the actual preview data lives in link_previews cached by
|
||||||
|
# URL and is looked up separately (see message_service.list_recent_
|
||||||
|
# messages), not eager-loaded as an ORM relationship.
|
||||||
|
preview_url: Mapped[str | None] = mapped_column(String(2048))
|
||||||
# clock_timestamp(), not now()/func.now() -- the WS handler (ws/chat.py)
|
# clock_timestamp(), not now()/func.now() -- the WS handler (ws/chat.py)
|
||||||
# shares one AsyncSession for a whole connection's lifetime, and a
|
# shares one AsyncSession for a whole connection's lifetime, and a
|
||||||
# read-only op (e.g. a "join" frame's membership check) can leave a
|
# read-only op (e.g. a "join" frame's membership check) can leave a
|
||||||
|
|||||||
@@ -13,7 +13,7 @@ from app.dependencies import (
|
|||||||
require_scope,
|
require_scope,
|
||||||
)
|
)
|
||||||
from app.models import MessageFile, MessageImage, RoomRole, User
|
from app.models import MessageFile, MessageImage, RoomRole, User
|
||||||
from app.schemas.message import MessageFileInfo, MessageRead
|
from app.schemas.message import LinkPreviewInfo, MessageFileInfo, MessageRead
|
||||||
from app.schemas.message_file import MessageFileCreated
|
from app.schemas.message_file import MessageFileCreated
|
||||||
from app.schemas.message_image import MessageImageCreated
|
from app.schemas.message_image import MessageImageCreated
|
||||||
from app.schemas.room import (
|
from app.schemas.room import (
|
||||||
@@ -35,6 +35,7 @@ from app.schemas.webhook import (
|
|||||||
WebhookIncomingCreate,
|
WebhookIncomingCreate,
|
||||||
WebhookIncomingRead,
|
WebhookIncomingRead,
|
||||||
)
|
)
|
||||||
|
from app.services.link_preview_service import get_link_previews_for_urls
|
||||||
from app.services.message_events import broadcast_room_added
|
from app.services.message_events import broadcast_room_added
|
||||||
from app.services.message_service import (
|
from app.services.message_service import (
|
||||||
get_reactions_for_messages,
|
get_reactions_for_messages,
|
||||||
@@ -78,7 +79,7 @@ from app.services.webhook_service import (
|
|||||||
revoke_event_subscription,
|
revoke_event_subscription,
|
||||||
revoke_incoming_webhook,
|
revoke_incoming_webhook,
|
||||||
)
|
)
|
||||||
from app.services.ssrf import UnsafeWebhookUrlError
|
from app.services.ssrf import UnsafeUrlError
|
||||||
from app.storage import (
|
from app.storage import (
|
||||||
ALLOWED_IMAGE_CONTENT_TYPES,
|
ALLOWED_IMAGE_CONTENT_TYPES,
|
||||||
UPLOADS_DIR,
|
UPLOADS_DIR,
|
||||||
@@ -341,6 +342,8 @@ async def get_room_messages_endpoint(
|
|||||||
await require_room_member(room_id, current_user, db)
|
await require_room_member(room_id, current_user, db)
|
||||||
messages = await list_recent_messages(db, room_id, limit)
|
messages = await list_recent_messages(db, room_id, limit)
|
||||||
reactions_by_message = await get_reactions_for_messages(db, [m.id for m in messages])
|
reactions_by_message = await get_reactions_for_messages(db, [m.id for m in messages])
|
||||||
|
preview_urls = {m.preview_url for m in messages if m.preview_url}
|
||||||
|
previews_by_url = await get_link_previews_for_urls(db, list(preview_urls))
|
||||||
return [
|
return [
|
||||||
MessageRead(
|
MessageRead(
|
||||||
id=m.id,
|
id=m.id,
|
||||||
@@ -350,6 +353,11 @@ async def get_room_messages_endpoint(
|
|||||||
content=m.content,
|
content=m.content,
|
||||||
image_id=m.image_id,
|
image_id=m.image_id,
|
||||||
file=_to_message_file_info(m.file) if m.file else None,
|
file=_to_message_file_info(m.file) if m.file else None,
|
||||||
|
link_preview=(
|
||||||
|
LinkPreviewInfo.model_validate(previews_by_url[m.preview_url])
|
||||||
|
if m.preview_url and m.preview_url in previews_by_url
|
||||||
|
else None
|
||||||
|
),
|
||||||
reactions=reactions_by_message.get(m.id, []),
|
reactions=reactions_by_message.get(m.id, []),
|
||||||
created_at=m.created_at,
|
created_at=m.created_at,
|
||||||
edited_at=m.edited_at,
|
edited_at=m.edited_at,
|
||||||
@@ -607,7 +615,7 @@ async def create_event_subscription_endpoint(
|
|||||||
)
|
)
|
||||||
except InvalidEventTypeError:
|
except InvalidEventTypeError:
|
||||||
raise HTTPException(status_code=400, detail="Unrecognized event type")
|
raise HTTPException(status_code=400, detail="Unrecognized event type")
|
||||||
except UnsafeWebhookUrlError:
|
except UnsafeUrlError:
|
||||||
raise HTTPException(
|
raise HTTPException(
|
||||||
status_code=400, detail="target_url is not allowed (internal/private address)"
|
status_code=400, detail="target_url is not allowed (internal/private address)"
|
||||||
)
|
)
|
||||||
|
|||||||
@@ -19,6 +19,16 @@ class MessageFileInfo(BaseModel):
|
|||||||
content_type: str
|
content_type: str
|
||||||
|
|
||||||
|
|
||||||
|
class LinkPreviewInfo(BaseModel):
|
||||||
|
model_config = ConfigDict(from_attributes=True)
|
||||||
|
|
||||||
|
url: str
|
||||||
|
title: str | None
|
||||||
|
description: str | None
|
||||||
|
image_url: str | None
|
||||||
|
site_name: str | None
|
||||||
|
|
||||||
|
|
||||||
class MessageRead(BaseModel):
|
class MessageRead(BaseModel):
|
||||||
model_config = ConfigDict(from_attributes=True)
|
model_config = ConfigDict(from_attributes=True)
|
||||||
|
|
||||||
@@ -29,6 +39,7 @@ class MessageRead(BaseModel):
|
|||||||
content: str | None
|
content: str | None
|
||||||
image_id: uuid.UUID | None
|
image_id: uuid.UUID | None
|
||||||
file: MessageFileInfo | None
|
file: MessageFileInfo | None
|
||||||
|
link_preview: LinkPreviewInfo | None
|
||||||
reactions: list[ReactionSummary]
|
reactions: list[ReactionSummary]
|
||||||
created_at: datetime
|
created_at: datetime
|
||||||
edited_at: datetime | None
|
edited_at: datetime | None
|
||||||
|
|||||||
@@ -0,0 +1,207 @@
|
|||||||
|
import logging
|
||||||
|
import re
|
||||||
|
import uuid
|
||||||
|
from datetime import datetime, timedelta, timezone
|
||||||
|
from html.parser import HTMLParser
|
||||||
|
|
||||||
|
import httpx
|
||||||
|
from sqlalchemy import select
|
||||||
|
from sqlalchemy.ext.asyncio import AsyncSession
|
||||||
|
|
||||||
|
from app.database import async_session_factory
|
||||||
|
from app.models import LinkPreview
|
||||||
|
from app.services.ssrf import UnsafeUrlError, validate_target_url
|
||||||
|
from app.ws.broadcaster import Broadcaster
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
# Deliberately not a full URL regex (e.g. no IPv6-literal-host support) --
|
||||||
|
# just enough to find "a URL-shaped thing" in a chat message the same way
|
||||||
|
# the frontend's Markdown renderer autolinks bare URLs. Trailing punctuation
|
||||||
|
# a sentence would naturally have after a URL (a period, closing paren from
|
||||||
|
# "(see https://example.com)", etc.) is trimmed off separately below.
|
||||||
|
_URL_RE = re.compile(r"https?://[^\s<>\"]+")
|
||||||
|
_TRAILING_PUNCTUATION = ".,;:!?)'\">"
|
||||||
|
|
||||||
|
_FETCH_TIMEOUT_SECONDS = 5.0
|
||||||
|
_MAX_BYTES = 512 * 1024
|
||||||
|
_MAX_REDIRECTS = 3
|
||||||
|
_USER_AGENT = "ds-chat-link-preview/1.0"
|
||||||
|
_CACHE_TTL = timedelta(days=7)
|
||||||
|
|
||||||
|
|
||||||
|
def extract_first_url(content: str | None) -> str | None:
|
||||||
|
if not content:
|
||||||
|
return None
|
||||||
|
match = _URL_RE.search(content)
|
||||||
|
if not match:
|
||||||
|
return None
|
||||||
|
return match.group(0).rstrip(_TRAILING_PUNCTUATION) or None
|
||||||
|
|
||||||
|
|
||||||
|
class _OpenGraphParser(HTMLParser):
|
||||||
|
"""Pulls og:title/og:description/og:image/og:site_name meta tags,
|
||||||
|
falling back to <title> -- stdlib html.parser is enough for meta-tag
|
||||||
|
scraping, no reason to add a full HTML parsing dependency (e.g.
|
||||||
|
BeautifulSoup) just for this."""
|
||||||
|
|
||||||
|
def __init__(self) -> None:
|
||||||
|
super().__init__(convert_charrefs=True)
|
||||||
|
self.og: dict[str, str] = {}
|
||||||
|
self.title: str | None = None
|
||||||
|
self._in_title = False
|
||||||
|
|
||||||
|
def handle_starttag(self, tag: str, attrs: list[tuple[str, str | None]]) -> None:
|
||||||
|
if tag == "meta":
|
||||||
|
attr_dict = dict(attrs)
|
||||||
|
prop = attr_dict.get("property") or attr_dict.get("name")
|
||||||
|
content = attr_dict.get("content")
|
||||||
|
if prop and content and prop.startswith("og:"):
|
||||||
|
self.og.setdefault(prop, content)
|
||||||
|
elif tag == "title":
|
||||||
|
self._in_title = True
|
||||||
|
|
||||||
|
def handle_endtag(self, tag: str) -> None:
|
||||||
|
if tag == "title":
|
||||||
|
self._in_title = False
|
||||||
|
|
||||||
|
def handle_data(self, data: str) -> None:
|
||||||
|
if self._in_title and self.title is None:
|
||||||
|
self.title = data.strip()
|
||||||
|
|
||||||
|
|
||||||
|
async def _fetch_preview_data(url: str) -> dict | None:
|
||||||
|
"""SSRF-safe fetch: validates (scheme + resolved-IP allowlist check,
|
||||||
|
see ssrf.py) before *every* hop, following redirects manually rather
|
||||||
|
than via httpx's own follow_redirects -- that would connect to each
|
||||||
|
intermediate hop before any of them got validated, defeating the point.
|
||||||
|
Still subject to the DNS-rebinding gap documented in ssrf.py (a window
|
||||||
|
between validating a hostname and httpx independently resolving it to
|
||||||
|
connect); accepted for the same reason it's accepted there.
|
||||||
|
"""
|
||||||
|
current_url = url
|
||||||
|
async with httpx.AsyncClient(timeout=_FETCH_TIMEOUT_SECONDS, follow_redirects=False) as client:
|
||||||
|
for _ in range(_MAX_REDIRECTS + 1):
|
||||||
|
try:
|
||||||
|
validate_target_url(current_url)
|
||||||
|
except UnsafeUrlError:
|
||||||
|
return None
|
||||||
|
|
||||||
|
try:
|
||||||
|
async with client.stream(
|
||||||
|
"GET", current_url, headers={"User-Agent": _USER_AGENT}
|
||||||
|
) as response:
|
||||||
|
if response.is_redirect:
|
||||||
|
location = response.headers.get("location")
|
||||||
|
if not location:
|
||||||
|
return None
|
||||||
|
current_url = str(httpx.URL(current_url).join(location))
|
||||||
|
continue
|
||||||
|
if response.status_code >= 400:
|
||||||
|
return None
|
||||||
|
content_type = response.headers.get("content-type", "")
|
||||||
|
if "text/html" not in content_type:
|
||||||
|
return None
|
||||||
|
body = b""
|
||||||
|
async for chunk in response.aiter_bytes():
|
||||||
|
body += chunk
|
||||||
|
if len(body) >= _MAX_BYTES:
|
||||||
|
break
|
||||||
|
break
|
||||||
|
except httpx.HTTPError:
|
||||||
|
return None
|
||||||
|
else:
|
||||||
|
return None
|
||||||
|
|
||||||
|
parser = _OpenGraphParser()
|
||||||
|
try:
|
||||||
|
parser.feed(body.decode("utf-8", errors="replace"))
|
||||||
|
except Exception:
|
||||||
|
return None
|
||||||
|
|
||||||
|
title = parser.og.get("og:title") or parser.title
|
||||||
|
if not title:
|
||||||
|
return None
|
||||||
|
|
||||||
|
return {
|
||||||
|
"title": title.strip()[:500],
|
||||||
|
"description": (parser.og.get("og:description") or "").strip()[:1000] or None,
|
||||||
|
"image_url": parser.og.get("og:image") or None,
|
||||||
|
"site_name": (parser.og.get("og:site_name") or "").strip()[:200] or None,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
async def _get_or_fetch(db: AsyncSession, url: str) -> LinkPreview | None:
|
||||||
|
"""Returns None only when there's genuinely nothing to show (a fresh
|
||||||
|
fetch failed, or a cached row says an earlier one did) -- callers don't
|
||||||
|
need to distinguish "still fetching" from "never going to have one"."""
|
||||||
|
existing = (await db.execute(select(LinkPreview).where(LinkPreview.url == url))).scalar_one_or_none()
|
||||||
|
|
||||||
|
if existing is not None and datetime.now(timezone.utc) - existing.fetched_at < _CACHE_TTL:
|
||||||
|
return None if existing.fetch_failed else existing
|
||||||
|
|
||||||
|
data = await _fetch_preview_data(url)
|
||||||
|
if existing is not None:
|
||||||
|
existing.fetch_failed = data is None
|
||||||
|
if data:
|
||||||
|
existing.title = data["title"]
|
||||||
|
existing.description = data["description"]
|
||||||
|
existing.image_url = data["image_url"]
|
||||||
|
existing.site_name = data["site_name"]
|
||||||
|
existing.fetched_at = datetime.now(timezone.utc)
|
||||||
|
await db.commit()
|
||||||
|
return None if data is None else existing
|
||||||
|
|
||||||
|
row = LinkPreview(
|
||||||
|
url=url,
|
||||||
|
fetch_failed=data is None,
|
||||||
|
**(data or {"title": None, "description": None, "image_url": None, "site_name": None}),
|
||||||
|
)
|
||||||
|
db.add(row)
|
||||||
|
await db.commit()
|
||||||
|
return None if data is None else row
|
||||||
|
|
||||||
|
|
||||||
|
async def fetch_and_broadcast_link_preview(
|
||||||
|
broadcaster: Broadcaster, room_id: uuid.UUID, message_id: uuid.UUID, url: str
|
||||||
|
) -> None:
|
||||||
|
"""Entry point for a fire-and-forget asyncio.create_task from
|
||||||
|
broadcast_new_message -- runs on its own DB session (see push_service.py's
|
||||||
|
send_push_to_user docstring for why a background task must never share
|
||||||
|
the caller's session) so a slow/hanging fetch can never delay message
|
||||||
|
delivery to anyone actually online.
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
async with async_session_factory() as db:
|
||||||
|
preview = await _get_or_fetch(db, url)
|
||||||
|
except Exception:
|
||||||
|
logger.warning("Link preview fetch failed for %s", url, exc_info=True)
|
||||||
|
return
|
||||||
|
|
||||||
|
if preview is None:
|
||||||
|
return
|
||||||
|
|
||||||
|
await broadcaster.publish(
|
||||||
|
room_id,
|
||||||
|
{
|
||||||
|
"type": "link_preview",
|
||||||
|
"room_id": str(room_id),
|
||||||
|
"id": str(message_id),
|
||||||
|
"url": preview.url,
|
||||||
|
"title": preview.title,
|
||||||
|
"description": preview.description,
|
||||||
|
"image_url": preview.image_url,
|
||||||
|
"site_name": preview.site_name,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
async def get_link_previews_for_urls(db: AsyncSession, urls: list[str]) -> dict[str, LinkPreview]:
|
||||||
|
"""Batch lookup for message history -- mirrors message_service.
|
||||||
|
get_reactions_for_messages's shape (one query, zipped back onto results
|
||||||
|
by the caller) rather than an ORM relationship, since the join key is a
|
||||||
|
plain string column, not a FK."""
|
||||||
|
if not urls:
|
||||||
|
return {}
|
||||||
|
result = await db.execute(select(LinkPreview).where(LinkPreview.url.in_(urls), LinkPreview.fetch_failed.is_(False)))
|
||||||
|
return {row.url: row for row in result.scalars().all()}
|
||||||
@@ -1,3 +1,4 @@
|
|||||||
|
import asyncio
|
||||||
import uuid
|
import uuid
|
||||||
|
|
||||||
from sqlalchemy import select
|
from sqlalchemy import select
|
||||||
@@ -5,6 +6,7 @@ from sqlalchemy.ext.asyncio import AsyncSession
|
|||||||
|
|
||||||
from app.models import Message, MessageFile, MessageMention, Room, RoomMembership, User
|
from app.models import Message, MessageFile, MessageMention, Room, RoomMembership, User
|
||||||
from app.schemas.message import ReactionSummary
|
from app.schemas.message import ReactionSummary
|
||||||
|
from app.services.link_preview_service import fetch_and_broadcast_link_preview
|
||||||
from app.services.push_service import send_push_to_user
|
from app.services.push_service import send_push_to_user
|
||||||
from app.services.webhook_service import dispatch_event
|
from app.services.webhook_service import dispatch_event
|
||||||
from app.ws.broadcaster import Broadcaster
|
from app.ws.broadcaster import Broadcaster
|
||||||
@@ -90,12 +92,24 @@ async def _message_payload(db: AsyncSession, message: Message, username: str) ->
|
|||||||
"content": message.content,
|
"content": message.content,
|
||||||
"image_id": str(message.image_id) if message.image_id else None,
|
"image_id": str(message.image_id) if message.image_id else None,
|
||||||
"file": file_payload,
|
"file": file_payload,
|
||||||
|
# Never populated here -- fetching it is a network call to a
|
||||||
|
# third-party URL, which has no business delaying message delivery.
|
||||||
|
# A separate "link_preview" envelope arrives shortly after (see
|
||||||
|
# _maybe_fetch_link_preview) once/if the fetch succeeds.
|
||||||
|
"link_preview": None,
|
||||||
"reactions": [],
|
"reactions": [],
|
||||||
"created_at": message.created_at.isoformat(),
|
"created_at": message.created_at.isoformat(),
|
||||||
"edited_at": message.edited_at.isoformat() if message.edited_at else None,
|
"edited_at": message.edited_at.isoformat() if message.edited_at else None,
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _maybe_fetch_link_preview(broadcaster: Broadcaster, room_id: uuid.UUID, message: Message) -> None:
|
||||||
|
if message.preview_url:
|
||||||
|
asyncio.create_task(
|
||||||
|
fetch_and_broadcast_link_preview(broadcaster, room_id, message.id, message.preview_url)
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
async def broadcast_new_message(
|
async def broadcast_new_message(
|
||||||
db: AsyncSession,
|
db: AsyncSession,
|
||||||
broadcaster: Broadcaster,
|
broadcaster: Broadcaster,
|
||||||
@@ -111,6 +125,7 @@ async def broadcast_new_message(
|
|||||||
await broadcaster.publish(room_id, payload)
|
await broadcaster.publish(room_id, payload)
|
||||||
await _notify_offline_members(db, broadcaster, presence, room_id, sender, message)
|
await _notify_offline_members(db, broadcaster, presence, room_id, sender, message)
|
||||||
await dispatch_event(db, "message.created", room_id, payload)
|
await dispatch_event(db, "message.created", room_id, payload)
|
||||||
|
_maybe_fetch_link_preview(broadcaster, room_id, message)
|
||||||
|
|
||||||
|
|
||||||
async def broadcast_message_update(
|
async def broadcast_message_update(
|
||||||
@@ -122,9 +137,15 @@ async def broadcast_message_update(
|
|||||||
"room_id": str(room_id),
|
"room_id": str(room_id),
|
||||||
"content": message.content,
|
"content": message.content,
|
||||||
"edited_at": message.edited_at.isoformat() if message.edited_at else None,
|
"edited_at": message.edited_at.isoformat() if message.edited_at else None,
|
||||||
|
# Lets the frontend clear a stale preview when an edit changes or
|
||||||
|
# removes the URL it came from -- it compares this against the
|
||||||
|
# link_preview it already has for the message rather than blindly
|
||||||
|
# keeping whatever was there before the edit.
|
||||||
|
"preview_url": message.preview_url,
|
||||||
}
|
}
|
||||||
await broadcaster.publish(room_id, payload)
|
await broadcaster.publish(room_id, payload)
|
||||||
await dispatch_event(db, "message.updated", room_id, payload)
|
await dispatch_event(db, "message.updated", room_id, payload)
|
||||||
|
_maybe_fetch_link_preview(broadcaster, room_id, message)
|
||||||
|
|
||||||
|
|
||||||
async def broadcast_reaction_update(
|
async def broadcast_reaction_update(
|
||||||
|
|||||||
@@ -8,6 +8,7 @@ from sqlalchemy.orm import selectinload
|
|||||||
|
|
||||||
from app.models import Message, MessageMention, MessageReaction
|
from app.models import Message, MessageMention, MessageReaction
|
||||||
from app.schemas.message import ReactionSummary
|
from app.schemas.message import ReactionSummary
|
||||||
|
from app.services.link_preview_service import extract_first_url
|
||||||
from app.services.mention_service import extract_mentioned_user_ids
|
from app.services.mention_service import extract_mentioned_user_ids
|
||||||
|
|
||||||
|
|
||||||
@@ -28,7 +29,12 @@ async def create_message(
|
|||||||
file_id: uuid.UUID | None = None,
|
file_id: uuid.UUID | None = None,
|
||||||
) -> Message:
|
) -> Message:
|
||||||
message = Message(
|
message = Message(
|
||||||
room_id=room_id, user_id=user_id, content=content, image_id=image_id, file_id=file_id
|
room_id=room_id,
|
||||||
|
user_id=user_id,
|
||||||
|
content=content,
|
||||||
|
image_id=image_id,
|
||||||
|
file_id=file_id,
|
||||||
|
preview_url=extract_first_url(content),
|
||||||
)
|
)
|
||||||
db.add(message)
|
db.add(message)
|
||||||
# message.id is available immediately (a Python-side uuid4 default, not
|
# message.id is available immediately (a Python-side uuid4 default, not
|
||||||
@@ -52,6 +58,7 @@ async def edit_message(
|
|||||||
raise NotMessageAuthorError()
|
raise NotMessageAuthorError()
|
||||||
|
|
||||||
message.content = content
|
message.content = content
|
||||||
|
message.preview_url = extract_first_url(content)
|
||||||
message.edited_at = datetime.now(timezone.utc)
|
message.edited_at = datetime.now(timezone.utc)
|
||||||
await db.commit()
|
await db.commit()
|
||||||
await db.refresh(message)
|
await db.refresh(message)
|
||||||
|
|||||||
@@ -3,27 +3,30 @@ import socket
|
|||||||
from urllib.parse import urlparse
|
from urllib.parse import urlparse
|
||||||
|
|
||||||
|
|
||||||
class UnsafeWebhookUrlError(Exception):
|
class UnsafeUrlError(Exception):
|
||||||
pass
|
pass
|
||||||
|
|
||||||
|
|
||||||
def validate_target_url(url: str) -> None:
|
def validate_target_url(url: str) -> None:
|
||||||
"""Creation-time-only SSRF check: rejects non-http(s) schemes and any
|
"""One-shot SSRF check: rejects non-http(s) schemes and any target whose
|
||||||
target whose hostname resolves to a private/loopback/link-local/
|
hostname resolves to a private/loopback/link-local/reserved/multicast
|
||||||
reserved/multicast address. Not re-checked per delivery, so this doesn't
|
address. Shared by two callers with different re-check needs: webhook
|
||||||
defend against DNS rebinding between creation and a later send -- a
|
subscriptions validate once at creation time and reuse the URL for many
|
||||||
documented known limitation, not an oversight.
|
future deliveries (a real but accepted DNS-rebinding gap, documented
|
||||||
|
here), while link_preview_service calls this fresh before *every* hop of
|
||||||
|
a redirect chain for a one-shot fetch, which closes that gap for its own
|
||||||
|
use case.
|
||||||
"""
|
"""
|
||||||
parsed = urlparse(url)
|
parsed = urlparse(url)
|
||||||
if parsed.scheme not in ("http", "https"):
|
if parsed.scheme not in ("http", "https"):
|
||||||
raise UnsafeWebhookUrlError()
|
raise UnsafeUrlError()
|
||||||
if not parsed.hostname:
|
if not parsed.hostname:
|
||||||
raise UnsafeWebhookUrlError()
|
raise UnsafeUrlError()
|
||||||
|
|
||||||
try:
|
try:
|
||||||
addrinfo = socket.getaddrinfo(parsed.hostname, None)
|
addrinfo = socket.getaddrinfo(parsed.hostname, None)
|
||||||
except socket.gaierror as exc:
|
except socket.gaierror as exc:
|
||||||
raise UnsafeWebhookUrlError() from exc
|
raise UnsafeUrlError() from exc
|
||||||
|
|
||||||
for *_rest, sockaddr in addrinfo:
|
for *_rest, sockaddr in addrinfo:
|
||||||
ip = ipaddress.ip_address(sockaddr[0])
|
ip = ipaddress.ip_address(sockaddr[0])
|
||||||
@@ -35,4 +38,4 @@ def validate_target_url(url: str) -> None:
|
|||||||
or ip.is_multicast
|
or ip.is_multicast
|
||||||
or ip.is_unspecified
|
or ip.is_unspecified
|
||||||
):
|
):
|
||||||
raise UnsafeWebhookUrlError()
|
raise UnsafeUrlError()
|
||||||
|
|||||||
@@ -0,0 +1,266 @@
|
|||||||
|
import asyncio
|
||||||
|
import uuid
|
||||||
|
|
||||||
|
import pytest_asyncio
|
||||||
|
|
||||||
|
from app.database import engine as _link_preview_engine
|
||||||
|
from app.schemas.user import UserCreate
|
||||||
|
from app.services.auth_service import register_user
|
||||||
|
from app.services.link_preview_service import extract_first_url
|
||||||
|
from tests.conftest import register_and_login
|
||||||
|
|
||||||
|
|
||||||
|
@pytest_asyncio.fixture(autouse=True)
|
||||||
|
async def _dispose_engine_between_tests():
|
||||||
|
# link_preview_service's background task uses app.database's module-
|
||||||
|
# level engine directly (see test_cli.py's identical fixture for why:
|
||||||
|
# pytest-asyncio's per-test event loops make a stale pooled connection
|
||||||
|
# from an earlier test's loop fail with asyncpg "another operation is
|
||||||
|
# in progress" -- silently, in this file's case, since
|
||||||
|
# fetch_and_broadcast_link_preview catches and logs rather than raises).
|
||||||
|
yield
|
||||||
|
await _link_preview_engine.dispose()
|
||||||
|
|
||||||
|
|
||||||
|
def _unique(prefix: str) -> str:
|
||||||
|
return f"{prefix}-{uuid.uuid4().hex[:8]}"
|
||||||
|
|
||||||
|
|
||||||
|
def _unique_url() -> str:
|
||||||
|
# link_previews rows are written via the background task's own,
|
||||||
|
# separately-committed DB session (see link_preview_service.py), not
|
||||||
|
# the db_session fixture's rollback-at-teardown transaction -- they're
|
||||||
|
# real, permanent rows that persist across test runs. A literal URL
|
||||||
|
# shared between tests (or repeated suite runs) would silently hit an
|
||||||
|
# earlier run's cached row instead of exercising a fresh fetch.
|
||||||
|
return f"http://8.8.8.8/{uuid.uuid4().hex}"
|
||||||
|
|
||||||
|
|
||||||
|
def _recv(ws) -> dict:
|
||||||
|
"""Reads the next frame, discarding member_updated presence-change
|
||||||
|
broadcasts -- same convention as test_message_edit.py."""
|
||||||
|
while True:
|
||||||
|
msg = ws.receive_json()
|
||||||
|
if msg.get("type") != "member_updated":
|
||||||
|
return msg
|
||||||
|
|
||||||
|
|
||||||
|
def _register_ws(ws_client, username: str) -> dict:
|
||||||
|
async def _seed():
|
||||||
|
async with ws_client.session_factory() as session:
|
||||||
|
await register_user(
|
||||||
|
session,
|
||||||
|
UserCreate(username=username, email=f"{username}@example.com", password="password123"),
|
||||||
|
)
|
||||||
|
|
||||||
|
ws_client.portal.call(_seed)
|
||||||
|
resp = ws_client.post(
|
||||||
|
"/api/auth/login", json={"username_or_email": username, "password": "password123"}
|
||||||
|
)
|
||||||
|
assert resp.status_code == 200, resp.text
|
||||||
|
return resp.json()
|
||||||
|
|
||||||
|
|
||||||
|
class _FakeResponse:
|
||||||
|
def __init__(self, status_code=200, headers=None, body=b""):
|
||||||
|
self.status_code = status_code
|
||||||
|
self.headers = headers or {}
|
||||||
|
self.is_redirect = status_code in (301, 302, 303, 307, 308)
|
||||||
|
self._body = body
|
||||||
|
|
||||||
|
async def aiter_bytes(self):
|
||||||
|
yield self._body
|
||||||
|
|
||||||
|
|
||||||
|
class _FakeStreamCtx:
|
||||||
|
def __init__(self, response):
|
||||||
|
self._response = response
|
||||||
|
|
||||||
|
async def __aenter__(self):
|
||||||
|
return self._response
|
||||||
|
|
||||||
|
async def __aexit__(self, *args):
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
_OG_HTML = (
|
||||||
|
b"<html><head>"
|
||||||
|
b'<meta property="og:title" content="Example Article">'
|
||||||
|
b'<meta property="og:description" content="A description of the article.">'
|
||||||
|
b'<meta property="og:image" content="https://8.8.8.8/image.png">'
|
||||||
|
b'<meta property="og:site_name" content="Example">'
|
||||||
|
b"</head></html>"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _fake_client_factory(html=_OG_HTML, status_code=200, call_log=None):
|
||||||
|
class _FakeAsyncClient:
|
||||||
|
def __init__(self, *args, **kwargs):
|
||||||
|
pass
|
||||||
|
|
||||||
|
async def __aenter__(self):
|
||||||
|
return self
|
||||||
|
|
||||||
|
async def __aexit__(self, *args):
|
||||||
|
return False
|
||||||
|
|
||||||
|
def stream(self, method, url, headers=None):
|
||||||
|
if call_log is not None:
|
||||||
|
call_log.append(url)
|
||||||
|
return _FakeStreamCtx(_FakeResponse(status_code, {"content-type": "text/html"}, html))
|
||||||
|
|
||||||
|
return _FakeAsyncClient
|
||||||
|
|
||||||
|
|
||||||
|
def test_extract_first_url_finds_url_in_content():
|
||||||
|
assert extract_first_url("check out https://example.com for more") == "https://example.com"
|
||||||
|
|
||||||
|
|
||||||
|
def test_extract_first_url_strips_trailing_punctuation():
|
||||||
|
assert extract_first_url("see (https://example.com/page).") == "https://example.com/page"
|
||||||
|
|
||||||
|
|
||||||
|
def test_extract_first_url_returns_none_without_url():
|
||||||
|
assert extract_first_url("no links here") is None
|
||||||
|
|
||||||
|
|
||||||
|
def test_extract_first_url_returns_none_for_empty_content():
|
||||||
|
assert extract_first_url(None) is None
|
||||||
|
assert extract_first_url("") is None
|
||||||
|
|
||||||
|
|
||||||
|
def test_ws_message_with_url_triggers_link_preview_broadcast(ws_client, monkeypatch):
|
||||||
|
monkeypatch.setattr("app.services.link_preview_service.httpx.AsyncClient", _fake_client_factory())
|
||||||
|
url = _unique_url()
|
||||||
|
|
||||||
|
username = _unique("alice")
|
||||||
|
_register_ws(ws_client, username=username)
|
||||||
|
room = ws_client.post("/api/rooms", json={"name": _unique("general")}).json()
|
||||||
|
|
||||||
|
with ws_client.websocket_connect("/ws/chat") as ws:
|
||||||
|
ws.send_json({"type": "join", "room_id": room["id"]})
|
||||||
|
assert _recv(ws)["type"] == "joined"
|
||||||
|
|
||||||
|
ws.send_json({"type": "message", "room_id": room["id"], "content": f"check this out {url}"})
|
||||||
|
message = _recv(ws)
|
||||||
|
assert message["type"] == "message"
|
||||||
|
# Never populated on the initial broadcast -- fetching is async.
|
||||||
|
assert message["link_preview"] is None
|
||||||
|
|
||||||
|
preview = _recv(ws)
|
||||||
|
assert preview["type"] == "link_preview"
|
||||||
|
assert preview["id"] == message["id"]
|
||||||
|
assert preview["url"] == url
|
||||||
|
assert preview["title"] == "Example Article"
|
||||||
|
assert preview["description"] == "A description of the article."
|
||||||
|
assert preview["image_url"] == "https://8.8.8.8/image.png"
|
||||||
|
assert preview["site_name"] == "Example"
|
||||||
|
|
||||||
|
|
||||||
|
def test_ws_message_with_private_url_gets_no_preview(ws_client, monkeypatch):
|
||||||
|
calls: list[str] = []
|
||||||
|
monkeypatch.setattr(
|
||||||
|
"app.services.link_preview_service.httpx.AsyncClient",
|
||||||
|
_fake_client_factory(call_log=calls),
|
||||||
|
)
|
||||||
|
|
||||||
|
username = _unique("alice")
|
||||||
|
_register_ws(ws_client, username=username)
|
||||||
|
room = ws_client.post("/api/rooms", json={"name": _unique("general")}).json()
|
||||||
|
|
||||||
|
with ws_client.websocket_connect("/ws/chat") as ws:
|
||||||
|
ws.send_json({"type": "join", "room_id": room["id"]})
|
||||||
|
assert _recv(ws)["type"] == "joined"
|
||||||
|
|
||||||
|
ws.send_json(
|
||||||
|
{
|
||||||
|
"type": "message",
|
||||||
|
"room_id": room["id"],
|
||||||
|
"content": "internal link http://127.0.0.1/secret",
|
||||||
|
}
|
||||||
|
)
|
||||||
|
message = _recv(ws)
|
||||||
|
assert message["link_preview"] is None
|
||||||
|
|
||||||
|
# Sync barrier: an idempotent second frame's ack proves the first
|
||||||
|
# message's entire async handling (including the spawned link-
|
||||||
|
# preview task, which is rejected before any real I/O) has settled
|
||||||
|
# without ever publishing a link_preview envelope.
|
||||||
|
ws.send_json({"type": "join", "room_id": room["id"]})
|
||||||
|
assert _recv(ws)["type"] == "joined"
|
||||||
|
|
||||||
|
assert calls == []
|
||||||
|
|
||||||
|
|
||||||
|
async def test_message_history_includes_cached_link_preview(client, db_session, monkeypatch):
|
||||||
|
captured_tasks: list[asyncio.Task] = []
|
||||||
|
real_create_task = asyncio.create_task
|
||||||
|
|
||||||
|
def fake_create_task(coro):
|
||||||
|
task = real_create_task(coro)
|
||||||
|
captured_tasks.append(task)
|
||||||
|
return task
|
||||||
|
|
||||||
|
monkeypatch.setattr("app.services.message_events.asyncio.create_task", fake_create_task)
|
||||||
|
monkeypatch.setattr("app.services.link_preview_service.httpx.AsyncClient", _fake_client_factory())
|
||||||
|
url = _unique_url()
|
||||||
|
|
||||||
|
await register_and_login(client, db_session, username="alice")
|
||||||
|
room = (await client.post("/api/rooms", json={"name": "general"})).json()
|
||||||
|
webhook = (await client.post(f"/api/rooms/{room['id']}/webhooks/incoming", json={})).json()
|
||||||
|
|
||||||
|
resp = await client.post(
|
||||||
|
f"/api/webhooks/incoming/{webhook['token']}",
|
||||||
|
json={"content": f"see {url}"},
|
||||||
|
)
|
||||||
|
assert resp.status_code == 204
|
||||||
|
await asyncio.gather(*captured_tasks)
|
||||||
|
|
||||||
|
history = (await client.get(f"/api/rooms/{room['id']}/messages")).json()
|
||||||
|
assert len(history) == 1
|
||||||
|
assert history[0]["link_preview"]["title"] == "Example Article"
|
||||||
|
assert history[0]["link_preview"]["url"] == url
|
||||||
|
|
||||||
|
|
||||||
|
async def test_link_preview_reused_across_messages_with_same_url(client, db_session, monkeypatch):
|
||||||
|
captured_tasks: list[asyncio.Task] = []
|
||||||
|
real_create_task = asyncio.create_task
|
||||||
|
|
||||||
|
def fake_create_task(coro):
|
||||||
|
task = real_create_task(coro)
|
||||||
|
captured_tasks.append(task)
|
||||||
|
return task
|
||||||
|
|
||||||
|
monkeypatch.setattr("app.services.message_events.asyncio.create_task", fake_create_task)
|
||||||
|
calls: list[str] = []
|
||||||
|
monkeypatch.setattr(
|
||||||
|
"app.services.link_preview_service.httpx.AsyncClient",
|
||||||
|
_fake_client_factory(call_log=calls),
|
||||||
|
)
|
||||||
|
url = _unique_url()
|
||||||
|
|
||||||
|
await register_and_login(client, db_session, username="alice")
|
||||||
|
room = (await client.post("/api/rooms", json={"name": "general"})).json()
|
||||||
|
webhook = (await client.post(f"/api/rooms/{room['id']}/webhooks/incoming", json={})).json()
|
||||||
|
|
||||||
|
# Awaited one at a time so the second send's cache check happens after
|
||||||
|
# the first's fetch has actually committed -- otherwise both could race
|
||||||
|
# past the "not cached yet" check concurrently and this would flake.
|
||||||
|
resp1 = await client.post(
|
||||||
|
f"/api/webhooks/incoming/{webhook['token']}", json={"content": f"see {url}"}
|
||||||
|
)
|
||||||
|
assert resp1.status_code == 204
|
||||||
|
await asyncio.gather(*captured_tasks)
|
||||||
|
captured_tasks.clear()
|
||||||
|
|
||||||
|
resp2 = await client.post(
|
||||||
|
f"/api/webhooks/incoming/{webhook['token']}", json={"content": f"again: {url}"}
|
||||||
|
)
|
||||||
|
assert resp2.status_code == 204
|
||||||
|
await asyncio.gather(*captured_tasks)
|
||||||
|
|
||||||
|
assert len(calls) == 1
|
||||||
|
|
||||||
|
history = (await client.get(f"/api/rooms/{room['id']}/messages")).json()
|
||||||
|
assert len(history) == 2
|
||||||
|
assert all(m["link_preview"]["title"] == "Example Article" for m in history)
|
||||||
@@ -10,22 +10,22 @@ from app.models import PushSubscription
|
|||||||
from app.schemas.user import UserCreate
|
from app.schemas.user import UserCreate
|
||||||
from app.services.auth_service import register_user
|
from app.services.auth_service import register_user
|
||||||
from app.services.room_service import join_room
|
from app.services.room_service import join_room
|
||||||
from app.services.ssrf import UnsafeWebhookUrlError, validate_target_url
|
from app.services.ssrf import UnsafeUrlError, validate_target_url
|
||||||
from tests.conftest import register_and_login
|
from tests.conftest import register_and_login
|
||||||
|
|
||||||
|
|
||||||
def test_validate_target_url_rejects_loopback():
|
def test_validate_target_url_rejects_loopback():
|
||||||
with pytest.raises(UnsafeWebhookUrlError):
|
with pytest.raises(UnsafeUrlError):
|
||||||
validate_target_url("http://127.0.0.1/hook")
|
validate_target_url("http://127.0.0.1/hook")
|
||||||
|
|
||||||
|
|
||||||
def test_validate_target_url_rejects_private_range():
|
def test_validate_target_url_rejects_private_range():
|
||||||
with pytest.raises(UnsafeWebhookUrlError):
|
with pytest.raises(UnsafeUrlError):
|
||||||
validate_target_url("http://10.0.0.5/hook")
|
validate_target_url("http://10.0.0.5/hook")
|
||||||
|
|
||||||
|
|
||||||
def test_validate_target_url_rejects_non_http_scheme():
|
def test_validate_target_url_rejects_non_http_scheme():
|
||||||
with pytest.raises(UnsafeWebhookUrlError):
|
with pytest.raises(UnsafeUrlError):
|
||||||
validate_target_url("ftp://8.8.8.8/hook")
|
validate_target_url("ftp://8.8.8.8/hook")
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -122,16 +122,45 @@ export function ChatPane({
|
|||||||
setLive((prev) => [...prev, envelope])
|
setLive((prev) => [...prev, envelope])
|
||||||
markRead()
|
markRead()
|
||||||
} else if (envelope.type === 'message_update' && envelope.room_id === room.id) {
|
} else if (envelope.type === 'message_update' && envelope.room_id === room.id) {
|
||||||
|
// link_preview only survives the edit if the URL it came from is
|
||||||
|
// still there -- an edit that changed or removed it clears the
|
||||||
|
// stale preview instead of leaving the old one showing. A new one
|
||||||
|
// (if the new URL has any) arrives via its own 'link_preview'
|
||||||
|
// envelope shortly after, same as a fresh send.
|
||||||
setHistory((prev) =>
|
setHistory((prev) =>
|
||||||
prev.map((m) =>
|
prev.map((m) =>
|
||||||
m.id === envelope.id ? { ...m, content: envelope.content, edited_at: envelope.edited_at } : m,
|
m.id === envelope.id
|
||||||
|
? {
|
||||||
|
...m,
|
||||||
|
content: envelope.content,
|
||||||
|
edited_at: envelope.edited_at,
|
||||||
|
link_preview: m.link_preview?.url === envelope.preview_url ? m.link_preview : null,
|
||||||
|
}
|
||||||
|
: m,
|
||||||
),
|
),
|
||||||
)
|
)
|
||||||
setLive((prev) =>
|
setLive((prev) =>
|
||||||
prev.map((m) =>
|
prev.map((m) =>
|
||||||
m.id === envelope.id ? { ...m, content: envelope.content, edited_at: envelope.edited_at } : m,
|
m.id === envelope.id
|
||||||
|
? {
|
||||||
|
...m,
|
||||||
|
content: envelope.content,
|
||||||
|
edited_at: envelope.edited_at,
|
||||||
|
link_preview: m.link_preview?.url === envelope.preview_url ? m.link_preview : null,
|
||||||
|
}
|
||||||
|
: m,
|
||||||
),
|
),
|
||||||
)
|
)
|
||||||
|
} else if (envelope.type === 'link_preview' && envelope.room_id === room.id) {
|
||||||
|
const linkPreview = {
|
||||||
|
url: envelope.url,
|
||||||
|
title: envelope.title,
|
||||||
|
description: envelope.description,
|
||||||
|
image_url: envelope.image_url,
|
||||||
|
site_name: envelope.site_name,
|
||||||
|
}
|
||||||
|
setHistory((prev) => prev.map((m) => (m.id === envelope.id ? { ...m, link_preview: linkPreview } : m)))
|
||||||
|
setLive((prev) => prev.map((m) => (m.id === envelope.id ? { ...m, link_preview: linkPreview } : m)))
|
||||||
} else if (envelope.type === 'reaction_update' && envelope.room_id === room.id) {
|
} else if (envelope.type === 'reaction_update' && envelope.room_id === room.id) {
|
||||||
setHistory((prev) =>
|
setHistory((prev) =>
|
||||||
prev.map((m) => (m.id === envelope.id ? { ...m, reactions: envelope.reactions } : m)),
|
prev.map((m) => (m.id === envelope.id ? { ...m, reactions: envelope.reactions } : m)),
|
||||||
|
|||||||
@@ -0,0 +1,65 @@
|
|||||||
|
.link-preview-card {
|
||||||
|
display: flex;
|
||||||
|
gap: 10px;
|
||||||
|
align-items: stretch;
|
||||||
|
background: var(--ds-surface-2);
|
||||||
|
border: 1px solid var(--ds-border);
|
||||||
|
border-left: 3px solid var(--ds-accent);
|
||||||
|
border-radius: var(--radius);
|
||||||
|
padding: 10px 12px;
|
||||||
|
margin-bottom: 4px;
|
||||||
|
max-width: min(420px, 100%);
|
||||||
|
text-decoration: none;
|
||||||
|
color: var(--ds-text);
|
||||||
|
}
|
||||||
|
|
||||||
|
.link-preview-card:hover {
|
||||||
|
border-color: var(--ds-accent);
|
||||||
|
border-left-color: var(--ds-accent);
|
||||||
|
}
|
||||||
|
|
||||||
|
.link-preview-image {
|
||||||
|
flex: none;
|
||||||
|
width: 64px;
|
||||||
|
height: 64px;
|
||||||
|
object-fit: cover;
|
||||||
|
border-radius: 6px;
|
||||||
|
background: var(--ds-void);
|
||||||
|
}
|
||||||
|
|
||||||
|
.link-preview-body {
|
||||||
|
display: flex;
|
||||||
|
flex-direction: column;
|
||||||
|
gap: 2px;
|
||||||
|
min-width: 0;
|
||||||
|
justify-content: center;
|
||||||
|
}
|
||||||
|
|
||||||
|
.link-preview-site {
|
||||||
|
font-size: 0.72rem;
|
||||||
|
font-weight: 700;
|
||||||
|
text-transform: uppercase;
|
||||||
|
letter-spacing: 0.03em;
|
||||||
|
color: var(--ds-muted);
|
||||||
|
}
|
||||||
|
|
||||||
|
.link-preview-title {
|
||||||
|
font-size: 0.86rem;
|
||||||
|
font-weight: 700;
|
||||||
|
color: var(--ds-accent-2, var(--ds-accent));
|
||||||
|
overflow: hidden;
|
||||||
|
text-overflow: ellipsis;
|
||||||
|
display: -webkit-box;
|
||||||
|
-webkit-line-clamp: 2;
|
||||||
|
-webkit-box-orient: vertical;
|
||||||
|
}
|
||||||
|
|
||||||
|
.link-preview-description {
|
||||||
|
font-size: 0.8rem;
|
||||||
|
color: var(--ds-muted);
|
||||||
|
overflow: hidden;
|
||||||
|
text-overflow: ellipsis;
|
||||||
|
display: -webkit-box;
|
||||||
|
-webkit-line-clamp: 2;
|
||||||
|
-webkit-box-orient: vertical;
|
||||||
|
}
|
||||||
@@ -0,0 +1,30 @@
|
|||||||
|
import type { LinkPreviewInfo } from '../types'
|
||||||
|
import './LinkPreviewCard.css'
|
||||||
|
|
||||||
|
interface LinkPreviewCardProps {
|
||||||
|
preview: LinkPreviewInfo
|
||||||
|
}
|
||||||
|
|
||||||
|
// Slack/Discord-style unfurl card, rendered under a message's text when the
|
||||||
|
// backend found a URL in it and successfully fetched Open Graph data for it
|
||||||
|
// (see link_preview_service.py -- title/description/image_url/site_name are
|
||||||
|
// all independently optional, since not every page sets every og: tag).
|
||||||
|
export function LinkPreviewCard({ preview }: LinkPreviewCardProps) {
|
||||||
|
return (
|
||||||
|
<a
|
||||||
|
href={preview.url}
|
||||||
|
target="_blank"
|
||||||
|
rel="noopener noreferrer"
|
||||||
|
className="link-preview-card"
|
||||||
|
>
|
||||||
|
{preview.image_url && (
|
||||||
|
<img src={preview.image_url} alt="" className="link-preview-image" loading="lazy" />
|
||||||
|
)}
|
||||||
|
<div className="link-preview-body">
|
||||||
|
{preview.site_name && <span className="link-preview-site">{preview.site_name}</span>}
|
||||||
|
{preview.title && <span className="link-preview-title">{preview.title}</span>}
|
||||||
|
{preview.description && <span className="link-preview-description">{preview.description}</span>}
|
||||||
|
</div>
|
||||||
|
</a>
|
||||||
|
)
|
||||||
|
}
|
||||||
@@ -7,6 +7,7 @@ import type { ChatMessageEnvelope, Message, MessageFileInfo, RoomMember } from '
|
|||||||
import { EMOJI_PICKER_MAX_HEIGHT, EmojiPicker } from './EmojiPicker'
|
import { EMOJI_PICKER_MAX_HEIGHT, EmojiPicker } from './EmojiPicker'
|
||||||
import { FilePreviewModal, getPreviewKind } from './FilePreviewModal'
|
import { FilePreviewModal, getPreviewKind } from './FilePreviewModal'
|
||||||
import { ImageLightbox } from './ImageLightbox'
|
import { ImageLightbox } from './ImageLightbox'
|
||||||
|
import { LinkPreviewCard } from './LinkPreviewCard'
|
||||||
import { MessageContent } from './MessageContent'
|
import { MessageContent } from './MessageContent'
|
||||||
import { UserAvatar } from './UserAvatar'
|
import { UserAvatar } from './UserAvatar'
|
||||||
import './MessageList.css'
|
import './MessageList.css'
|
||||||
@@ -171,6 +172,7 @@ export function MessageList({ roomId, messages, members, onEdit, onReact }: Mess
|
|||||||
{msg.edited_at && <span className="message-edited"> (edited)</span>}
|
{msg.edited_at && <span className="message-edited"> (edited)</span>}
|
||||||
</div>
|
</div>
|
||||||
)}
|
)}
|
||||||
|
{msg.link_preview && <LinkPreviewCard preview={msg.link_preview} />}
|
||||||
{msg.reactions.length > 0 && (
|
{msg.reactions.length > 0 && (
|
||||||
<div className="message-reaction-pills">
|
<div className="message-reaction-pills">
|
||||||
{msg.reactions.map((r) => {
|
{msg.reactions.map((r) => {
|
||||||
|
|||||||
@@ -108,6 +108,14 @@ export interface MessageFileInfo {
|
|||||||
content_type: string
|
content_type: string
|
||||||
}
|
}
|
||||||
|
|
||||||
|
export interface LinkPreviewInfo {
|
||||||
|
url: string
|
||||||
|
title: string | null
|
||||||
|
description: string | null
|
||||||
|
image_url: string | null
|
||||||
|
site_name: string | null
|
||||||
|
}
|
||||||
|
|
||||||
export interface Message {
|
export interface Message {
|
||||||
id: string
|
id: string
|
||||||
room_id: string
|
room_id: string
|
||||||
@@ -116,6 +124,7 @@ export interface Message {
|
|||||||
content: string | null
|
content: string | null
|
||||||
image_id: string | null
|
image_id: string | null
|
||||||
file: MessageFileInfo | null
|
file: MessageFileInfo | null
|
||||||
|
link_preview: LinkPreviewInfo | null
|
||||||
reactions: ReactionSummary[]
|
reactions: ReactionSummary[]
|
||||||
created_at: string
|
created_at: string
|
||||||
edited_at: string | null
|
edited_at: string | null
|
||||||
@@ -130,6 +139,7 @@ export interface ChatMessageEnvelope {
|
|||||||
content: string | null
|
content: string | null
|
||||||
image_id: string | null
|
image_id: string | null
|
||||||
file: MessageFileInfo | null
|
file: MessageFileInfo | null
|
||||||
|
link_preview: LinkPreviewInfo | null
|
||||||
reactions: ReactionSummary[]
|
reactions: ReactionSummary[]
|
||||||
created_at: string
|
created_at: string
|
||||||
edited_at: string | null
|
edited_at: string | null
|
||||||
@@ -141,6 +151,21 @@ export interface ChatMessageUpdateEnvelope {
|
|||||||
room_id: string
|
room_id: string
|
||||||
content: string
|
content: string
|
||||||
edited_at: string | null
|
edited_at: string | null
|
||||||
|
// Lets the frontend clear a stale preview when an edit changes/removes
|
||||||
|
// the URL it came from -- compare against whatever link_preview.url the
|
||||||
|
// message currently has rather than assuming it's still valid.
|
||||||
|
preview_url: string | null
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface ChatLinkPreviewEnvelope {
|
||||||
|
type: 'link_preview'
|
||||||
|
id: string
|
||||||
|
room_id: string
|
||||||
|
url: string
|
||||||
|
title: string | null
|
||||||
|
description: string | null
|
||||||
|
image_url: string | null
|
||||||
|
site_name: string | null
|
||||||
}
|
}
|
||||||
|
|
||||||
export interface ChatReactionUpdateEnvelope {
|
export interface ChatReactionUpdateEnvelope {
|
||||||
@@ -181,6 +206,7 @@ export type ServerEnvelope =
|
|||||||
| ChatMessageEnvelope
|
| ChatMessageEnvelope
|
||||||
| ChatMessageUpdateEnvelope
|
| ChatMessageUpdateEnvelope
|
||||||
| ChatReactionUpdateEnvelope
|
| ChatReactionUpdateEnvelope
|
||||||
|
| ChatLinkPreviewEnvelope
|
||||||
| ChatJoinedEnvelope
|
| ChatJoinedEnvelope
|
||||||
| ChatErrorEnvelope
|
| ChatErrorEnvelope
|
||||||
| ChatRoomAddedEnvelope
|
| ChatRoomAddedEnvelope
|
||||||
|
|||||||
Reference in New Issue
Block a user