Commit Graph
4 Commits
Author SHA1 Message Date
ksmithandClaude Sonnet 5 dc8b9ff83b Make the feed-poll interval site-wide instead of per-feed
Replaces Feed.fetch_interval_minutes with a single admin-editable
SiteSettings.fetch_interval_minutes (apps.core), following the same
singleton-row pattern as accounts.SetupLock. Per-feed error backoff is
unchanged — only the configured baseline interval is now shared across
every feed, loaded once per poll_due_feeds run rather than queried per
feed. EVECTION_MIN_FETCH_INTERVAL_MINUTES keeps working as the floor
underneath it.

SiteSettings is registered in Django admin as a singleton: "Add" is
only offered when no row exists yet, and delete is always refused.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-10-31 12:59:47 -06:00
ksmithandClaude Sonnet 5 64f25789f5 Isolate unexpected fetch failures and harden task durability (T2)
fetch_feed had no time limit and no handling for anything beyond the
network/parse errors fetch_and_ingest already catches internally — an
unexpected exception (a parser/sanitizer bug, a DB error) crashed the
task with no error recorded on the Feed, the claim left held for the
full CLAIM_STALE_MINUTES, and no visibility beyond a bare traceback.
fetch_feed now catches that, records it the same way a handled failure
would be (reusing fetcher.record_fetch_error, promoted from a private
helper), clears the claim, and re-raises so Celery still marks the task
FAILED. soft_time_limit/time_limit bound a single fetch that somehow
outruns FETCH_TIMEOUT_SECONDS + ssrf_guard's response cap during
parsing/ingestion.

Also: a worker process killed mid-task (OOM, deploy restart, crash)
previously lost that task silently — Celery's default is to ack a
message the moment a worker *starts* it. CELERY_TASK_ACKS_LATE +
CELERY_TASK_REJECT_ON_WORKER_LOST (config/settings/base.py) make the
message redeliverable instead, and CELERY_WORKER_PREFETCH_MULTIPLIER=1
keeps a worker from grabbing a batch of tasks it could lose all at once.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-10-31 12:46:29 -06:00
ksmithandClaude Sonnet 5 9439f9cc59 Add SSRF protection to feed fetching: validate scheme/address, bound redirects and response size (S1)
fetch_and_ingest fetched a feed_url supplied straight from the "add
feed" form or an uploaded OPML file, with requests following redirects
on its own and no check that the target (or a redirect's target) was
actually a public, fetchable feed rather than an internal service,
localhost, or a cloud metadata endpoint — reachable from wherever this
app runs, on any authenticated user's behalf.

apps.feeds.services.ssrf_guard.safe_get() replaces the direct
requests.get() call: it resolves and validates the hostname (rejecting
anything that isn't a globally routable address) before the initial
request and before following each redirect hop manually, caps the
number of hops, and bounds the response body size by streaming it
instead of buffering it unbounded. OPML-imported feeds go through the
same check the first time poll_due_feeds fetches them, since this is
the one chokepoint every feed URL passes through regardless of which
path created the Feed row.

Known residual gap, documented in ssrf_guard.py: this validates then
connects rather than pinning the TCP connection to the exact address it
validated, so a narrow DNS-rebinding race isn't closed. Fixing that
needs a custom transport and felt like more machinery than this
self-hosted app's threat model currently justifies.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-10-31 12:25:20 -06:00
ksmithandClaude Sonnet 5 d200ab80f7 Add core data models and the feed fetch pipeline
Implements Folder/Feed/Subscription (feeds app) and Article/
UserArticleState/Tag/ArticleTag (articles app), preserving the
shared-Feed/shared-Article dedup design: a feed is fetched once
regardless of subscriber count, and per-user read/star/save/hide state
is created lazily rather than eagerly per subscriber.

Adds the fetch pipeline: feedparser-based parsing with conditional GET
(ETag/Last-Modified), nh3 HTML sanitization, guid-based dedup with
content-hash change detection that never resets a user's read/star
state, and exponential backoff on fetch errors respecting a configured
minimum interval floor. Wired into Celery via poll_due_feeds/fetch_feed
tasks, plus a minimal "add feed by URL" view that fetches synchronously
so new subscribers see results immediately.

Verified live against a real feed (hnrss.org) for ingestion, 304
dedup, and error backoff, in addition to the unit/integration test
suite (28 tests, all passing).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-10-30 16:03:20 -06:00