An OPML upload was read fully into memory, parsed into a whole tree,
walked recursively, and (for large imports) embedded in a Celery
message. Nothing bounded raw size, ZIP entries, nesting depth, outline
count or feed count.
Uploads are capped at 5 MB (checked on the reported size before reading)
and ZIPs at 50 entries, with the decompressed member bounded by what it
actually yields rather than only its header. Parsing now streams with
iterparse and aborts the moment a limit is crossed — 24 levels deep,
20,000 outlines, 5,000 feeds — so a hostile file never builds a huge or
deep tree; a 50,000-level nesting bomb is rejected cleanly. Limit errors
are shown to the user. Large imports go to the worker by cache reference
(deleted once read, expires after an hour) instead of as the whole
document inside the broker message.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Adding a feed and the admin retry queued fetch_feed directly without
claiming the feed, so the periodic poller (which sees a never-fetched
or just-reset feed as due) could claim and fetch it too: duplicate
fetches, and one task overwriting another's claim/ETag/next-fetch state.
The retry path also queued the backoff reset and the fetch as two
independent tasks that could run in either order.
claim_feed/enqueue_fetch in apps.feeds.tasks are now the only way a
fetch gets dispatched — the poller, add-feed and retry all use them, so
a feed can only have one fetch in flight. If queueing fails the claim is
released (the poller recovers it), and add-feed still saves the
subscription during a broker outage. The retry resets backoff
synchronously before claiming, and says so when a fetch is already
running.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
add_feed ran fetch_and_ingest synchronously during the POST request, so
a slow or large feed tied up a Gunicorn worker for the duration of the
fetch (up to the 15s HTTP timeout, plus parse/ingest time) and a user
could repeat it with distinct URLs to reduce request capacity.
The subscription is now created synchronously as before, but the first
fetch is dispatched via fetch_feed.delay() — the same task the periodic
poll uses, so the save/error bookkeeping is unchanged. The user sees an
immediate redirect with a message that the fetch is in progress rather
than waiting on it in the request/response cycle.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
FreshRSS's Export feature produces a .zip (an .opml.xml plus a starred-
articles .json this app doesn't import) rather than a bare .opml file —
uploading that zip as-is failed with "That file doesn't look like a valid
OPML export" since defusedxml can't parse a zip as XML.
apps.feeds.opml.extract_opml_bytes detects a zip upload and pulls the
.opml/.opml.xml member out of it before parsing; a bare .opml upload is
returned unchanged, so existing behavior is untouched. Verified against
the reporter's actual FreshRSS export (18 feeds across 6 categories) both
directly through extract_opml_bytes/parse_opml and as a real upload
through the running app.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
UserArticleState.is_starred/is_saved_for_later/starred_at are gone —
starring and saving now add/remove an ArticleTag against a per-user
"Starred" / "Saved for later" Tag, created lazily on first use and marked
Tag.is_system=True. is_read/is_hidden stay on UserArticleState since
they're genuine reading state, not categorization, which is what made
these two candidates for unifying into the general tag mechanism.
System tags are excluded from the Manage Tags page and from delete_tag
(get_object_or_404 now requires is_system=False, so attempting to delete
one 404s) and from the subscription-tag autocomplete, since they
categorize articles, not feeds. The star/save icons are still exactly
where they were — this only changes what powers them.
apps.rules.engine's star action now does get_system_tag + ArticleTag.
get_or_create instead of setting a boolean field. A data migration
converts every existing is_starred=True/is_saved_for_later=True row into
its ArticleTag equivalent before the fields are dropped — verified by
seeding the pre-removal schema directly (including a row with both flags
set) and confirming the migration produces the right tags.
apps.articles.queries.with_user_state gained one more query (tag lookup
for the current page's articles) to stay in the same "fixed number of
queries regardless of article count" shape it already had.
Verified end to end against a running dev server: toggling star/saved,
the starred/saved filters, the sidebar's Favourites count, and that the
Starred tag doesn't appear on the Manage Tags page.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Folders are gone — organization is tags-only, via the SubscriptionTag
model added last commit. A migration converts every existing Folder into
a same-named Tag applied to its feeds, and every folder-scoped FilterRule
into an equivalent tag-scoped one, before the Folder table is dropped, so
upgrading an existing instance loses nothing (verified by seeding the
pre-removal schema directly and running the migration chain against it).
Unlike folders, a feed can carry more than one tag, so:
- apps.articles.tasks.apply_filter_rules_for_feed and apps.rules.tasks's
backlog re-apply now scope by "does this feed carry tag X" rather than
"does this feed belong to folder X".
- apps.feeds.opml: a feed's tags export as a comma-separated "category"
attribute (the standard OPML way to represent multiple categories)
instead of nesting under one folder outline. Import reads both that
attribute and nested category outlines, turning each enclosing level
into its own tag rather than collapsing to the nearest one.
- The sidebar groups feeds by tag instead of folder, so a feed with two
tags now appears in both groups; untagged feeds fall into "Untagged".
AddFeedForm gained a comma-separated tag_names field for parity with the
old add-time folder picker. EditSubscriptionForm dropped its folder
field entirely — tagging a subscription happens through the tag editor
already on that page.
Verified end to end: the migration chain from empty, the data migration
against a seeded pre-removal schema (folder/subscription/rule all convert
correctly), and the running app's sidebar grouping, OPML export category
attributes, and tag editor.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Subscription editing was a real gap: there was no way to rename a feed's
display title, move it between folders, or create a folder at all outside
OPML import. Added an edit page (apps/feeds/views.py:edit_subscription,
EditSubscriptionForm) reachable from the subscriptions list.
Added feed-level tagging via a new SubscriptionTag model (apps/articles/
models.py), mirroring the existing ArticleTag — tag a whole feed on its
edit page, and every one of its articles surfaces under that tag in the
stream alongside individually-tagged articles (apps.reader.views._apply_
scope now matches either). Renamed AddTagToArticleForm to AddTagForm
since it's shared by both now.
The stream also now remembers the last filter (unread/all/starred/saved)
in the session instead of always defaulting to "unread" when a sidebar
feed/folder/tag link omits ?filter= — sidebar links no longer force
?filter=all, letting the remembered choice apply.
Verified end to end against a running dev server: editing a subscription's
title/folder, creating a folder inline, tagging a feed and seeing its
articles appear under that tag, and the remembered-filter behavior across
requests with no ?filter= param.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Implements OPML parsing with defusedxml (untrusted upload, XXE risk)
and export with the stdlib's plain ElementTree (trusted, app-generated
output). Nesting deeper than one level flattens into its nearest
enclosing folder, matching the app's flat-folder model. Import reuses
existing Feed/Subscription rows via get_or_create, so re-importing the
same file is idempotent and cross-user Feed dedup is preserved.
Large imports (>50 entries) dispatch to a Celery task instead of
blocking the request; newly imported feeds aren't fetched immediately
since they have no last_fetched_at, so the next poll_due_feeds beat
cycle picks them up naturally.
Adds a Subscriptions page (replacing the standalone "Add feed" link in
nav) listing current subscriptions with unsubscribe, plus Import/
Export OPML actions.
Verified live: imported a 2-feed/1-folder OPML file against the
running dev server (via an authenticated session, since this browser
tool has no file-picker automation), confirmed the resulting
subscriptions and folder in the UI, exported and inspected the
resulting OPML structure, and unsubscribed from one feed.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Implements Folder/Feed/Subscription (feeds app) and Article/
UserArticleState/Tag/ArticleTag (articles app), preserving the
shared-Feed/shared-Article dedup design: a feed is fetched once
regardless of subscriber count, and per-user read/star/save/hide state
is created lazily rather than eagerly per subscriber.
Adds the fetch pipeline: feedparser-based parsing with conditional GET
(ETag/Last-Modified), nh3 HTML sanitization, guid-based dedup with
content-hash change detection that never resets a user's read/star
state, and exponential backoff on fetch errors respecting a configured
minimum interval floor. Wired into Celery via poll_due_feeds/fetch_feed
tasks, plus a minimal "add feed by URL" view that fetches synchronously
so new subscribers see results immediately.
Verified live against a real feed (hnrss.org) for ingestion, 304
dedup, and error backoff, in addition to the unit/integration test
suite (28 tests, all passing).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Sets up the Evection project structure: split settings supporting both
SQLite and PostgreSQL via DATABASE_URL, Celery+Redis wiring, and the
planned app layout (accounts, feeds, articles, rules, search, reader,
pwa, core). Implements multi-user auth with a FreshRSS-style first-run
setup wizard, login/logout, and admin-only user management, covered by
automated tests.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>