Commit Graph
14 Commits
Author SHA1 Message Date
ksmithandClaude Sonnet 5.5 37e658c49a Add configurable article retention (C7)
Articles and full HTML bodies accumulated forever, growing the
database, search index, and backups without bound. Site settings gets
a "Retention days" field (0, the default, keeps everything); a nightly
beat task deletes articles fetched longer ago than that in bounded
chunks, always keeping anything a user starred or saved for later.
Read state and tags cascade with the article and the FTS triggers drop
search entries. The dashboard shows the article count and oldest
article's age, and docs/install.md covers retention, the readiness
check, and SQLite's WAL tuning.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-11-01 08:51:20 -07:00
ksmithandClaude Sonnet 5.5 e5f1277014 Batch feed ingestion, add a readiness check, fail open without Redis (C2, C8)
Ingestion did a get_or_create per entry. It now prefetches existing
articles for the poll in one query, bulk-inserts new ones and
bulk-updates changed ones (falling back to row-by-row only if a
concurrent fetch wins a race), and dispatches rule evaluation in
bounded chunks so a huge first fetch is several small tasks.

/health/ stays a DB-only liveness check (upgrade.sh gates on it right
after restart). New /health/ready/ also checks Redis and that
poll_due_feeds ticked within five minutes — proving beat and a worker
are both alive — and returns 503 with per-check detail otherwise.

The sidebar unread-count cache and login throttle now degrade instead
of erroring when Redis is unreachable: every authenticated page and
every login would otherwise 500 during a cache outage. Found while
verifying in a sandbox with no Redis.

Also includes the dashboard's oldest-article metric (core/views.py).

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-11-01 08:51:20 -07:00
ksmithandClaude Sonnet 5 cb0ce5a336 Batch rule/tag lookups and index FilterRule for the ingest query (C9)
FilterRule had no indexes beyond the default primary/foreign-key ones,
and apply_filter_rules_for_feed issued 2 queries per subscriber (its
enabled rules, its subscription tags) inside a loop over every
subscriber of the feed — both cost grew with subscriber/rule count on
every single feed poll.

Adds a composite index on (user, is_enabled, priority), matching the
exact filter+order shape every caller already uses (this task,
apps.rules.tasks' backlog re-apply). apply_filter_rules_for_feed now
fetches rules and subscription tags once for the whole feed's subscriber
set, grouped by user/subscription in Python, instead of once per
subscriber.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-10-31 16:09:36 -06:00
ksmithandClaude Sonnet 5 abf4cef55b Cache sidebar unread counts for a short TTL (C3)
apps.core.context_processors.sidebar runs on every authenticated page
render and called unread_counts_by_feed, a join/group-by over the
user's whole article corpus, every single time — cost grew with the
archive's size regardless of which page was being viewed.

unread_counts_by_feed now caches its {feed_id: count} result per user
for 30 seconds (apps.core.cache_utils). The reader views that change
the current user's own read state (toggle_read, mark_all_read, viewing
an article) invalidate it immediately so the sidebar never visibly lags
behind the action that just caused it; other paths (rule-driven state
changes during background ingestion) aren't explicitly invalidated and
simply self-heal within the TTL, which is an acceptable tradeoff
already made elsewhere in this app (the stream's own "new articles"
banner polls on the same 30s cadence).

Also adds a global cache.clear() autouse fixture (conftest.py) — the
test CACHES backend (LocMemCache) lives in process memory and survives
Django's per-test DB rollback, so a cached count keyed by a user's pk
could otherwise leak into an unrelated later test reusing that pk.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-10-31 16:09:28 -06:00
ksmithandClaude Sonnet 5 91c0b1dd26 Centralize rule feed/tag scoping in the engine and fix backlog re-apply (T5)
reapply_rules_for_user_backlog narrowed its *backlog* to the triggering
rule's feed/tag scope, but then loaded every enabled rule for the user
and ran all of them against that backlog with no per-rule scope check
— so re-applying one feed- or tag-scoped rule could also fire an
unrelated rule scoped to a different feed/tag, if it happened to match.

rule_in_scope() moves the feed/tag scope check into the engine itself
(apply_filter_rules_for_feed already computed this per-subscription via
query filters; it now just passes the tag ids and lets the engine
decide), so backlog re-apply and ingest-time dispatch can't drift into
scoping rules differently again.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-10-31 12:19:56 -06:00
ksmithandClaude Sonnet 5 753ae9efc7 Make Starred and Saved-for-later pre-configured system tags
UserArticleState.is_starred/is_saved_for_later/starred_at are gone —
starring and saving now add/remove an ArticleTag against a per-user
"Starred" / "Saved for later" Tag, created lazily on first use and marked
Tag.is_system=True. is_read/is_hidden stay on UserArticleState since
they're genuine reading state, not categorization, which is what made
these two candidates for unifying into the general tag mechanism.

System tags are excluded from the Manage Tags page and from delete_tag
(get_object_or_404 now requires is_system=False, so attempting to delete
one 404s) and from the subscription-tag autocomplete, since they
categorize articles, not feeds. The star/save icons are still exactly
where they were — this only changes what powers them.

apps.rules.engine's star action now does get_system_tag + ArticleTag.
get_or_create instead of setting a boolean field. A data migration
converts every existing is_starred=True/is_saved_for_later=True row into
its ArticleTag equivalent before the fields are dropped — verified by
seeding the pre-removal schema directly (including a row with both flags
set) and confirming the migration produces the right tags.

apps.articles.queries.with_user_state gained one more query (tag lookup
for the current page's articles) to stay in the same "fixed number of
queries regardless of article count" shape it already had.

Verified end to end against a running dev server: toggling star/saved,
the starred/saved filters, the sidebar's Favourites count, and that the
Starred tag doesn't appear on the Manage Tags page.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-10-30 21:36:44 -06:00
ksmithandClaude Sonnet 5 0e132e9ab9 Remove folders; feeds are organized by tags now
Folders are gone — organization is tags-only, via the SubscriptionTag
model added last commit. A migration converts every existing Folder into
a same-named Tag applied to its feeds, and every folder-scoped FilterRule
into an equivalent tag-scoped one, before the Folder table is dropped, so
upgrading an existing instance loses nothing (verified by seeding the
pre-removal schema directly and running the migration chain against it).

Unlike folders, a feed can carry more than one tag, so:
- apps.articles.tasks.apply_filter_rules_for_feed and apps.rules.tasks's
  backlog re-apply now scope by "does this feed carry tag X" rather than
  "does this feed belong to folder X".
- apps.feeds.opml: a feed's tags export as a comma-separated "category"
  attribute (the standard OPML way to represent multiple categories)
  instead of nesting under one folder outline. Import reads both that
  attribute and nested category outlines, turning each enclosing level
  into its own tag rather than collapsing to the nearest one.
- The sidebar groups feeds by tag instead of folder, so a feed with two
  tags now appears in both groups; untagged feeds fall into "Untagged".

AddFeedForm gained a comma-separated tag_names field for parity with the
old add-time folder picker. EditSubscriptionForm dropped its folder
field entirely — tagging a subscription happens through the tag editor
already on that page.

Verified end to end: the migration chain from empty, the data migration
against a seeded pre-removal schema (folder/subscription/rule all convert
correctly), and the running app's sidebar grouping, OPML export category
attributes, and tag editor.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-10-30 21:30:06 -06:00
ksmithandClaude Sonnet 5 3b58fdcb7e Add subscription editing, feed-level tags, and a remembered stream filter
Subscription editing was a real gap: there was no way to rename a feed's
display title, move it between folders, or create a folder at all outside
OPML import. Added an edit page (apps/feeds/views.py:edit_subscription,
EditSubscriptionForm) reachable from the subscriptions list.

Added feed-level tagging via a new SubscriptionTag model (apps/articles/
models.py), mirroring the existing ArticleTag — tag a whole feed on its
edit page, and every one of its articles surfaces under that tag in the
stream alongside individually-tagged articles (apps.reader.views._apply_
scope now matches either). Renamed AddTagToArticleForm to AddTagForm
since it's shared by both now.

The stream also now remembers the last filter (unread/all/starred/saved)
in the session instead of always defaulting to "unread" when a sidebar
feed/folder/tag link omits ?filter= — sidebar links no longer force
?filter=all, letting the remembered choice apply.

Verified end to end against a running dev server: editing a subscription's
title/folder, creating a folder inline, tagging a feed and seeing its
articles appear under that tag, and the remembered-filter behavior across
requests with no ?filter= param.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-10-30 21:20:37 -06:00
ksmithandClaude Sonnet 5 cfa64b5485 Rebuild the reader UI to match FreshRSS's two-column layout
The previous UI was plain unstyled lists. Rebuilt around what FreshRSS
actually looks like (per reference screenshots): a persistent left sidebar
(folders/feeds with unread counts, favourites, tags) and a single main
column with a date-grouped article list, where clicking a title expands
the full article inline in place rather than navigating to a separate page.

- apps/core/context_processors.py: feeds the sidebar's folder/feed tree,
  tags, and starred count to every authenticated page in one shared query
  set, so base.html can render it without each view wiring it up.
- apps/reader/views.py: date-grouped stream (itertools.groupby over the
  already-ordered queryset), ?feed=/?folder= scoping alongside the existing
  ?tag=, a bulk mark-all-read endpoint (bulk_create/update, not one
  UserArticleState per article), and article_inline — the htmx endpoint
  behind the inline expand, sharing its mark-read logic with the full
  article_detail view via _load_article_and_mark_read.
- templates/reader/_article_inline.html returns the article content plus
  an out-of-band swap of _article_actions.html, so the row's read/star
  icons update immediately when an article is expanded and marked read.
- static/js/stream.js: the accordion toggle (one article open at a time),
  calling htmx.ajax() directly rather than hx-* attributes so a second
  click on an already-open row can collapse it without re-fetching.
- static/css/base.css: full FreshRSS-like dark theme (with a light
  variant under prefers-color-scheme) replacing the previous unstyled
  pass — sidebar, topbar search, row/icon states, inline-expand styling.

Verified end to end against a running dev server: folder/feed/tag scoping,
mark-all-read, the inline-expand mark-read side effect and its icon
update, and — against config.settings.prod specifically — that no inline
<script> crept back in anywhere (would silently break under the CSP added
in the previous commit) and the CSP header is still present on every page.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-10-30 21:11:17 -06:00
ksmithandClaude Sonnet 5 8653dbeaf9 Add full-text search with per-backend SQLite/Postgres implementations
Implements the backend-agnostic search interface from the plan,
selected at runtime by connection.vendor:

- SQLite: an FTS5 virtual table (external-content, mirroring
  Article.title/content_html) created via a vendor-conditional
  migration, kept in sync by insert/update/delete triggers. Queries
  wrap the whole user input as a single FTS5 phrase so punctuation in
  a search box can't break MATCH syntax. `reindex_search` management
  command rebuilds it after bulk imports.
- Postgres: computes the search vector on the fly per the plan's
  decision (title weighted above content, no stored/indexed column)
  rather than diverging the schema from SQLite.

Both backends scope results to the requesting user's own subscriptions
and share the stream/detail article-row template and read/star/save
actions via a new apps.articles.queries.with_user_state() helper,
extracted from apps.reader.views to avoid duplicating that annotation
logic for search results.

Verified live: the SQLite path end-to-end in-browser against the real
Hacker News subscription (search UI, phrase matching, no-match case),
and the Postgres path against a throwaway Postgres container (search,
feed-scoping, and title-vs-content ranking), torn down afterward — not
part of this project's own infrastructure.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-10-30 16:37:22 -06:00
ksmithandClaude Sonnet 5 5ed0499845 Add the filter/rules engine
Implements FilterRule (one condition -> action, scoped to a feed and/or
folder or global) with an evaluation engine applied at ingest time:
after the fetch pipeline commits new/updated articles, each affected
subscriber's enabled rules run in priority order against them, with
tag/star/mark_read actions stacking and hide short-circuiting further
evaluation for that article.

Adds rule management (create/edit/delete) and a "Re-apply" action that
re-evaluates a user's full ruleset against their unread backlog, scoped
by the triggering rule's own feed/folder — so a newly added rule can
retroactively affect already-ingested articles, not just future ones.

Verified live in-browser: created a "title contains 'Show HN' -> star"
rule scoped to the subscribed Hacker News feed, re-applied it against
the existing unread backlog, and confirmed both matching articles were
starred.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-10-30 16:31:33 -06:00
ksmithandClaude Sonnet 5 579696efc0 Add tags and tag-based stream filtering
Implements tag management (create/delete) and per-article tagging with
a freeform "add tag" input that creates the tag on first use instead
of requiring a separate create-then-assign flow. Tags render as pills
in the stream header; clicking one filters the stream to articles
carrying that tag, combinable with the existing read/starred/saved
filters.

The "Saved" filter already added in the reading-UI milestone functions
as the reading list, so this closes out the reading-list/tags
milestone without a separate dedicated view.

Verified live in-browser: creating a tag from an article, the pill
appearing in the stream nav, filtering by it, and deleting it from the
tag-management page, all via htmx without full page reloads.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-10-30 16:26:30 -06:00
ksmithandClaude Sonnet 5 d200ab80f7 Add core data models and the feed fetch pipeline
Implements Folder/Feed/Subscription (feeds app) and Article/
UserArticleState/Tag/ArticleTag (articles app), preserving the
shared-Feed/shared-Article dedup design: a feed is fetched once
regardless of subscriber count, and per-user read/star/save/hide state
is created lazily rather than eagerly per subscriber.

Adds the fetch pipeline: feedparser-based parsing with conditional GET
(ETag/Last-Modified), nh3 HTML sanitization, guid-based dedup with
content-hash change detection that never resets a user's read/star
state, and exponential backoff on fetch errors respecting a configured
minimum interval floor. Wired into Celery via poll_due_feeds/fetch_feed
tasks, plus a minimal "add feed by URL" view that fetches synchronously
so new subscribers see results immediately.

Verified live against a real feed (hnrss.org) for ingestion, 304
dedup, and error backoff, in addition to the unit/integration test
suite (28 tests, all passing).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-10-30 16:03:20 -06:00
ksmithandClaude Sonnet 5 c635cd9330 Scaffold Django project and add first-run auth flow
Sets up the Evection project structure: split settings supporting both
SQLite and PostgreSQL via DATABASE_URL, Celery+Redis wiring, and the
planned app layout (accounts, feeds, articles, rules, search, reader,
pwa, core). Implements multi-user auth with a FreshRSS-style first-run
setup wizard, login/logout, and admin-only user management, covered by
automated tests.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-10-30 15:56:58 -06:00