Compare commits

...
Author SHA1 Message Date
ksmithandClaude Sonnet 5.5 7afda7d0ba Fix inline article expand and back-to-stream scope in production
Clicking an article title opened the detail page instead of expanding
it inline, and "Back to stream" then reset to every feed. Cause: the
title (and the new-articles banner) used inline onclick= handlers,
which production's CSP (script-src 'self') blocks — dev sends no CSP,
so it worked there. The title link just fell through to its href.

Behavior moves to delegated listeners in static/js/stream.js (modified
clicks still open in a new tab). A test now fails if any template gains
an inline on*= handler. The stream also passes its filter/tag/feed
scope through the article link so the detail page's back link returns
to it. Verified in a dev server enforcing the same CSP: title click
expands and collapses in place, URL unchanged.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-11-01 09:02:51 -07:00
ksmithandClaude Sonnet 5.5 37e658c49a Add configurable article retention (C7)
Articles and full HTML bodies accumulated forever, growing the
database, search index, and backups without bound. Site settings gets
a "Retention days" field (0, the default, keeps everything); a nightly
beat task deletes articles fetched longer ago than that in bounded
chunks, always keeping anything a user starred or saved for later.
Read state and tags cascade with the article and the FTS triggers drop
search entries. The dashboard shows the article count and oldest
article's age, and docs/install.md covers retention, the readiness
check, and SQLite's WAL tuning.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-11-01 08:51:20 -07:00
ksmithandClaude Sonnet 5.5 e5f1277014 Batch feed ingestion, add a readiness check, fail open without Redis (C2, C8)
Ingestion did a get_or_create per entry. It now prefetches existing
articles for the poll in one query, bulk-inserts new ones and
bulk-updates changed ones (falling back to row-by-row only if a
concurrent fetch wins a race), and dispatches rule evaluation in
bounded chunks so a huge first fetch is several small tasks.

/health/ stays a DB-only liveness check (upgrade.sh gates on it right
after restart). New /health/ready/ also checks Redis and that
poll_due_feeds ticked within five minutes — proving beat and a worker
are both alive — and returns 503 with per-check detail otherwise.

The sidebar unread-count cache and login throttle now degrade instead
of erroring when Redis is unreachable: every authenticated page and
every login would otherwise 500 during a cache outage. Found while
verifying in a sandbox with no Redis.

Also includes the dashboard's oldest-article metric (core/views.py).

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-11-01 08:51:20 -07:00
ksmithandClaude Sonnet 5.5 8d2900424a Use keyset pagination for the stream (C4)
The stream did a COUNT(*) and OFFSET slice on every request, ordered by
a non-unique (published_at, fetched_at) — deep pages got slower and
articles sharing a timestamp could repeat or vanish across pages.

It now orders by (Coalesce(published_at, fetched_at), id) and pages with
before/after cursors, fetching one extra row to know whether another
page exists instead of counting. Pagination links also keep the feed
scope, which the old links silently dropped.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-11-01 08:51:20 -07:00
ksmithandClaude Sonnet 5.5 70b0d4f277 Index Postgres search and harden SQLite search/concurrency (C5, C6)
Postgres recomputed a tsvector over every article's full text on every
search. A migration now adds a stored, GIN-indexed search_vector column
kept current by a trigger (so no ingest/update path can forget it), and
the backend reads that column. Untested here: no Postgres in the dev
sandbox, so the SQL is reviewed but not executed — worth one run of the
(skipped-on-SQLite) Postgres test module against a real instance.

SQLite search built one bound parameter per subscribed feed, so enough
feeds would exceed the variable limit, and the OperationalError was
swallowed into "no results". It now scopes through a subquery on
Subscription and logs failures. SQLite connections also get WAL mode,
a 20s busy timeout, and IMMEDIATE transactions so gunicorn, Celery, and
the FTS triggers stop tripping over each other.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-11-01 08:51:20 -07:00
ksmithandClaude Sonnet 5 cb0ce5a336 Batch rule/tag lookups and index FilterRule for the ingest query (C9)
FilterRule had no indexes beyond the default primary/foreign-key ones,
and apply_filter_rules_for_feed issued 2 queries per subscriber (its
enabled rules, its subscription tags) inside a loop over every
subscriber of the feed — both cost grew with subscriber/rule count on
every single feed poll.

Adds a composite index on (user, is_enabled, priority), matching the
exact filter+order shape every caller already uses (this task,
apps.rules.tasks' backlog re-apply). apply_filter_rules_for_feed now
fetches rules and subscription tags once for the whole feed's subscriber
set, grouped by user/subscription in Python, instead of once per
subscriber.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-10-31 16:09:36 -06:00
ksmithandClaude Sonnet 5 abf4cef55b Cache sidebar unread counts for a short TTL (C3)
apps.core.context_processors.sidebar runs on every authenticated page
render and called unread_counts_by_feed, a join/group-by over the
user's whole article corpus, every single time — cost grew with the
archive's size regardless of which page was being viewed.

unread_counts_by_feed now caches its {feed_id: count} result per user
for 30 seconds (apps.core.cache_utils). The reader views that change
the current user's own read state (toggle_read, mark_all_read, viewing
an article) invalidate it immediately so the sidebar never visibly lags
behind the action that just caused it; other paths (rule-driven state
changes during background ingestion) aren't explicitly invalidated and
simply self-heal within the TTL, which is an acceptable tradeoff
already made elsewhere in this app (the stream's own "new articles"
banner polls on the same 30s cadence).

Also adds a global cache.clear() autouse fixture (conftest.py) — the
test CACHES backend (LocMemCache) lives in process memory and survives
Django's per-test DB rollback, so a cached count keyed by a user's pk
could otherwise leak into an unrelated later test reusing that pk.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-10-31 16:09:28 -06:00
ksmithandClaude Sonnet 5 b33fe905d2 Push due-feed scheduling into an indexed DB query (C1)
poll_due_feeds loaded every active feed's timing fields and recomputed
each one's error-backoff due-ness in Python on every beat tick (every
1-2 minutes) — scheduler CPU and DB reads grew linearly with total feed
count regardless of how many were actually due.

Feed.next_fetch_at (indexed together with is_active) is now set by
fetch_feed itself right after every fetch attempt, success or error,
using the same backoff formula that used to run per-feed in
poll_due_feeds. The scheduler's query is now a direct indexed range
scan (next_fetch_at <= now), ordered so the most overdue feeds claim
first, and bounded to POLL_BATCH_SIZE so a large backlog spreads across
several ticks instead of dispatching everything at once. A migration
backfills next_fetch_at for existing feeds using the same formula.
reset_feed_backoff also clears next_fetch_at so a manually-reset feed
is immediately due again, matching its prior behavior.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-10-31 16:09:18 -06:00
ksmithandClaude Sonnet 5 b84cfe6474 Document the Nginx Proxy Manager fix for S8's X-Forwarded-For spoofing
docs/install.md covered the raw-nginx case via nginx.conf.example, but
gave no concrete guidance for Nginx Proxy Manager specifically, whose
generated config also defaults to the appending (spoofable) form of
X-Forwarded-For — the same production deployment this review is based
on uses NPM. Documents the Advanced tab / Custom Nginx Configuration
override, why it takes effect ahead of NPM's own default header, and a
note for anyone with something in front of NPM itself.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-10-31 15:35:07 -06:00
41 changed files with 1307 additions and 154 deletions

No files matched your search

+1
View File
@@ -45,4 +45,5 @@ def _setup_exempt_paths():
reverse("pwa:service_worker"), reverse("pwa:service_worker"),
reverse("pwa:offline"), reverse("pwa:offline"),
reverse("core:health"), reverse("core:health"),
reverse("core:health_ready"),
} }
+18
View File
@@ -72,3 +72,21 @@ def test_too_many_failed_attempts_against_one_username_blocks_that_username(clie
response = client.post(reverse("accounts:login"), {"username": "bob", "password": "pw-bob-123"}) response = client.post(reverse("accounts:login"), {"username": "bob", "password": "pw-bob-123"})
assert response.status_code == 302 assert response.status_code == 302
assert response.wsgi_request.user == bob assert response.wsgi_request.user == bob
@pytest.mark.django_db
def test_login_still_works_when_the_throttle_cache_is_down(client, monkeypatch):
from unittest.mock import Mock
from django.contrib.auth import get_user_model
get_user_model().objects.create_user("alice", password="pw-alice-123")
broken = Mock(side_effect=ConnectionError("redis down"))
monkeypatch.setattr("apps.accounts.views.cache.get", broken)
monkeypatch.setattr("apps.accounts.views.cache.incr", broken)
monkeypatch.setattr("apps.accounts.views.cache.set", broken)
bad = client.post(reverse("accounts:login"), {"username": "alice", "password": "wrong"})
assert bad.status_code == 200
good = client.post(reverse("accounts:login"), {"username": "alice", "password": "pw-alice-123"})
assert good.status_code == 302
+14
View File
@@ -1,3 +1,5 @@
import logging
from django.contrib import messages from django.contrib import messages
from django.contrib.auth import login from django.contrib.auth import login
from django.contrib.auth.decorators import user_passes_test from django.contrib.auth.decorators import user_passes_test
@@ -10,6 +12,8 @@ from .forms import CreateUserForm, SetupForm
from .middleware import SetupWizardMiddleware from .middleware import SetupWizardMiddleware
from .models import SetupLock, User from .models import SetupLock, User
logger = logging.getLogger(__name__)
is_admin = user_passes_test(lambda u: u.is_authenticated and u.is_staff) is_admin = user_passes_test(lambda u: u.is_authenticated and u.is_staff)
# Two independent throttles, either of which can block an attempt: by # Two independent throttles, either of which can block an attempt: by
@@ -41,14 +45,24 @@ def _attempts_key(kind: str, value: str) -> str:
def _too_many_attempts(key: str, limit: int) -> bool: def _too_many_attempts(key: str, limit: int) -> bool:
# Fails open if the cache (Redis) is unreachable: logging in must keep
# working through a cache outage, at the cost of no throttling until
# it's back — an outage shouldn't lock every user out of the app.
try:
return cache.get(key, 0) >= limit return cache.get(key, 0) >= limit
except Exception:
logger.warning("Login throttle cache unavailable; not rate limiting", exc_info=True)
return False
def _record_failed_attempt(key: str) -> None: def _record_failed_attempt(key: str) -> None:
try:
try: try:
cache.incr(key) cache.incr(key)
except ValueError: except ValueError:
cache.set(key, 1, LOGIN_RATE_LIMIT_WINDOW_SECONDS) cache.set(key, 1, LOGIN_RATE_LIMIT_WINDOW_SECONDS)
except Exception:
logger.warning("Login throttle cache unavailable; attempt not recorded", exc_info=True)
class LoginView(DjangoLoginView): class LoginView(DjangoLoginView):
+33 -4
View File
@@ -1,7 +1,14 @@
import logging
from django.core.cache import cache
from django.db.models import Count from django.db.models import Count
from apps.core.cache_utils import SIDEBAR_UNREAD_CACHE_TTL_SECONDS, sidebar_unread_cache_key
from .models import SAVED_TAG_NAME, STARRED_TAG_NAME, Article, ArticleTag, UserArticleState from .models import SAVED_TAG_NAME, STARRED_TAG_NAME, Article, ArticleTag, UserArticleState
logger = logging.getLogger(__name__)
def with_user_state(articles, user): def with_user_state(articles, user):
"""Annotates an iterable of Article instances with is_read/is_starred/ """Annotates an iterable of Article instances with is_read/is_starred/
@@ -44,9 +51,26 @@ def with_user_state(articles, user):
def unread_counts_by_feed(user): def unread_counts_by_feed(user):
"""{feed_id: unread_count} for every feed the user subscribes to, in one """{feed_id: unread_count} for every feed the user subscribes to — used
query — used by the sidebar, so it must stay cheap regardless of how by the sidebar's context processor, which runs on every authenticated
many feeds/articles exist.""" page render, so the underlying join/group-by over the user's whole
article corpus is cached for a short TTL rather than recomputed on
every single page view (C3). apps.core.cache_utils.
invalidate_sidebar_unread_cache is called wherever a request changes
the current user's own read state, so this is rarely visibly stale.
"""
cache_key = sidebar_unread_cache_key(user.pk)
# The cache is an optimization here, never a dependency: every
# authenticated page render goes through this, so a Redis outage must
# degrade to recomputing the count, not to a 500 on every page.
try:
counts = cache.get(cache_key)
except Exception:
logger.warning("Sidebar cache unavailable; computing unread counts directly")
counts = None
if counts is not None:
return counts
rows = ( rows = (
Article.objects.filter(feed__subscriptions__user=user) Article.objects.filter(feed__subscriptions__user=user)
.exclude(user_states__user=user, user_states__is_read=True) .exclude(user_states__user=user, user_states__is_read=True)
@@ -54,4 +78,9 @@ def unread_counts_by_feed(user):
.values("feed_id") .values("feed_id")
.annotate(unread=Count("id")) .annotate(unread=Count("id"))
) )
return {row["feed_id"]: row["unread"] for row in rows} counts = {row["feed_id"]: row["unread"] for row in rows}
try:
cache.set(cache_key, counts, SIDEBAR_UNREAD_CACHE_TTL_SECONDS)
except Exception:
logger.warning("Sidebar cache unavailable; skipping cache write")
return counts
+59 -13
View File
@@ -1,10 +1,14 @@
from celery import shared_task from datetime import timedelta
from celery import shared_task
from django.utils import timezone
from apps.core.models import SiteSettings
from apps.feeds.models import Subscription from apps.feeds.models import Subscription
from apps.rules.engine import evaluate_rules_for_article from apps.rules.engine import evaluate_rules_for_article
from apps.rules.models import FilterRule from apps.rules.models import FilterRule
from .models import Article, SubscriptionTag from .models import Article, ArticleTag, SubscriptionTag
@shared_task @shared_task
@@ -17,6 +21,10 @@ def apply_filter_rules_for_feed(feed_id: int, article_ids: list[int]):
Per-rule scoping is left to evaluate_rules_for_article/rule_in_scope Per-rule scoping is left to evaluate_rules_for_article/rule_in_scope
rather than filtered into this query, so this and the backlog rather than filtered into this query, so this and the backlog
re-apply task (apps.rules.tasks) can't scope rules differently. re-apply task (apps.rules.tasks) can't scope rules differently.
Rules and subscription tags are fetched once for every subscriber of
this feed, not once per subscriber (C9) — a popular feed with many
subscribers previously issued 2 queries per subscriber here.
""" """
if not article_ids: if not article_ids:
return 0 return 0
@@ -25,25 +33,63 @@ def apply_filter_rules_for_feed(feed_id: int, article_ids: list[int]):
if not articles: if not articles:
return 0 return 0
subscriptions = Subscription.objects.filter(feed_id=feed_id).select_related("user") subscriptions = list(Subscription.objects.filter(feed_id=feed_id).select_related("user"))
if not subscriptions:
return 0
rules_by_user_id: dict[int, list[FilterRule]] = {}
for rule in FilterRule.objects.filter(
user_id__in={s.user_id for s in subscriptions}, is_enabled=True
).order_by("priority", "id"):
rules_by_user_id.setdefault(rule.user_id, []).append(rule)
tag_ids_by_subscription_id: dict[int, set[int]] = {}
for subscription_id, tag_id in SubscriptionTag.objects.filter(
subscription_id__in=[s.pk for s in subscriptions]
).values_list("subscription_id", "tag_id"):
tag_ids_by_subscription_id.setdefault(subscription_id, set()).add(tag_id)
evaluated = 0 evaluated = 0
for subscription in subscriptions: for subscription in subscriptions:
rules = list( rules = rules_by_user_id.get(subscription.user_id)
FilterRule.objects.filter(user=subscription.user, is_enabled=True).order_by(
"priority", "id"
)
)
if not rules: if not rules:
continue continue
subscription_tag_ids = set( subscription_tag_ids = tag_ids_by_subscription_id.get(subscription.pk, set())
SubscriptionTag.objects.filter(
user=subscription.user, subscription=subscription
).values_list("tag_id", flat=True)
)
for article in articles: for article in articles:
evaluate_rules_for_article(subscription.user, article, rules, subscription_tag_ids) evaluate_rules_for_article(subscription.user, article, rules, subscription_tag_ids)
evaluated += 1 evaluated += 1
return evaluated return evaluated
PURGE_CHUNK_SIZE = 500
@shared_task
def purge_old_articles():
"""Deletes articles older than SiteSettings.retention_days (C7), in
chunks so one run never holds a huge delete in a single transaction.
Anything a user starred or saved for later (a system ArticleTag) is
kept — those are explicit "keep this" signals. UserArticleState and
ArticleTag rows go with their article via ON DELETE CASCADE, and the
SQLite FTS triggers drop the search entries. 0 days means keep
everything forever (the default).
"""
retention_days = SiteSettings.load().retention_days
if retention_days <= 0:
return 0
cutoff = timezone.now() - timedelta(days=retention_days)
keep = ArticleTag.objects.filter(tag__is_system=True).values("article_id")
deleted_total = 0
while True:
chunk = list(
Article.objects.filter(fetched_at__lt=cutoff)
.exclude(pk__in=keep)
.values_list("pk", flat=True)[:PURGE_CHUNK_SIZE]
)
if not chunk:
return deleted_total
Article.objects.filter(pk__in=chunk).delete()
deleted_total += len(chunk)
+76
View File
@@ -0,0 +1,76 @@
import pytest
from django.contrib.auth import get_user_model
from apps.core.cache_utils import invalidate_sidebar_unread_cache
from apps.feeds.models import Feed, Subscription
from .models import Article
from .queries import unread_counts_by_feed
User = get_user_model()
@pytest.fixture
def alice(db):
return User.objects.create_user("alice", password="pw-alice-123")
@pytest.fixture
def feed(db):
return Feed.objects.create(feed_url="https://example.com/feed.xml")
@pytest.mark.django_db
def test_unread_counts_by_feed_is_cached_across_calls(alice, feed):
"""unread_counts_by_feed (C3) caches its result — a second call right
after a new unread article appears must still see the stale cached
value until invalidated or the TTL elapses, proving it isn't
recomputing the query every time."""
Subscription.objects.create(user=alice, feed=feed)
Article.objects.create(feed=feed, guid="a-1")
first = unread_counts_by_feed(alice)
assert first == {feed.pk: 1}
Article.objects.create(feed=feed, guid="a-2")
second = unread_counts_by_feed(alice)
assert second == {feed.pk: 1}, "expected the cached count, not a fresh recount"
@pytest.mark.django_db
def test_invalidate_sidebar_unread_cache_forces_a_recount(alice, feed):
Subscription.objects.create(user=alice, feed=feed)
Article.objects.create(feed=feed, guid="a-1")
unread_counts_by_feed(alice)
Article.objects.create(feed=feed, guid="a-2")
invalidate_sidebar_unread_cache(alice.pk)
assert unread_counts_by_feed(alice) == {feed.pk: 2}
@pytest.mark.django_db
def test_unread_counts_by_feed_cache_is_per_user(alice, feed):
bob = User.objects.create_user("bob", password="pw-bob-123")
Subscription.objects.create(user=alice, feed=feed)
Subscription.objects.create(user=bob, feed=feed)
Article.objects.create(feed=feed, guid="a-1")
assert unread_counts_by_feed(alice) == {feed.pk: 1}
assert unread_counts_by_feed(bob) == {feed.pk: 1}
@pytest.mark.django_db
def test_unread_counts_still_work_when_the_cache_is_down(alice, feed, monkeypatch):
"""A Redis outage must degrade the sidebar to recomputing counts, not
500 every authenticated page."""
from unittest.mock import Mock
Subscription.objects.create(user=alice, feed=feed)
Article.objects.create(feed=feed, guid="a-1")
broken = Mock(side_effect=ConnectionError("redis down"))
monkeypatch.setattr("apps.articles.queries.cache.get", broken)
monkeypatch.setattr("apps.articles.queries.cache.set", broken)
assert unread_counts_by_feed(alice) == {feed.pk: 1}
invalidate_sidebar_unread_cache(alice.pk)
+74
View File
@@ -0,0 +1,74 @@
from datetime import timedelta
import pytest
from django.contrib.auth import get_user_model
from django.utils import timezone
from apps.core.models import SiteSettings
from apps.feeds.models import Feed
from .models import STARRED_TAG_NAME, Article, ArticleTag, UserArticleState, get_system_tag
from .tasks import purge_old_articles
User = get_user_model()
def _article(feed, guid, age_days):
article = Article.objects.create(feed=feed, guid=guid, title=guid)
Article.objects.filter(pk=article.pk).update(
fetched_at=timezone.now() - timedelta(days=age_days)
)
return article
@pytest.fixture
def feed(db):
return Feed.objects.create(feed_url="https://example.com/feed.xml")
@pytest.mark.django_db
def test_purge_is_a_noop_with_the_default_keep_forever_setting(feed):
_article(feed, "ancient", age_days=5000)
assert purge_old_articles() == 0
assert Article.objects.count() == 1
@pytest.mark.django_db
def test_purge_deletes_only_articles_older_than_the_retention_window(feed):
SiteSettings.objects.create(retention_days=30)
old = _article(feed, "old", age_days=45)
recent = _article(feed, "recent", age_days=5)
assert purge_old_articles() == 1
assert not Article.objects.filter(pk=old.pk).exists()
assert Article.objects.filter(pk=recent.pk).exists()
@pytest.mark.django_db
def test_purge_keeps_starred_articles_and_cascades_user_state(feed):
SiteSettings.objects.create(retention_days=30)
user = User.objects.create_user("alice", password="pw-alice-123")
starred = _article(feed, "starred-old", age_days=90)
ArticleTag.objects.create(
user=user, article=starred, tag=get_system_tag(user, STARRED_TAG_NAME)
)
doomed = _article(feed, "plain-old", age_days=90)
state = UserArticleState.objects.create(user=user, article=doomed, is_read=True)
purge_old_articles()
assert Article.objects.filter(pk=starred.pk).exists()
assert not Article.objects.filter(pk=doomed.pk).exists()
assert not UserArticleState.objects.filter(pk=state.pk).exists()
@pytest.mark.django_db
def test_purge_works_through_more_than_one_chunk(feed, monkeypatch):
monkeypatch.setattr("apps.articles.tasks.PURGE_CHUNK_SIZE", 2)
SiteSettings.objects.create(retention_days=30)
for i in range(5):
_article(feed, f"old-{i}", age_days=60)
assert purge_old_articles() == 5
assert Article.objects.count() == 0
+37
View File
@@ -150,3 +150,40 @@ def test_disabled_rule_does_not_apply(feed, alice):
@pytest.mark.django_db @pytest.mark.django_db
def test_dispatch_with_empty_article_ids_is_a_noop(feed): def test_dispatch_with_empty_article_ids_is_a_noop(feed):
assert apply_filter_rules_for_feed(feed.id, []) == 0 assert apply_filter_rules_for_feed(feed.id, []) == 0
@pytest.mark.django_db
def test_dispatch_fetches_rules_and_tags_once_per_feed_not_once_per_subscriber(feed):
"""Rules/subscription tags are batched across every subscriber of the
feed in one query each (C9), not re-queried per subscriber — assert
this by checking the query count stays flat as subscribers grow."""
from django.db import connection
from django.test.utils import CaptureQueriesContext
users = [User.objects.create_user(f"user{i}", password="pw-123456") for i in range(5)]
for user in users:
sub = Subscription.objects.create(user=user, feed=feed)
tag = Tag.objects.create(user=user, name="Tech")
SubscriptionTag.objects.create(user=user, subscription=sub, tag=tag)
FilterRule.objects.create(
user=user,
tag=tag,
match_field=FilterRule.FIELD_TITLE,
match_type=FilterRule.MATCH_CONTAINS,
match_value="Python",
action=FilterRule.ACTION_STAR,
)
article = Article.objects.create(feed=feed, guid="guid-1", title="Python news")
with CaptureQueriesContext(connection) as ctx:
apply_filter_rules_for_feed(feed.id, [article.pk])
# Exactly one SELECT against each table regardless of subscriber
# count (5 here) — a regression back to per-subscription querying
# would show up as 5 of each instead of 1.
rule_selects = [q for q in ctx.captured_queries if "rules_filterrule" in q["sql"]]
tag_selects = [q for q in ctx.captured_queries if "articles_subscriptiontag" in q["sql"]]
assert len(rule_selects) == 1
assert len(tag_selects) == 1
for user in users:
assert is_starred(user, article) is True
+1 -1
View File
@@ -14,7 +14,7 @@ class SiteSettingsAdmin(admin.ModelAdmin):
install. No changelist/add/delete views exist at all here. install. No changelist/add/delete views exist at all here.
""" """
fields = ["fetch_interval_minutes"] fields = ["fetch_interval_minutes", "retention_days"]
def has_add_permission(self, request): def has_add_permission(self, request):
return False return False
+29
View File
@@ -0,0 +1,29 @@
import logging
from django.core.cache import cache
logger = logging.getLogger(__name__)
# Short enough that a missed invalidation point self-heals within one
# page load's worth of staleness; long enough to actually absorb repeat
# authenticated-page renders (apps.core.context_processors.sidebar runs
# on every one) without recomputing unread_counts_by_feed's join/group-by
# over the user's whole article corpus each time (C3).
SIDEBAR_UNREAD_CACHE_TTL_SECONDS = 30
def sidebar_unread_cache_key(user_id: int) -> str:
return f"sidebar-unread:{user_id}"
def invalidate_sidebar_unread_cache(user_id: int) -> None:
"""Call this wherever a request changes the current user's own
read/unread state (toggle_read, mark_all_read, viewing an article) so
the sidebar reflects it on the very next render instead of waiting
out the TTL above. Rule-driven state changes during background feed
ingestion aren't invalidated this way — those self-heal within the
TTL, same as a missed call site here would."""
try:
cache.delete(sidebar_unread_cache_key(user_id))
except Exception:
logger.warning("Sidebar cache unavailable; nothing to invalidate")
@@ -0,0 +1,18 @@
# Generated by Django 5.2.18 on 2026-11-01 15:48
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
('core', '0001_initial'),
]
operations = [
migrations.AddField(
model_name='sitesettings',
name='retention_days',
field=models.PositiveIntegerField(default=0, help_text="Articles older than this many days (by when they were fetched) are deleted daily, along with everyone's read state and tags for them — except articles anyone has starred or saved for later. 0 keeps everything forever."),
),
]
+10
View File
@@ -18,6 +18,16 @@ class SiteSettings(models.Model):
), ),
) )
retention_days = models.PositiveIntegerField(
default=0,
help_text=(
"Articles older than this many days (by when they were fetched) are "
"deleted daily, along with everyone's read state and tags for them — "
"except articles anyone has starred or saved for later. 0 keeps "
"everything forever."
),
)
class Meta: class Meta:
verbose_name = "Site settings" verbose_name = "Site settings"
verbose_name_plural = "Site settings" verbose_name_plural = "Site settings"
-27
View File
@@ -1,13 +1,6 @@
from datetime import timedelta
from unittest.mock import patch
import pytest import pytest
from django.contrib.auth import get_user_model from django.contrib.auth import get_user_model
from django.urls import reverse from django.urls import reverse
from django.utils import timezone
from apps.feeds.models import Feed
from apps.feeds.tasks import poll_due_feeds
from .models import SiteSettings from .models import SiteSettings
@@ -30,26 +23,6 @@ def test_load_returns_the_existing_row_without_creating_a_second_one():
assert SiteSettings.objects.count() == 1 assert SiteSettings.objects.count() == 1
@pytest.mark.django_db
def test_poll_due_feeds_uses_the_site_wide_interval_for_every_feed():
SiteSettings.objects.create(fetch_interval_minutes=120)
due = Feed.objects.create(
feed_url="https://a.example.com/feed.xml",
last_fetched_at=timezone.now() - timedelta(minutes=150),
)
not_due = Feed.objects.create(
feed_url="https://b.example.com/feed.xml",
last_fetched_at=timezone.now() - timedelta(minutes=90),
)
with patch("apps.feeds.tasks.fetch_feed.delay") as mock_delay:
poll_due_feeds()
dispatched_ids = {call.args[0] for call in mock_delay.call_args_list}
assert dispatched_ids == {due.id}
assert not_due.id not in dispatched_ids
@pytest.mark.django_db @pytest.mark.django_db
def test_admin_changelist_redirects_straight_to_the_change_form(client): def test_admin_changelist_redirects_straight_to_the_change_form(client):
assert not SiteSettings.objects.exists() assert not SiteSettings.objects.exists()
+42
View File
@@ -22,6 +22,48 @@ def test_health_check_works_before_first_run_setup(client):
assert response.status_code == 200 assert response.status_code == 200
@pytest.mark.django_db
def test_health_ready_is_degraded_until_a_poll_tick_has_happened(client):
response = client.get(reverse("core:health_ready"))
assert response.status_code == 503
body = response.json()
assert body["status"] == "degraded"
assert body["checks"]["database"] is True
assert body["checks"]["redis"] is True
assert body["checks"]["poll_heartbeat"] is False
@pytest.mark.django_db
def test_health_ready_is_ok_after_poll_due_feeds_runs(client):
from apps.feeds.tasks import poll_due_feeds
poll_due_feeds()
response = client.get(reverse("core:health_ready"))
assert response.status_code == 200
assert response.json()["checks"] == {
"database": True,
"redis": True,
"poll_heartbeat": True,
}
@pytest.mark.django_db
def test_health_ready_flags_a_stale_poll_heartbeat(client):
from datetime import timedelta
from django.core.cache import cache
from django.utils import timezone
from apps.feeds.tasks import POLL_HEARTBEAT_CACHE_KEY
cache.set(POLL_HEARTBEAT_CACHE_KEY, timezone.now() - timedelta(minutes=30))
response = client.get(reverse("core:health_ready"))
assert response.status_code == 503
assert response.json()["checks"]["poll_heartbeat"] is False
@pytest.mark.django_db @pytest.mark.django_db
def test_dashboard_requires_staff(client, django_user_model): def test_dashboard_requires_staff(client, django_user_model):
django_user_model.objects.create_user("regular", password="pw-regular-123") django_user_model.objects.create_user("regular", password="pw-regular-123")
+1
View File
@@ -6,6 +6,7 @@ app_name = "core"
urlpatterns = [ urlpatterns = [
path("health/", views.health, name="health"), path("health/", views.health, name="health"),
path("health/ready/", views.health_ready, name="health_ready"),
path("dashboard/", views.dashboard, name="dashboard"), path("dashboard/", views.dashboard, name="dashboard"),
path("dashboard/feeds/<int:feed_id>/retry/", views.retry_feed, name="retry_feed"), path("dashboard/feeds/<int:feed_id>/retry/", views.retry_feed, name="retry_feed"),
] ]
+50 -3
View File
@@ -1,20 +1,27 @@
from django.contrib import messages from django.contrib import messages
from django.contrib.auth.decorators import user_passes_test from django.contrib.auth.decorators import user_passes_test
from django.core.cache import cache
from django.db import connection from django.db import connection
from django.db.models import Count, Q from django.db.models import Count, Min, Q
from django.http import JsonResponse from django.http import JsonResponse
from django.shortcuts import get_object_or_404, redirect, render from django.shortcuts import get_object_or_404, redirect, render
from django.utils import timezone
from django.views.decorators.http import require_POST from django.views.decorators.http import require_POST
from apps.articles.models import Article from apps.articles.models import Article
from apps.feeds.models import Feed, Subscription from apps.feeds.models import Feed, Subscription
from apps.feeds.tasks import fetch_feed, reset_feed_backoff from apps.feeds.tasks import POLL_HEARTBEAT_CACHE_KEY, fetch_feed, reset_feed_backoff
is_admin = user_passes_test(lambda u: u.is_authenticated and u.is_staff) is_admin = user_passes_test(lambda u: u.is_authenticated and u.is_staff)
# Beat ticks poll_due_feeds every minute; this many minutes without a tick
# means beat or the worker (or the broker between them) isn't working.
POLL_HEARTBEAT_MAX_AGE_SECONDS = 5 * 60
def health(request): def health(request):
"""Unauthenticated liveness/readiness check — reachable even before """Unauthenticated liveness check — reachable even before
first-run setup (see SetupWizardMiddleware). Used by scripts/upgrade.sh first-run setup (see SetupWizardMiddleware). Used by scripts/upgrade.sh
to confirm a freshly restarted deploy actually came up, and is a to confirm a freshly restarted deploy actually came up, and is a
reasonable target for any external uptime monitor too.""" reasonable target for any external uptime monitor too."""
@@ -26,6 +33,45 @@ def health(request):
return JsonResponse({"status": "ok"}) return JsonResponse({"status": "ok"})
def health_ready(request):
"""Readiness: liveness plus everything background fetching depends on
(C8) — the database, the Redis cache/broker, and a recent
poll_due_feeds tick, which proves beat is scheduling AND a worker is
consuming. /health/ above stays DB-only on purpose: scripts/upgrade.sh
gates a deploy on it right after restart, before beat has had a
chance to tick. Point an uptime monitor at this one instead."""
checks = {"database": False, "redis": False, "poll_heartbeat": False}
detail = {}
try:
with connection.cursor() as cursor:
cursor.execute("SELECT 1")
checks["database"] = True
except Exception:
pass
try:
cache.set("health-ready-probe", "1", timeout=10)
checks["redis"] = cache.get("health-ready-probe") == "1"
except Exception:
pass
try:
last_tick = cache.get(POLL_HEARTBEAT_CACHE_KEY)
except Exception:
last_tick = None
if last_tick is not None:
age = (timezone.now() - last_tick).total_seconds()
detail["poll_heartbeat_age_seconds"] = int(age)
checks["poll_heartbeat"] = age <= POLL_HEARTBEAT_MAX_AGE_SECONDS
ok = all(checks.values())
return JsonResponse(
{"status": "ok" if ok else "degraded", "checks": checks, **detail},
status=200 if ok else 503,
)
@is_admin @is_admin
def dashboard(request): def dashboard(request):
feeds = Feed.objects.annotate(subscriber_count=Count("subscriptions")) feeds = Feed.objects.annotate(subscriber_count=Count("subscriptions"))
@@ -39,6 +85,7 @@ def dashboard(request):
"error_feed_count": erroring_feeds.count(), "error_feed_count": erroring_feeds.count(),
"subscription_count": Subscription.objects.count(), "subscription_count": Subscription.objects.count(),
"article_count": Article.objects.count(), "article_count": Article.objects.count(),
"oldest_article_at": Article.objects.aggregate(oldest=Min("fetched_at"))["oldest"],
} }
return render( return render(
@@ -0,0 +1,24 @@
# Generated by Django 5.2.18 on 2026-10-31 22:00
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
("feeds", "0004_remove_feed_fetch_interval_minutes"),
]
operations = [
migrations.AddField(
model_name="feed",
name="next_fetch_at",
field=models.DateTimeField(blank=True, null=True),
),
migrations.AddIndex(
model_name="feed",
index=models.Index(
fields=["is_active", "next_fetch_at"], name="feeds_feed_is_acti_5c232e_idx"
),
),
]
@@ -0,0 +1,49 @@
from datetime import timedelta
from django.conf import settings
from django.db import migrations
def backfill_next_fetch_at(apps, schema_editor):
"""One-time backfill for feeds that existed before next_fetch_at (C1) —
everything created after this migration gets it set by fetch_feed
itself. Mirrors the exact formula apps.feeds.services.fetcher.
effective_interval_minutes used to compute on every poll tick, just
run once here instead of forever after.
"""
Feed = apps.get_model("feeds", "Feed")
SiteSettings = apps.get_model("core", "SiteSettings")
configured_interval = 60
settings_row = SiteSettings.objects.filter(pk=1).first()
if settings_row is not None:
configured_interval = settings_row.fetch_interval_minutes
floor = getattr(settings, "EVECTION_MIN_FETCH_INTERVAL_MINUTES", 15)
base = max(configured_interval, floor)
max_backoff = 24 * 60
for feed in Feed.objects.filter(last_fetched_at__isnull=False).only(
"id", "last_fetched_at", "fetch_error_count"
):
if feed.fetch_error_count <= 0:
interval = base
else:
interval = min(base * (2**feed.fetch_error_count), max_backoff)
feed.next_fetch_at = feed.last_fetched_at + timedelta(minutes=interval)
feed.save(update_fields=["next_fetch_at"])
def noop_reverse(apps, schema_editor):
pass
class Migration(migrations.Migration):
dependencies = [
("feeds", "0005_feed_next_fetch_at_and_more"),
("core", "0001_initial"),
]
operations = [
migrations.RunPython(backfill_next_fetch_at, noop_reverse),
]
+12
View File
@@ -42,6 +42,15 @@ class Feed(models.Model):
fetch_error_count = models.PositiveIntegerField(default=0) fetch_error_count = models.PositiveIntegerField(default=0)
is_active = models.BooleanField(default=True) is_active = models.BooleanField(default=True)
# Set by fetch_feed right after every fetch attempt (success or error)
# to last_fetched_at + effective_interval_minutes — precomputed once,
# at write time, instead of poll_due_feeds re-deriving due-ness (which
# depends on fetch_error_count's exponential backoff) for every active
# feed on every tick (C1). NULL means "due now", same as a feed that's
# never been fetched. Indexed with is_active below so the scheduler's
# query is a direct index range scan, not a full table scan.
next_fetch_at = models.DateTimeField(null=True, blank=True)
# Set by poll_due_feeds' atomic claim (an UPDATE ... WHERE clause, not # Set by poll_due_feeds' atomic claim (an UPDATE ... WHERE clause, not
# select_for_update — Celery Beat and fetch_feed run in separate # select_for_update — Celery Beat and fetch_feed run in separate
# transactions) and cleared once fetch_feed finishes, so overlapping # transactions) and cleared once fetch_feed finishes, so overlapping
@@ -53,6 +62,9 @@ class Feed(models.Model):
created_at = models.DateTimeField(auto_now_add=True) created_at = models.DateTimeField(auto_now_add=True)
class Meta:
indexes = [models.Index(fields=["is_active", "next_fetch_at"])]
def __str__(self): def __str__(self):
return self.title or self.feed_url return self.title or self.feed_url
+69 -31
View File
@@ -18,6 +18,7 @@ import feedparser
import nh3 import nh3
import requests import requests
from django.conf import settings from django.conf import settings
from django.db import IntegrityError, transaction
from django.utils import timezone from django.utils import timezone
from apps.articles.models import Article from apps.articles.models import Article
@@ -131,6 +132,26 @@ def _entry_content_html(entry) -> str:
return getattr(entry, "summary", "") or "" return getattr(entry, "summary", "") or ""
UPDATABLE_ARTICLE_FIELDS = (
"title",
"link",
"author",
"summary",
"content_html",
"published_at",
"content_hash",
)
def _ensure_pks(feed: Feed, articles: list[Article]) -> list[int]:
"""bulk_create only populates primary keys on backends that can
RETURN them (Postgres, SQLite 3.35+); look the rest up by guid."""
if all(a.pk for a in articles):
return [a.pk for a in articles]
guids = [a.guid for a in articles]
return list(Article.objects.filter(feed=feed, guid__in=guids).values_list("pk", flat=True))
def effective_interval_minutes(feed: Feed, configured_interval_minutes: int) -> int: def effective_interval_minutes(feed: Feed, configured_interval_minutes: int) -> int:
"""The actual polling interval to use for `feed`, applying exponential """The actual polling interval to use for `feed`, applying exponential
backoff on top of the site-wide `configured_interval_minutes` (see backoff on top of the site-wide `configured_interval_minutes` (see
@@ -217,52 +238,69 @@ def fetch_and_ingest(feed: Feed) -> FetchResult:
) )
entries = entries[:MAX_ENTRIES_PER_POLL] entries = entries[:MAX_ENTRIES_PER_POLL]
# One prefetch + bulk writes for the whole poll (C2), instead of a
# get_or_create (SELECT, then INSERT) per entry. Entries are keyed by
# guid so a feed that repeats one within a single response is only
# processed once.
parsed_entries: dict[str, dict] = {}
for entry in entries: for entry in entries:
guid = _entry_guid(entry) guid = _entry_guid(entry)
if guid in parsed_entries:
continue
title = (getattr(entry, "title", "") or "")[:1000] title = (getattr(entry, "title", "") or "")[:1000]
link = (getattr(entry, "link", "") or "")[:2000] link = (getattr(entry, "link", "") or "")[:2000]
author = (getattr(entry, "author", "") or "")[:500] author = (getattr(entry, "author", "") or "")[:500]
summary = (getattr(entry, "summary", "") or "")[:5000] summary = (getattr(entry, "summary", "") or "")[:5000]
content_html = sanitize_html(_entry_content_html(entry)) content_html = sanitize_html(_entry_content_html(entry))
published_at = _entry_published_at(entry) published_at = _entry_published_at(entry)
content_hash = _content_hash(title, link, author, summary, content_html, published_at) parsed_entries[guid] = {
article, created = Article.objects.get_or_create(
feed=feed,
guid=guid,
defaults={
"link": link,
"title": title, "title": title,
"link": link,
"author": author, "author": author,
"content_html": content_html,
"summary": summary, "summary": summary,
"content_hash": content_hash, "content_html": content_html,
"published_at": published_at, "published_at": published_at,
}, "content_hash": _content_hash(title, link, author, summary, content_html, published_at),
) }
if created: existing = {a.guid: a for a in Article.objects.filter(feed=feed, guid__in=parsed_entries)}
new_ids.append(article.pk)
elif article.content_hash != content_hash: to_create = [
article.title = title Article(feed=feed, guid=guid, **fields)
article.link = link for guid, fields in parsed_entries.items()
article.author = author if guid not in existing
article.summary = summary
article.content_html = content_html
article.published_at = published_at
article.content_hash = content_hash
article.save(
update_fields=[
"title",
"link",
"author",
"summary",
"content_html",
"published_at",
"content_hash",
] ]
to_update = []
for guid, article in existing.items():
fields = parsed_entries[guid]
if article.content_hash != fields["content_hash"]:
for name, value in fields.items():
setattr(article, name, value)
to_update.append(article)
if to_update:
Article.objects.bulk_update(to_update, list(UPDATABLE_ARTICLE_FIELDS), batch_size=100)
updated_ids.extend(a.pk for a in to_update)
if to_create:
try:
with transaction.atomic():
Article.objects.bulk_create(to_create)
except IntegrityError:
# Another worker inserted some of these guids between the
# prefetch above and now (two fetches of the same feed can
# overlap) — fall back to row-by-row so nothing is lost or
# duplicated, at the cost of the old per-entry round trips
# for this rare case only.
for article in to_create:
fields = {name: getattr(article, name) for name in UPDATABLE_ARTICLE_FIELDS}
obj, created = Article.objects.get_or_create(
feed=feed, guid=article.guid, defaults=fields
) )
updated_ids.append(article.pk) if created:
new_ids.append(obj.pk)
else:
new_ids.extend(_ensure_pks(feed, to_create))
return FetchResult( return FetchResult(
status=Feed.STATUS_OK, new_article_ids=new_ids, updated_article_ids=updated_ids status=Feed.STATUS_OK, new_article_ids=new_ids, updated_article_ids=updated_ids
+48 -23
View File
@@ -4,6 +4,7 @@ from io import BytesIO
from celery import shared_task from celery import shared_task
from django.contrib.auth import get_user_model from django.contrib.auth import get_user_model
from django.core.cache import cache
from django.db.models import Q from django.db.models import Q
from django.utils import timezone from django.utils import timezone
@@ -22,27 +23,38 @@ logger = logging.getLogger(__name__)
# while a worker that died mid-fetch doesn't wedge the feed forever. # while a worker that died mid-fetch doesn't wedge the feed forever.
CLAIM_STALE_MINUTES = 10 CLAIM_STALE_MINUTES = 10
# Caps how many feeds one poll_due_feeds tick will claim/dispatch, so a
# large backlog (first run after a long outage, a huge import) spreads
# across several beat ticks instead of enqueuing everything — and every
# bounded Feed query below stays index-only work, not a full scan (C1).
POLL_BATCH_SIZE = 500
RULE_DISPATCH_CHUNK_SIZE = 50
# Written on every poll_due_feeds tick; apps.core.views.health_ready
# reads it to tell whether beat is scheduling and a worker is consuming.
POLL_HEARTBEAT_CACHE_KEY = "poll-due-feeds-heartbeat"
POLL_HEARTBEAT_TTL_SECONDS = 3600
@shared_task @shared_task
def poll_due_feeds(): def poll_due_feeds():
"""Find every active Feed whose polling interval has elapsed, atomically """Find a bounded batch of active Feeds whose next_fetch_at has
claim it (so overlapping beat ticks or a slow-running previous fetch elapsed (set by fetch_feed itself after every attempt — see Feed.
can't dispatch the same feed twice), and dispatch a fetch_feed task for next_fetch_at), atomically claim them (so overlapping beat ticks or a
each claimed feed — fan-out happens here, not via one Celery Beat entry slow-running previous fetch can't dispatch the same feed twice), and
per feed. dispatch a fetch_feed task for each claimed feed — fan-out happens
here, not via one Celery Beat entry per feed.
""" """
now = timezone.now() now = timezone.now()
cache.set(POLL_HEARTBEAT_CACHE_KEY, now, timeout=POLL_HEARTBEAT_TTL_SECONDS)
claim_stale_before = now - timedelta(minutes=CLAIM_STALE_MINUTES) claim_stale_before = now - timedelta(minutes=CLAIM_STALE_MINUTES)
# Loaded once per call, not per feed — the interval is site-wide due_feed_ids = list(
# (apps.core.models.SiteSettings), so every feed below shares it. Feed.objects.filter(is_active=True)
configured_interval = SiteSettings.load().fetch_interval_minutes .filter(Q(next_fetch_at__isnull=True) | Q(next_fetch_at__lte=now))
due_feed_ids = [ .filter(Q(claimed_at__isnull=True) | Q(claimed_at__lt=claim_stale_before))
feed.id .order_by("next_fetch_at")
for feed in Feed.objects.filter(is_active=True).only( .values_list("id", flat=True)[:POLL_BATCH_SIZE]
"id", "last_fetched_at", "fetch_error_count"
) )
if _is_due(feed, configured_interval)
]
claimed_ids = [] claimed_ids = []
for feed_id in due_feed_ids: for feed_id in due_feed_ids:
# The UPDATE's WHERE clause re-checks staleness at the database # The UPDATE's WHERE clause re-checks staleness at the database
@@ -61,11 +73,14 @@ def poll_due_feeds():
return len(claimed_ids) return len(claimed_ids)
def _is_due(feed: Feed, configured_interval_minutes: int) -> bool: def _schedule_next_fetch(feed: Feed) -> None:
if feed.last_fetched_at is None: """Sets feed.next_fetch_at from its (just-updated) fetch_error_count —
return True called right before every feed.save() in fetch_feed, so the next
interval = effective_interval_minutes(feed, configured_interval_minutes) poll_due_feeds tick can find it with a plain indexed range query
return feed.last_fetched_at <= timezone.now() - timedelta(minutes=interval) instead of recomputing every active feed's backoff in Python (C1)."""
configured_interval = SiteSettings.load().fetch_interval_minutes
interval = effective_interval_minutes(feed, configured_interval)
feed.next_fetch_at = timezone.now() + timedelta(minutes=interval)
# Bounds a single feed's fetch+parse+ingest, independent of # Bounds a single feed's fetch+parse+ingest, independent of
@@ -102,10 +117,12 @@ def fetch_feed(feed_id: int):
record_fetch_error(feed, "Unexpected error — see worker logs") record_fetch_error(feed, "Unexpected error — see worker logs")
feed.last_fetched_at = timezone.now() feed.last_fetched_at = timezone.now()
feed.claimed_at = None feed.claimed_at = None
_schedule_next_fetch(feed)
feed.save() feed.save()
raise raise
feed.claimed_at = None feed.claimed_at = None
_schedule_next_fetch(feed)
feed.save() feed.save()
changed_article_ids = result.new_article_ids + result.updated_article_ids changed_article_ids = result.new_article_ids + result.updated_article_ids
@@ -116,7 +133,13 @@ def fetch_feed(feed_id: int):
len(result.new_article_ids), len(result.new_article_ids),
len(result.updated_article_ids), len(result.updated_article_ids),
) )
apply_filter_rules_for_feed.delay(feed_id, changed_article_ids) # Chunked so one huge poll (a first fetch of a long archive feed)
# becomes several bounded rule-evaluation tasks, each well inside
# the worker's time limits, instead of one unbounded one (C2).
for start in range(0, len(changed_article_ids), RULE_DISPATCH_CHUNK_SIZE):
apply_filter_rules_for_feed.delay(
feed_id, changed_article_ids[start : start + RULE_DISPATCH_CHUNK_SIZE]
)
return { return {
"status": result.status, "status": result.status,
@@ -127,9 +150,11 @@ def fetch_feed(feed_id: int):
@shared_task @shared_task
def reset_feed_backoff(feed_id: int): def reset_feed_backoff(feed_id: int):
"""Admin-triggered: clear a feed's error count so it's retried at its """Admin-triggered: clear a feed's error count and due-date backoff so
normal configured interval instead of the backed-off one.""" it's retried at its normal configured interval instead of the backed-
Feed.objects.filter(pk=feed_id).update(fetch_error_count=0) off one (next_fetch_at=None also makes it immediately due for the
next poll_due_feeds tick, same as a feed that's never been fetched)."""
Feed.objects.filter(pk=feed_id).update(fetch_error_count=0, next_fetch_at=None)
@shared_task @shared_task
+74
View File
@@ -264,3 +264,77 @@ def test_fetch_only_processes_the_first_max_entries_per_poll(feed):
assert len(result.new_article_ids) == MAX_ENTRIES_PER_POLL assert len(result.new_article_ids) == MAX_ENTRIES_PER_POLL
assert Article.objects.filter(feed=feed).count() == MAX_ENTRIES_PER_POLL assert Article.objects.filter(feed=feed).count() == MAX_ENTRIES_PER_POLL
assert not Article.objects.filter(feed=feed, guid=f"guid-{MAX_ENTRIES_PER_POLL}").exists() assert not Article.objects.filter(feed=feed, guid=f"guid-{MAX_ENTRIES_PER_POLL}").exists()
def _feed_xml(count, title_suffix=""):
items = "".join(
f"<item><title>Item {i}{title_suffix}</title><guid>guid-{i}</guid></item>"
for i in range(count)
)
return (
f'<?xml version="1.0"?><rss version="2.0"><channel><title>T</title>{items}</channel></rss>'
)
@pytest.mark.django_db
@responses.activate
def test_ingest_uses_a_constant_number_of_article_queries(feed):
"""C2: one prefetch + bulk writes per poll, not a get_or_create per
entry — query count must not grow with the number of entries."""
from django.db import connection
from django.test.utils import CaptureQueriesContext
responses.add(responses.GET, FEED_URL, body=_feed_xml(40), status=200)
with CaptureQueriesContext(connection) as ctx:
result = fetch_and_ingest(feed)
assert len(result.new_article_ids) == 40
assert len(ctx.captured_queries) < 10
assert Article.objects.filter(feed=feed).count() == 40
@pytest.mark.django_db
@responses.activate
def test_ingest_dedups_a_guid_repeated_within_one_response(feed):
xml = (
'<?xml version="1.0"?><rss version="2.0"><channel><title>T</title>'
"<item><title>A</title><guid>same</guid></item>"
"<item><title>B</title><guid>same</guid></item>"
"</channel></rss>"
)
responses.add(responses.GET, FEED_URL, body=xml, status=200)
result = fetch_and_ingest(feed)
assert len(result.new_article_ids) == 1
assert Article.objects.filter(feed=feed).count() == 1
@pytest.mark.django_db
@responses.activate
def test_ingest_reports_updates_separately_from_new_articles(feed):
responses.add(responses.GET, FEED_URL, body=_feed_xml(3), status=200)
fetch_and_ingest(feed)
responses.replace(responses.GET, FEED_URL, body=_feed_xml(4, " edited"), status=200)
result = fetch_and_ingest(feed)
assert len(result.new_article_ids) == 1
assert len(result.updated_article_ids) == 3
assert Article.objects.get(feed=feed, guid="guid-0").title == "Item 0 edited"
@pytest.mark.django_db
@responses.activate
def test_ingest_falls_back_to_row_by_row_when_a_concurrent_insert_wins(feed):
from unittest.mock import patch
from django.db import IntegrityError
responses.add(responses.GET, FEED_URL, body=_feed_xml(3), status=200)
with patch.object(Article.objects, "bulk_create", side_effect=IntegrityError("race")):
result = fetch_and_ingest(feed)
assert len(result.new_article_ids) == 3
assert Article.objects.filter(feed=feed).count() == 3
+87
View File
@@ -12,18 +12,23 @@ from apps.feeds.tasks import fetch_feed, poll_due_feeds
@pytest.mark.django_db @pytest.mark.django_db
def test_poll_due_feeds_dispatches_only_due_feeds(): def test_poll_due_feeds_dispatches_only_due_feeds():
# next_fetch_at (set by fetch_feed itself, C1), not last_fetched_at,
# is what poll_due_feeds' query filters on now.
never_fetched = Feed.objects.create(feed_url="https://a.example.com/feed.xml") never_fetched = Feed.objects.create(feed_url="https://a.example.com/feed.xml")
recently_fetched = Feed.objects.create( recently_fetched = Feed.objects.create(
feed_url="https://b.example.com/feed.xml", feed_url="https://b.example.com/feed.xml",
last_fetched_at=timezone.now(), last_fetched_at=timezone.now(),
next_fetch_at=timezone.now() + timedelta(hours=1),
) )
overdue = Feed.objects.create( overdue = Feed.objects.create(
feed_url="https://c.example.com/feed.xml", feed_url="https://c.example.com/feed.xml",
last_fetched_at=timezone.now() - timedelta(hours=2), last_fetched_at=timezone.now() - timedelta(hours=2),
next_fetch_at=timezone.now() - timedelta(hours=1),
) )
inactive_but_overdue = Feed.objects.create( inactive_but_overdue = Feed.objects.create(
feed_url="https://d.example.com/feed.xml", feed_url="https://d.example.com/feed.xml",
last_fetched_at=timezone.now() - timedelta(hours=2), last_fetched_at=timezone.now() - timedelta(hours=2),
next_fetch_at=timezone.now() - timedelta(hours=1),
is_active=False, is_active=False,
) )
@@ -45,6 +50,7 @@ def test_poll_due_feeds_does_not_reclaim_a_feed_it_just_claimed():
overdue = Feed.objects.create( overdue = Feed.objects.create(
feed_url="https://c.example.com/feed.xml", feed_url="https://c.example.com/feed.xml",
last_fetched_at=timezone.now() - timedelta(hours=2), last_fetched_at=timezone.now() - timedelta(hours=2),
next_fetch_at=timezone.now() - timedelta(hours=1),
) )
with patch("apps.feeds.tasks.fetch_feed.delay") as mock_delay: with patch("apps.feeds.tasks.fetch_feed.delay") as mock_delay:
@@ -65,6 +71,7 @@ def test_poll_due_feeds_reclaims_a_stale_claim():
overdue = Feed.objects.create( overdue = Feed.objects.create(
feed_url="https://c.example.com/feed.xml", feed_url="https://c.example.com/feed.xml",
last_fetched_at=timezone.now() - timedelta(hours=2), last_fetched_at=timezone.now() - timedelta(hours=2),
next_fetch_at=timezone.now() - timedelta(hours=1),
claimed_at=timezone.now() - timedelta(minutes=CLAIM_STALE_MINUTES + 1), claimed_at=timezone.now() - timedelta(minutes=CLAIM_STALE_MINUTES + 1),
) )
@@ -141,3 +148,83 @@ def test_fetch_feed_task_records_error_and_clears_claim_on_unexpected_exception(
assert feed.fetch_error_count == 1 assert feed.fetch_error_count == 1
assert feed.claimed_at is None assert feed.claimed_at is None
assert feed.last_fetched_at is not None assert feed.last_fetched_at is not None
assert feed.next_fetch_at is not None
@pytest.mark.django_db
@responses.activate
def test_fetch_feed_task_sets_next_fetch_at_so_poll_due_feeds_can_find_it_later():
"""next_fetch_at (C1) is what poll_due_feeds' query filters on — a
feed fetch_feed just succeeded on must come back due again only after
the configured interval, not immediately."""
feed = Feed.objects.create(feed_url="https://example.com/feed.xml")
responses.add(responses.GET, feed.feed_url, body=GOOD_FEED_XML, status=200)
fetch_feed(feed.id)
feed.refresh_from_db()
assert feed.next_fetch_at is not None
assert feed.next_fetch_at > timezone.now()
with patch("apps.feeds.tasks.fetch_feed.delay") as mock_delay:
poll_due_feeds()
mock_delay.assert_not_called()
@pytest.mark.django_db
def test_reset_feed_backoff_makes_the_feed_immediately_due():
from apps.feeds.tasks import reset_feed_backoff
feed = Feed.objects.create(
feed_url="https://example.com/feed.xml",
fetch_error_count=5,
last_fetched_at=timezone.now(),
next_fetch_at=timezone.now() + timedelta(hours=12),
)
reset_feed_backoff(feed.id)
feed.refresh_from_db()
assert feed.fetch_error_count == 0
assert feed.next_fetch_at is None
@pytest.mark.django_db
@responses.activate
def test_fetch_feed_schedules_the_next_fetch_using_the_site_wide_interval():
"""fetch_feed (not poll_due_feeds — see apps.feeds.tasks.
_schedule_next_fetch, C1) is what reads SiteSettings' site-wide
interval and sets Feed.next_fetch_at from it after every fetch."""
from apps.core.models import SiteSettings
SiteSettings.objects.create(fetch_interval_minutes=120)
feed = Feed.objects.create(feed_url="https://example.com/feed.xml")
responses.add(responses.GET, feed.feed_url, body=GOOD_FEED_XML, status=200)
before = timezone.now()
fetch_feed(feed.id)
feed.refresh_from_db()
assert feed.next_fetch_at is not None
expected = before + timedelta(minutes=120)
assert abs((feed.next_fetch_at - expected).total_seconds()) < 5
@pytest.mark.django_db
@responses.activate
def test_fetch_feed_dispatches_rule_evaluation_in_bounded_chunks(monkeypatch):
monkeypatch.setattr("apps.feeds.tasks.RULE_DISPATCH_CHUNK_SIZE", 2)
feed = Feed.objects.create(feed_url="https://example.com/feed.xml")
items = "".join(f"<item><title>T{i}</title><guid>g{i}</guid></item>" for i in range(5))
responses.add(
responses.GET,
feed.feed_url,
body=f'<?xml version="1.0"?><rss version="2.0"><channel>{items}</channel></rss>',
status=200,
)
with patch("apps.feeds.tasks.apply_filter_rules_for_feed.delay") as mock_delay:
fetch_feed(feed.id)
chunk_sizes = [len(call.args[1]) for call in mock_delay.call_args_list]
assert chunk_sizes == [2, 2, 1]
+184
View File
@@ -289,3 +289,187 @@ def test_stream_exposes_the_latest_article_id_and_scope_for_the_banner(client, a
assert f"since={article.pk}" in content assert f"since={article.pk}" in content
assert "filter=all" in content assert "filter=all" in content
@pytest.mark.django_db
def test_sidebar_unread_count_updates_immediately_after_reading_an_article(client, alice):
"""unread_counts_by_feed caches its result for a short TTL (C3) — the
reader views that change the current user's own read state must
invalidate it, or the sidebar would show a stale count until the TTL
elapses on every one of these very common actions."""
feed = Feed.objects.create(feed_url="https://example.com/feed.xml")
Subscription.objects.create(user=alice, feed=feed)
article = Article.objects.create(feed=feed, guid="a-1", title="Hello")
client.login(username="alice", password="pw-alice-123")
response = client.get(reverse("reader:stream"))
assert '<span class="unread-count">1</span>' in response.content.decode()
client.get(reverse("reader:article_detail", args=[article.pk]))
response = client.get(reverse("reader:stream"))
assert "unread-count" not in response.content.decode()
@pytest.mark.django_db
def test_sidebar_unread_count_updates_immediately_after_toggle_read(client, alice):
feed = Feed.objects.create(feed_url="https://example.com/feed.xml")
Subscription.objects.create(user=alice, feed=feed)
article = Article.objects.create(feed=feed, guid="a-1", title="Hello")
UserArticleState.objects.create(user=alice, article=article, is_read=True)
client.login(username="alice", password="pw-alice-123")
response = client.get(reverse("reader:stream"))
assert "unread-count" not in response.content.decode()
client.post(reverse("reader:toggle_read", args=[article.pk]))
response = client.get(reverse("reader:stream"))
assert '<span class="unread-count">1</span>' in response.content.decode()
@pytest.mark.django_db
def test_sidebar_unread_count_updates_immediately_after_mark_all_read(client, alice):
feed = Feed.objects.create(feed_url="https://example.com/feed.xml")
Subscription.objects.create(user=alice, feed=feed)
Article.objects.create(feed=feed, guid="a-1", title="Hello")
client.login(username="alice", password="pw-alice-123")
response = client.get(reverse("reader:stream"))
assert '<span class="unread-count">1</span>' in response.content.decode()
client.post(reverse("reader:mark_all_read"))
response = client.get(reverse("reader:stream"))
assert "unread-count" not in response.content.decode()
def _stream_article_ids(client, query):
response = client.get(reverse("reader:stream"), query)
content = response.content.decode()
ids = [int(m) for m in re.findall(r'<li id="article-(\d+)"', content)]
return ids, content
def _link_query(content, label):
match = re.search(rf'<a href="\?([^"]+)">{label}', content)
if not match:
return None
from urllib.parse import parse_qsl
return dict(parse_qsl(match.group(1).replace("&amp;", "&")))
@pytest.mark.django_db
def test_stream_keyset_pagination_pages_through_tied_timestamps_without_gaps(client, alice):
"""Keyset pagination (C4) on (sort_at, id): articles sharing an exact
timestamp must neither repeat nor go missing across page boundaries,
which offset paging over a non-unique order can't guarantee."""
from django.utils import timezone
from apps.reader.views import PAGE_SIZE
feed = Feed.objects.create(feed_url="https://example.com/feed.xml")
Subscription.objects.create(user=alice, feed=feed)
same_moment = timezone.now()
total = PAGE_SIZE * 2 + 5
created = [
Article.objects.create(feed=feed, guid=f"g{i}", title=f"T{i}", published_at=same_moment)
for i in range(total)
]
client.login(username="alice", password="pw-alice-123")
seen = []
query = {"filter": "all"}
pages = 0
while query:
ids, content = _stream_article_ids(client, query)
assert len(ids) <= PAGE_SIZE
seen.extend(ids)
query = _link_query(content, "Older")
pages += 1
assert pages < 10
assert pages == 3
assert len(seen) == total
assert sorted(seen) == sorted(a.pk for a in created)
@pytest.mark.django_db
def test_stream_newer_link_returns_to_the_previous_page(client, alice):
from django.utils import timezone
from apps.reader.views import PAGE_SIZE
feed = Feed.objects.create(feed_url="https://example.com/feed.xml")
Subscription.objects.create(user=alice, feed=feed)
now = timezone.now()
for i in range(PAGE_SIZE + 5):
Article.objects.create(
feed=feed,
guid=f"g{i}",
title=f"T{i}",
published_at=now - timezone.timedelta(minutes=i),
)
client.login(username="alice", password="pw-alice-123")
first_ids, first_content = _stream_article_ids(client, {"filter": "all"})
assert _link_query(first_content, "Newer") is None
second_ids, second_content = _stream_article_ids(client, _link_query(first_content, "Older"))
assert len(second_ids) == 5
assert _link_query(second_content, "Older") is None
back_ids, _ = _stream_article_ids(client, _link_query(second_content, "Newer"))
assert back_ids == first_ids
@pytest.mark.django_db
def test_stream_pagination_links_keep_the_feed_scope(client, alice):
from apps.reader.views import PAGE_SIZE
feed = Feed.objects.create(feed_url="https://example.com/feed.xml")
Subscription.objects.create(user=alice, feed=feed)
for i in range(PAGE_SIZE + 1):
Article.objects.create(feed=feed, guid=f"g{i}", title=f"T{i}")
client.login(username="alice", password="pw-alice-123")
_, content = _stream_article_ids(client, {"filter": "all", "feed": feed.pk})
assert _link_query(content, "Older")["feed"] == str(feed.pk)
def test_templates_use_no_inline_event_handlers():
"""Production's CSP (script-src 'self') blocks inline on*= handlers —
they work in dev, where no CSP is sent, and silently do nothing in
production. Behavior belongs in static/js."""
from pathlib import Path
from django.conf import settings
pattern = re.compile(r"""<[^>]+\s(on[a-z]+)\s*=""", re.IGNORECASE)
offenders = []
for template in Path(settings.BASE_DIR, "templates").rglob("*.html"):
for match in pattern.finditer(template.read_text()):
offenders.append(f"{template.name}: {match.group(1)}")
assert offenders == []
@pytest.mark.django_db
def test_article_detail_back_link_keeps_the_stream_scope(client, alice):
feed = Feed.objects.create(feed_url="https://example.com/feed.xml")
Subscription.objects.create(user=alice, feed=feed)
article = Article.objects.create(feed=feed, guid="g1", title="Hello")
client.login(username="alice", password="pw-alice-123")
stream = client.get(
reverse("reader:stream"), {"filter": "all", "feed": feed.pk}
).content.decode()
link = re.search(r'href="(/articles/\d+/\?[^"]+)"', stream)
assert link is not None
detail = client.get(link.group(1).replace("&amp;", "&")).content.decode()
assert f"filter=all&amp;feed={feed.pk}" in detail
assert "Back to stream" in detail
assert article.title in detail
+70 -13
View File
@@ -1,8 +1,10 @@
from datetime import datetime
from itertools import groupby from itertools import groupby
from urllib.parse import urlencode from urllib.parse import urlencode
from django.contrib.auth.decorators import login_required from django.contrib.auth.decorators import login_required
from django.db.models import Max, Q from django.db.models import Max, Q
from django.db.models.functions import Coalesce
from django.http import Http404 from django.http import Http404
from django.shortcuts import get_object_or_404, redirect, render from django.shortcuts import get_object_or_404, redirect, render
from django.urls import reverse from django.urls import reverse
@@ -19,6 +21,7 @@ from apps.articles.models import (
get_system_tag, get_system_tag,
) )
from apps.articles.queries import with_user_state from apps.articles.queries import with_user_state
from apps.core.cache_utils import invalidate_sidebar_unread_cache
from apps.feeds.models import Feed from apps.feeds.models import Feed
FILTER_CHOICES = [ FILTER_CHOICES = [
@@ -156,26 +159,68 @@ def _filtered_queryset(request, active_filter):
return _apply_scope(request, queryset) return _apply_scope(request, queryset)
def _encode_cursor(article) -> str:
return f"{article.sort_at.isoformat()},{article.pk}"
def _decode_cursor(raw):
"""(sort_at, id) from a cursor produced by _encode_cursor, or None for
a missing/garbled one (treated as "no cursor" — the first page)."""
if not raw:
return None
try:
timestamp, pk = raw.rsplit(",", 1)
return datetime.fromisoformat(timestamp), int(pk)
except ValueError:
return None
@login_required @login_required
def stream(request): def stream(request):
active_filter = _resolve_active_filter(request) active_filter = _resolve_active_filter(request)
queryset, active_tag, active_feed = _filtered_queryset(request, active_filter) queryset, active_tag, active_feed = _filtered_queryset(request, active_filter)
try:
page = max(int(request.GET.get("page", 1)), 1)
except ValueError:
page = 1
start, end = (page - 1) * PAGE_SIZE, page * PAGE_SIZE
total_count = queryset.count()
articles = with_user_state(queryset[start:end], request.user)
scope_params = {"filter": active_filter} scope_params = {"filter": active_filter}
if active_tag: if active_tag:
scope_params["tag"] = active_tag.pk scope_params["tag"] = active_tag.pk
if active_feed: if active_feed:
scope_params["feed"] = active_feed.pk scope_params["feed"] = active_feed.pk
# Keyset pagination (C4) over (sort_at, id) — sort_at falls back to
# fetched_at for entries a feed gave no published date, and id breaks
# ties so pages neither shift nor repeat articles that share a
# timestamp. No COUNT(*): fetching one row past the page size is
# enough to know whether another page exists in that direction.
ordered = queryset.annotate(sort_at=Coalesce("published_at", "fetched_at"))
after = _decode_cursor(request.GET.get("after"))
before = _decode_cursor(request.GET.get("before"))
if after:
sort_at, pk = after
rows = list(
ordered.filter(Q(sort_at__gt=sort_at) | Q(sort_at=sort_at, id__gt=pk)).order_by(
"sort_at", "id"
)[: PAGE_SIZE + 1]
)
has_newer_page = len(rows) > PAGE_SIZE
has_older_page = True
rows = rows[:PAGE_SIZE][::-1]
else:
if before:
sort_at, pk = before
ordered = ordered.filter(Q(sort_at__lt=sort_at) | Q(sort_at=sort_at, id__lt=pk))
rows = list(ordered.order_by("-sort_at", "-id")[: PAGE_SIZE + 1])
has_older_page = len(rows) > PAGE_SIZE
has_newer_page = bool(before)
rows = rows[:PAGE_SIZE]
articles = with_user_state(rows, request.user)
newer_qs = older_qs = ""
if articles:
newer_qs = urlencode({**scope_params, "after": _encode_cursor(articles[0])})
older_qs = urlencode({**scope_params, "before": _encode_cursor(articles[-1])})
context = { context = {
"date_groups": _group_by_date(articles), "date_groups": _group_by_date(articles),
"active_filter": active_filter, "active_filter": active_filter,
@@ -183,15 +228,17 @@ def stream(request):
"active_tag": active_tag, "active_tag": active_tag,
"active_feed": active_feed, "active_feed": active_feed,
"user_tags": Tag.objects.filter(user=request.user), "user_tags": Tag.objects.filter(user=request.user),
"page": page, "has_newer_page": has_newer_page and bool(articles),
"has_next_page": end < total_count, "has_older_page": has_older_page and bool(articles),
"has_previous_page": page > 1, "newer_qs": newer_qs,
"older_qs": older_qs,
# Lets the "new articles" banner ask, on this exact same scope, # Lets the "new articles" banner ask, on this exact same scope,
# whether anything with a higher id has shown up since this page # whether anything with a higher id has shown up since this page
# was rendered — id (not published_at) because it's monotonic # was rendered — id (not published_at) because it's monotonic
# regardless of how a feed backdates an entry's published date. # regardless of how a feed backdates an entry's published date.
"latest_article_id": queryset.aggregate(Max("id"))["id__max"] or 0, "latest_article_id": queryset.aggregate(Max("id"))["id__max"] or 0,
"new_check_qs": urlencode(scope_params), "new_check_qs": urlencode(scope_params),
"scope_qs": urlencode(scope_params),
} }
return render(request, "reader/stream.html", context) return render(request, "reader/stream.html", context)
@@ -240,6 +287,7 @@ def mark_all_read(request):
for aid in new_article_ids for aid in new_article_ids
] ]
) )
invalidate_sidebar_unread_cache(request.user.pk)
stream_url = reverse("reader:stream") stream_url = reverse("reader:stream")
query = request.GET.urlencode() query = request.GET.urlencode()
@@ -254,6 +302,7 @@ def _load_article_and_mark_read(user, article_id):
state.is_read = True state.is_read = True
state.read_at = timezone.now() state.read_at = timezone.now()
state.save(update_fields=["is_read", "read_at"]) state.save(update_fields=["is_read", "read_at"])
invalidate_sidebar_unread_cache(user.pk)
article.is_read = state.is_read article.is_read = state.is_read
article.is_starred, article.is_saved_for_later = _article_tag_flags(user, article) article.is_starred, article.is_saved_for_later = _article_tag_flags(user, article)
@@ -263,7 +312,14 @@ def _load_article_and_mark_read(user, article_id):
@login_required @login_required
def article_detail(request, article_id): def article_detail(request, article_id):
article = _load_article_and_mark_read(request.user, article_id) article = _load_article_and_mark_read(request.user, article_id)
return render(request, "reader/article_detail.html", {"article": article}) # The stream passes its filter/tag/feed scope through the link so
# "Back to stream" returns to it instead of resetting to every feed.
back_params = {k: request.GET[k] for k in ("filter", "tag", "feed") if request.GET.get(k)}
return render(
request,
"reader/article_detail.html",
{"article": article, "back_qs": urlencode(back_params)},
)
@login_required @login_required
@@ -285,6 +341,7 @@ def toggle_read(request, article_id):
state.is_read = not state.is_read state.is_read = not state.is_read
state.read_at = timezone.now() if state.is_read else None state.read_at = timezone.now() if state.is_read else None
state.save(update_fields=["is_read", "read_at"]) state.save(update_fields=["is_read", "read_at"])
invalidate_sidebar_unread_cache(request.user.pk)
article.is_read = state.is_read article.is_read = state.is_read
article.is_starred, article.is_saved_for_later = _article_tag_flags(request.user, article) article.is_starred, article.is_saved_for_later = _article_tag_flags(request.user, article)
@@ -0,0 +1,23 @@
# Generated by Django 5.2.18 on 2026-10-31 22:05
from django.conf import settings
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
("articles", "0004_starred_saved_as_system_tags"),
("feeds", "0006_backfill_feed_next_fetch_at"),
("rules", "0002_folder_scope_to_tag_scope"),
migrations.swappable_dependency(settings.AUTH_USER_MODEL),
]
operations = [
migrations.AddIndex(
model_name="filterrule",
index=models.Index(
fields=["user", "is_enabled", "priority"], name="rules_filte_user_id_dcb19b_idx"
),
),
]
+10
View File
@@ -75,6 +75,16 @@ class FilterRule(models.Model):
class Meta: class Meta:
ordering = ["priority", "id"] ordering = ["priority", "id"]
indexes = [
# Matches the ingest-time query shape exactly (C9): every
# caller — apps.articles.tasks.apply_filter_rules_for_feed,
# apps.rules.tasks.reapply_rules_for_user_backlog — filters by
# (user, is_enabled) and orders by priority. `user` alone
# already has an index from the ForeignKey, but it doesn't
# cover is_enabled or let the database satisfy the ORDER BY
# from the index itself.
models.Index(fields=["user", "is_enabled", "priority"]),
]
def __str__(self): def __str__(self):
label = self.name or f"{self.get_match_field_display()} {self.get_match_type_display()}" label = self.name or f"{self.get_match_field_display()} {self.get_match_type_display()}"
+13 -12
View File
@@ -1,27 +1,28 @@
from django.contrib.postgres.search import SearchQuery, SearchRank, SearchVector
from apps.articles.models import Article from apps.articles.models import Article
from .base import SearchBackend from .base import SearchBackend
# Matches the stored search_vector column, GIN index, and maintaining
# trigger created by apps/search/migrations/0002_postgres_search_vector.py
# — the query reads that column directly instead of recomputing a
# tsvector over every article's full text on every search (C5).
_VECTOR = "articles_article.search_vector"
_QUERY = "plainto_tsquery('english', %s)"
class PostgresSearchBackend(SearchBackend): class PostgresSearchBackend(SearchBackend):
"""Computes the search vector on the fly at query time rather than
storing/indexing it on Article — keeps the schema identical to the
SQLite backend. Revisit with a stored+GIN-indexed column only if an
instance's archive grows large enough to need it."""
def search(self, query: str, user, limit: int = 50): def search(self, query: str, user, limit: int = 50):
if not query.strip(): if not query.strip():
return Article.objects.none() return Article.objects.none()
vector = SearchVector("title", weight="A") + SearchVector("content_html", weight="B")
search_query = SearchQuery(query)
return ( return (
Article.objects.filter(feed__subscriptions__user=user) Article.objects.filter(feed__subscriptions__user=user)
.select_related("feed") .select_related("feed")
.annotate(search=vector, rank=SearchRank(vector, search_query)) .extra(
.filter(search=search_query) select={"rank": f"ts_rank({_VECTOR}, {_QUERY})"},
select_params=[query],
where=[f"{_VECTOR} @@ {_QUERY}"],
params=[query],
)
.order_by("-rank", "-published_at")[:limit] .order_by("-rank", "-published_at")[:limit]
) )
+15 -9
View File
@@ -1,12 +1,14 @@
import logging
import sqlite3 import sqlite3
from django.db import connection from django.db import connection
from apps.articles.models import Article from apps.articles.models import Article
from apps.feeds.models import Subscription
from .base import SearchBackend from .base import SearchBackend
logger = logging.getLogger(__name__)
def _fts_phrase_query(raw_query: str) -> str: def _fts_phrase_query(raw_query: str) -> str:
"""Wraps the whole user input as a single FTS5 phrase so arbitrary """Wraps the whole user input as a single FTS5 phrase so arbitrary
@@ -21,24 +23,28 @@ class SQLiteFTSBackend(SearchBackend):
if not query.strip(): if not query.strip():
return [] return []
feed_ids = list(Subscription.objects.filter(user=user).values_list("feed_id", flat=True)) # Scoped through a subquery on Subscription rather than an IN
if not feed_ids: # list built from every subscribed feed id (C6): a user with
return [] # thousands of feeds would otherwise exceed SQLite's bound-
# parameter limit and the whole search would fail.
placeholders = ",".join(["%s"] * len(feed_ids)) sql = """
sql = f"""
SELECT a.id SELECT a.id
FROM articles_article a FROM articles_article a
JOIN articles_fts f ON f.rowid = a.id JOIN articles_fts f ON f.rowid = a.id
WHERE articles_fts MATCH %s AND a.feed_id IN ({placeholders}) WHERE articles_fts MATCH %s
AND a.feed_id IN (SELECT feed_id FROM feeds_subscription WHERE user_id = %s)
ORDER BY rank ORDER BY rank
LIMIT %s LIMIT %s
""" """
try: try:
with connection.cursor() as cursor: with connection.cursor() as cursor:
cursor.execute(sql, [_fts_phrase_query(query), *feed_ids, limit]) cursor.execute(sql, [_fts_phrase_query(query), user.pk, limit])
ordered_ids = [row[0] for row in cursor.fetchall()] ordered_ids = [row[0] for row in cursor.fetchall()]
except sqlite3.OperationalError: except sqlite3.OperationalError:
# A genuinely bad FTS query is rare (the input is quoted as a
# phrase), so this is far more likely a real operational
# failure — log it instead of silently showing "no results".
logger.exception("SQLite FTS search failed for user %s", user.pk)
return [] return []
if not ordered_ids: if not ordered_ids:
@@ -0,0 +1,61 @@
"""Stored, GIN-indexed tsvector for Postgres search (C5), kept current by a
trigger so no ingestion/update code path can forget to maintain it.
No-ops entirely on SQLite — its FTS5 table (0001_sqlite_fts) already plays
this role. The column is deliberately not a Django model field: a
SearchVectorField would break SQLite, and nothing but the search backend
ever reads it (apps/search/backends/postgres.py).
"""
from django.db import migrations
VECTOR_EXPR = (
"setweight(to_tsvector('english', coalesce({row}title, '')), 'A') || "
"setweight(to_tsvector('english', coalesce({row}content_html, '')), 'B')"
)
CREATE = f"""
ALTER TABLE articles_article ADD COLUMN IF NOT EXISTS search_vector tsvector;
UPDATE articles_article SET search_vector = {VECTOR_EXPR.format(row="")};
CREATE INDEX IF NOT EXISTS articles_article_search_vector_gin
ON articles_article USING GIN (search_vector);
CREATE OR REPLACE FUNCTION articles_article_search_vector_update() RETURNS trigger AS $$
BEGIN
NEW.search_vector := {VECTOR_EXPR.format(row="NEW.")};
RETURN NEW;
END
$$ LANGUAGE plpgsql;
DROP TRIGGER IF EXISTS articles_article_search_vector_trg ON articles_article;
CREATE TRIGGER articles_article_search_vector_trg
BEFORE INSERT OR UPDATE OF title, content_html ON articles_article
FOR EACH ROW EXECUTE FUNCTION articles_article_search_vector_update();
"""
DROP = """
DROP TRIGGER IF EXISTS articles_article_search_vector_trg ON articles_article;
DROP FUNCTION IF EXISTS articles_article_search_vector_update();
DROP INDEX IF EXISTS articles_article_search_vector_gin;
ALTER TABLE articles_article DROP COLUMN IF EXISTS search_vector;
"""
def create_vector(apps, schema_editor):
if schema_editor.connection.vendor != "postgresql":
return
schema_editor.execute(CREATE)
def drop_vector(apps, schema_editor):
if schema_editor.connection.vendor != "postgresql":
return
schema_editor.execute(DROP)
class Migration(migrations.Migration):
dependencies = [("search", "0001_sqlite_fts"), ("articles", "0001_initial")]
operations = [migrations.RunPython(create_vector, drop_vector)]
+4
View File
@@ -17,4 +17,8 @@ app.conf.beat_schedule = {
"task": "apps.feeds.tasks.poll_due_feeds", "task": "apps.feeds.tasks.poll_due_feeds",
"schedule": crontab(minute="*/1"), "schedule": crontab(minute="*/1"),
}, },
"purge-old-articles": {
"task": "apps.articles.tasks.purge_old_articles",
"schedule": crontab(hour=3, minute=30),
},
} }
+15
View File
@@ -77,6 +77,21 @@ ASGI_APPLICATION = "config.asgi.application"
# postgres://user:pass@host:5432/dbname # postgres://user:pass@host:5432/dbname
DATABASES = {"default": env.db_url("DATABASE_URL", default=f"sqlite:///{BASE_DIR / 'db.sqlite3'}")} DATABASES = {"default": env.db_url("DATABASE_URL", default=f"sqlite:///{BASE_DIR / 'db.sqlite3'}")}
# gunicorn, Celery workers, and the FTS triggers all write to the same
# SQLite file (C6): without WAL, readers and a writer block each other,
# and without a busy timeout a contended write fails immediately with
# "database is locked" instead of waiting its turn. IMMEDIATE takes the
# write lock at transaction start so a read-then-write transaction can't
# deadlock upgrading it mid-way.
if DATABASES["default"]["ENGINE"] == "django.db.backends.sqlite3":
DATABASES["default"].setdefault("OPTIONS", {}).update(
{
"timeout": 20,
"transaction_mode": "IMMEDIATE",
"init_command": "PRAGMA journal_mode=WAL; PRAGMA synchronous=NORMAL;",
}
)
AUTH_USER_MODEL = "accounts.User" AUTH_USER_MODEL = "accounts.User"
AUTH_PASSWORD_VALIDATORS = [ AUTH_PASSWORD_VALIDATORS = [
+1 -1
View File
@@ -36,7 +36,7 @@ SECURE_PROXY_SSL_HEADER = ("HTTP_X_FORWARDED_PROTO", "https")
# over loopback (see ALLOWED_HOSTS above) it has no X-Forwarded-Proto # over loopback (see ALLOWED_HOSTS above) it has no X-Forwarded-Proto
# header to prove it's "secure", and there's no TLS listener on gunicorn's # header to prove it's "secure", and there's no TLS listener on gunicorn's
# own port to redirect to anyway. # own port to redirect to anyway.
SECURE_REDIRECT_EXEMPT = [r"^health/$"] SECURE_REDIRECT_EXEMPT = [r"^health/$", r"^health/ready/$"]
# WhiteNoise serves /static/ directly from gunicorn instead of relying on # WhiteNoise serves /static/ directly from gunicorn instead of relying on
# the reverse proxy to alias the staticfiles directory on disk — the proxy # the reverse proxy to alias the staticfiles directory on disk — the proxy
+14
View File
@@ -1,8 +1,22 @@
import pytest import pytest
from django.core.cache import cache
from apps.accounts.middleware import SetupWizardMiddleware from apps.accounts.middleware import SetupWizardMiddleware
@pytest.fixture(autouse=True)
def _reset_django_cache():
"""config.settings.test overrides CACHES to LocMemCache, which — like
SetupWizardMiddleware's own cache below — lives in process memory and
survives Django's per-test DB rollback. Without this, a cached
sidebar unread count (apps.core.cache_utils) keyed by a user's pk
could leak into a later, unrelated test that happens to create a user
with the same auto-incrementing pk."""
cache.clear()
yield
cache.clear()
@pytest.fixture(autouse=True) @pytest.fixture(autouse=True)
def _reset_setup_wizard_cache(): def _reset_setup_wizard_cache():
"""SetupWizardMiddleware caches "does any User exist" on the class """SetupWizardMiddleware caches "does any User exist" on the class
+39 -2
View File
@@ -274,6 +274,25 @@ next paragraph for why it matters). Whichever proxy you use,
headers that need forwarding either way, including that overwrite-not- headers that need forwarding either way, including that overwrite-not-
append distinction for `X-Forwarded-For`. append distinction for `X-Forwarded-For`.
**Nginx Proxy Manager specifically:** NPM's generated config already sets
`X-Forwarded-For` to the appending (unsafe) form by default — override it
per proxy host rather than editing NPM's own templates: open the proxy
host entry for Evection, go to its **Advanced** tab, and add this to the
**Custom Nginx Configuration** box:
```nginx
proxy_set_header X-Forwarded-For $remote_addr;
```
This lands inside NPM's own `location` block, after its default headers,
so nginx uses this one (the last `proxy_set_header` for a given name in
the same block wins) — no edit to any file NPM manages, and it survives
NPM regenerating its config. Save, and NPM reloads nginx for you. If
something else sits in front of NPM itself (a CDN, another load balancer,
Cloudflare, ...), `$remote_addr` at NPM's layer is that upstream's
address, not the original client — forward whatever *that* layer's own
trusted client-IP header says instead.
Either way, the proxy must set `X-Forwarded-Proto: https` on requests it Either way, the proxy must set `X-Forwarded-Proto: https` on requests it
forwards once it's terminating TLS — Evection's `prod` settings forwards once it's terminating TLS — Evection's `prod` settings
(`SECURE_PROXY_SSL_HEADER` in `config/settings/prod.py`) trust that header to (`SECURE_PROXY_SSL_HEADER` in `config/settings/prod.py`) trust that header to
@@ -294,8 +313,26 @@ surfaces any feeds that are failing to fetch, which is the fastest way to catch
misconfigured feed or an outage at the source. misconfigured feed or an outage at the source.
`/health/` is an unauthenticated liveness check (DB connectivity only) — `/health/` is an unauthenticated liveness check (DB connectivity only) —
`scripts/upgrade.sh` uses it to confirm a deploy actually came back up, and `scripts/upgrade.sh` uses it to confirm a deploy actually came back up.
it's a reasonable target for an external uptime monitor too. Point an external uptime monitor at `/health/ready/` instead: it also checks
Redis and that the background poller has ticked in the last five minutes
(proving Celery beat is scheduling *and* a worker is consuming), and returns
503 with a per-check breakdown when any of them fails — so "the site loads
but nothing is being fetched" actually alerts.
**Retention.** By default every fetched article is kept forever. In the Django
admin, "Site settings" has a *Retention days* field: set it to delete articles
fetched longer ago than that, nightly (03:30 UTC, via Celery beat). Articles
anyone has starred or saved for later are always kept; read state and tags for
deleted articles go with them. The dashboard shows the article count and the
age of the oldest one, so you can see what retention is doing.
**SQLite.** SQLite installs run in WAL mode with a 20-second busy timeout (set
automatically in `config/settings/base.py`) so gunicorn, the Celery worker, and
the full-text-search triggers can share the file without "database is locked"
errors. That's plenty for a personal or small-team instance; for many users or
a very large archive, use PostgreSQL (its search uses a stored, GIN-indexed
vector kept current by a trigger, added by migration automatically).
## Backups ## Backups
+23 -5
View File
@@ -1,8 +1,11 @@
function toggleArticle(event, id) { // Delegated listeners, not inline onclick="" attributes: production's
event.preventDefault(); // Content Security Policy (config/settings/prod.py, script-src 'self')
var link = event.currentTarget; // blocks inline event handlers, which silently turned the title click into
// a plain link to the detail page and the new-articles banner into a no-op.
function toggleArticle(link) {
var row = link.closest("li.article-row"); var row = link.closest("li.article-row");
var body = document.getElementById("article-body-" + id); var body = row.querySelector(".article-inline-body");
var wasOpen = row.classList.contains("is-expanded"); var wasOpen = row.classList.contains("is-expanded");
document.querySelectorAll("li.article-row.is-expanded").forEach(function (openRow) { document.querySelectorAll("li.article-row.is-expanded").forEach(function (openRow) {
@@ -25,5 +28,20 @@ function toggleArticle(event, id) {
); );
} }
} }
return false;
} }
document.addEventListener("click", function (event) {
var link = event.target.closest("a.article-title[data-inline-url]");
if (link) {
// Leave modified clicks alone so "open in new tab" still works.
if (event.button !== 0 || event.ctrlKey || event.metaKey || event.shiftKey || event.altKey) {
return;
}
event.preventDefault();
toggleArticle(link);
return;
}
if (event.target.closest(".new-articles-banner")) {
location.reload();
}
});
+1 -1
View File
@@ -6,7 +6,7 @@
<ul class="dashboard-stats"> <ul class="dashboard-stats">
<li>{{ stats.feed_count }} feed(s) — {{ stats.active_feed_count }} active</li> <li>{{ stats.feed_count }} feed(s) — {{ stats.active_feed_count }} active</li>
<li>{{ stats.subscription_count }} subscription(s)</li> <li>{{ stats.subscription_count }} subscription(s)</li>
<li>{{ stats.article_count }} article(s)</li> <li>{{ stats.article_count }} article(s){% if stats.oldest_article_at %}, oldest fetched <time datetime="{{ stats.oldest_article_at|date:"c" }}" data-local-format="long">{{ stats.oldest_article_at|date:"M j, Y H:i" }}</time>{% endif %}</li>
<li>{{ stats.error_feed_count }} feed(s) currently erroring</li> <li>{{ stats.error_feed_count }} feed(s) currently erroring</li>
</ul> </ul>
+4 -4
View File
@@ -10,10 +10,10 @@
</ul> </ul>
<nav class="pagination"> <nav class="pagination">
{% if has_previous_page %} {% if has_newer_page %}
<a href="?filter={{ active_filter }}{% if active_tag %}&tag={{ active_tag.pk }}{% endif %}&page={{ page|add:"-1" }}">&larr; Newer</a> <a href="?{{ newer_qs }}">&larr; Newer</a>
{% endif %} {% endif %}
{% if has_next_page %} {% if has_older_page %}
<a href="?filter={{ active_filter }}{% if active_tag %}&tag={{ active_tag.pk }}{% endif %}&page={{ page|add:"1" }}">Older &rarr;</a> <a href="?{{ older_qs }}">Older &rarr;</a>
{% endif %} {% endif %}
</nav> </nav>
+2 -3
View File
@@ -6,9 +6,8 @@
<img src="{{ article.feed.favicon_url }}" alt="" class="feed-favicon"> <img src="{{ article.feed.favicon_url }}" alt="" class="feed-favicon">
{% endif %} {% endif %}
<span class="article-feed-name">{{ article.feed.title|default:article.feed.feed_url }}</span> <span class="article-feed-name">{{ article.feed.title|default:article.feed.feed_url }}</span>
<a href="{% url 'reader:article_detail' article.pk %}" class="article-title" <a href="{% url 'reader:article_detail' article.pk %}{% if scope_qs %}?{{ scope_qs }}{% endif %}" class="article-title"
data-inline-url="{% url 'reader:article_inline' article.pk %}" data-inline-url="{% url 'reader:article_inline' article.pk %}">{{ article.title|default:"(untitled)" }}</a>
onclick="return toggleArticle(event, {{ article.pk }})">{{ article.title|default:"(untitled)" }}</a>
{% if article.published_at %} {% if article.published_at %}
<time class="article-time" datetime="{{ article.published_at|date:"c" }}" data-local-format="short">{{ article.published_at|date:"g:i A" }}</time> <time class="article-time" datetime="{{ article.published_at|date:"c" }}" data-local-format="short">{{ article.published_at|date:"g:i A" }}</time>
{% endif %} {% endif %}
+1 -1
View File
@@ -1,5 +1,5 @@
{% if new_count %} {% if new_count %}
<button type="button" class="new-articles-banner" onclick="location.reload()"> <button type="button" class="new-articles-banner">
{{ new_count }} new article{{ new_count|pluralize }} — click to show {{ new_count }} new article{{ new_count|pluralize }} — click to show
</button> </button>
{% endif %} {% endif %}
+1 -1
View File
@@ -1,7 +1,7 @@
{% extends "base.html" %} {% extends "base.html" %}
{% block title %}{{ article.title|default:"Article" }} — Evection{% endblock %} {% block title %}{{ article.title|default:"Article" }} — Evection{% endblock %}
{% block content %} {% block content %}
<p><a href="{% url 'reader:stream' %}">&larr; Back to stream</a></p> <p><a href="{% url 'reader:stream' %}{% if back_qs %}?{{ back_qs }}{% endif %}">&larr; Back to stream</a></p>
<article> <article>
{% include "reader/_article_actions.html" %} {% include "reader/_article_actions.html" %}
{% include "reader/_article_content.html" %} {% include "reader/_article_content.html" %}