Compare commits

...
Author SHA1 Message Date
ksmithandClaude Sonnet 5 e760eeda1d Show each tag's total unread count and remember collapsed sections
Closes gitea#24: the sidebar's per-feed unread counts were already
computed by sidebar() (apps.core.context_processors); each tag group's
total is now the sum of its own feeds' counts (a feed can carry more
than one tag, so this reuses the same per-feed data rather than a
second query) and renders next to the tag name like a feed's own
count.

Closes gitea#25: each sidebar <details> section (one per tag, plus
"Untagged") now carries a stable data-sidebar-key, and
static/js/sidebar-state.js remembers which ones are collapsed in
localStorage — this isn't a SPA, so every click re-renders the sidebar
server-side, and previously every section reset back to expanded on
the next page load. Per-browser, same pattern as theme.js/theme-init.js,
not synced across devices/accounts.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-10-31 12:34:57 -06:00
ksmithandClaude Sonnet 5 9439f9cc59 Add SSRF protection to feed fetching: validate scheme/address, bound redirects and response size (S1)
fetch_and_ingest fetched a feed_url supplied straight from the "add
feed" form or an uploaded OPML file, with requests following redirects
on its own and no check that the target (or a redirect's target) was
actually a public, fetchable feed rather than an internal service,
localhost, or a cloud metadata endpoint — reachable from wherever this
app runs, on any authenticated user's behalf.

apps.feeds.services.ssrf_guard.safe_get() replaces the direct
requests.get() call: it resolves and validates the hostname (rejecting
anything that isn't a globally routable address) before the initial
request and before following each redirect hop manually, caps the
number of hops, and bounds the response body size by streaming it
instead of buffering it unbounded. OPML-imported feeds go through the
same check the first time poll_due_feeds fetches them, since this is
the one chokepoint every feed URL passes through regardless of which
path created the Feed row.

Known residual gap, documented in ssrf_guard.py: this validates then
connects rather than pinning the TCP connection to the exact address it
validated, so a narrow DNS-rebinding race isn't closed. Fixing that
needs a custom transport and felt like more machinery than this
self-hosted app's threat model currently justifies.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-10-31 12:25:20 -06:00
ksmithandClaude Sonnet 5 fd8eeb605e Scope backlog re-apply per-feed and always queue it through Celery (T5, S5)
Completes the T5 fix: reapply_rules_for_user_backlog now computes each
feed's subscription-tag ids up front and passes them to
evaluate_rules_for_article per article, so a feed/tag-scoped rule from
a different feed can't fire on this backlog (see the previous commit's
rule_in_scope() in apps.rules.engine).

Also completes S5's second half: the "Re-apply" view now always
dispatches reapply_rules_for_user.delay() instead of calling the
backlog function inline from the request — a large backlog or a
regex that outruns the new timeout guard no longer risks tying up a
web worker.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-10-31 12:20:25 -06:00
ksmithandClaude Sonnet 5 91c0b1dd26 Centralize rule feed/tag scoping in the engine and fix backlog re-apply (T5)
reapply_rules_for_user_backlog narrowed its *backlog* to the triggering
rule's feed/tag scope, but then loaded every enabled rule for the user
and ran all of them against that backlog with no per-rule scope check
— so re-applying one feed- or tag-scoped rule could also fire an
unrelated rule scoped to a different feed/tag, if it happened to match.

rule_in_scope() moves the feed/tag scope check into the engine itself
(apply_filter_rules_for_feed already computed this per-subscription via
query filters; it now just passes the tag ids and lets the engine
decide), so backlog re-apply and ingest-time dispatch can't drift into
scoping rules differently again.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-10-31 12:19:56 -06:00
17 changed files with 722 additions and 36 deletions

No files matched your search

+12 -12
View File
@@ -1,5 +1,4 @@
from celery import shared_task
from django.db.models import Q
from apps.feeds.models import Subscription
from apps.rules.engine import evaluate_rules_for_article
@@ -14,6 +13,10 @@ def apply_filter_rules_for_feed(feed_id: int, article_ids: list[int]):
FilterRules (scoped to this feed/tag or unscoped) against the given
articles. Dispatched by the fetch pipeline after new/updated articles
are committed for a feed.
Per-rule scoping is left to evaluate_rules_for_article/rule_in_scope
rather than filtered into this query, so this and the backlog
re-apply task (apps.rules.tasks) can't scope rules differently.
"""
if not article_ids:
return 0
@@ -26,24 +29,21 @@ def apply_filter_rules_for_feed(feed_id: int, article_ids: list[int]):
evaluated = 0
for subscription in subscriptions:
rules = FilterRule.objects.filter(user=subscription.user, is_enabled=True).filter(
Q(feed_id=feed_id) | Q(feed__isnull=True)
rules = list(
FilterRule.objects.filter(user=subscription.user, is_enabled=True).order_by(
"priority", "id"
)
)
subscription_tag_ids = list(
if not rules:
continue
subscription_tag_ids = set(
SubscriptionTag.objects.filter(
user=subscription.user, subscription=subscription
).values_list("tag_id", flat=True)
)
if subscription_tag_ids:
rules = rules.filter(Q(tag_id__in=subscription_tag_ids) | Q(tag__isnull=True))
else:
rules = rules.filter(tag__isnull=True)
rules = list(rules.order_by("priority", "id"))
if not rules:
continue
for article in articles:
evaluate_rules_for_article(subscription.user, article, rules)
evaluate_rules_for_article(subscription.user, article, rules, subscription_tag_ids)
evaluated += 1
return evaluated
+13 -1
View File
@@ -44,7 +44,19 @@ def sidebar(request):
return {
"sidebar_tag_groups": [
{"tag": tag, "subscriptions": feeds_by_tag.get(tag.pk, [])} for tag in tags
{
"tag": tag,
"subscriptions": feeds_by_tag.get(tag.pk, []),
# A feed can carry more than one tag, so this is a sum
# over whichever of the user's feeds this tag happens to
# cover right now — not a separately-tracked counter — and
# reuses the per-feed counts already computed above rather
# than a second query.
"unread_count": sum(
subscription.unread_count for subscription in feeds_by_tag.get(tag.pk, [])
),
}
for tag in tags
],
"sidebar_untagged": untagged,
"sidebar_starred_count": ArticleTag.objects.filter(
+82
View File
@@ -0,0 +1,82 @@
import pytest
from django.contrib.auth import get_user_model
from django.test import RequestFactory
from django.urls import reverse
from apps.articles.models import Article, SubscriptionTag, Tag
from apps.feeds.models import Feed, Subscription
from .context_processors import sidebar
User = get_user_model()
@pytest.fixture
def alice(db):
return User.objects.create_user("alice", password="pw-alice-123")
def _sidebar_context(user):
request = RequestFactory().get("/")
request.user = user
return sidebar(request)
@pytest.mark.django_db
def test_tag_unread_count_sums_its_feeds(alice):
feed_a = Feed.objects.create(feed_url="https://a.example.com/feed.xml")
feed_b = Feed.objects.create(feed_url="https://b.example.com/feed.xml")
sub_a = Subscription.objects.create(user=alice, feed=feed_a)
sub_b = Subscription.objects.create(user=alice, feed=feed_b)
tag = Tag.objects.create(user=alice, name="work")
SubscriptionTag.objects.create(user=alice, subscription=sub_a, tag=tag)
SubscriptionTag.objects.create(user=alice, subscription=sub_b, tag=tag)
for i in range(3):
Article.objects.create(feed=feed_a, guid=f"a-{i}", title=f"A{i}")
for i in range(4):
Article.objects.create(feed=feed_b, guid=f"b-{i}", title=f"B{i}")
context = _sidebar_context(alice)
[group] = context["sidebar_tag_groups"]
assert group["tag"] == tag
assert group["unread_count"] == 7
@pytest.mark.django_db
def test_tag_unread_count_is_zero_with_no_unread_articles(alice):
feed = Feed.objects.create(feed_url="https://example.com/feed.xml")
sub = Subscription.objects.create(user=alice, feed=feed)
tag = Tag.objects.create(user=alice, name="empty")
SubscriptionTag.objects.create(user=alice, subscription=sub, tag=tag)
context = _sidebar_context(alice)
[group] = context["sidebar_tag_groups"]
assert group["unread_count"] == 0
@pytest.mark.django_db
def test_sidebar_sections_carry_a_stable_collapse_state_key(client, alice):
feed = Feed.objects.create(feed_url="https://example.com/feed.xml")
sub = Subscription.objects.create(user=alice, feed=feed)
tag = Tag.objects.create(user=alice, name="work")
SubscriptionTag.objects.create(user=alice, subscription=sub, tag=tag)
other_feed = Feed.objects.create(feed_url="https://other.example.com/feed.xml")
Subscription.objects.create(user=alice, feed=other_feed) # untagged
client.login(username="alice", password="pw-alice-123")
response = client.get(reverse("reader:stream"))
content = response.content.decode()
# static/js/sidebar-state.js reads these to restore/persist each
# section's collapsed state across page loads.
assert f'data-sidebar-key="tag-{tag.pk}"' in content
assert 'data-sidebar-key="untagged"' in content
@pytest.mark.django_db
def test_base_template_loads_the_sidebar_state_script(client, alice):
client.login(username="alice", password="pw-alice-123")
response = client.get(reverse("reader:stream"))
assert "sidebar-state.js" in response.content.decode()
+21
View File
@@ -0,0 +1,21 @@
import socket
import pytest
@pytest.fixture(autouse=True)
def _fake_public_dns(monkeypatch):
"""apps.feeds.services.ssrf_guard resolves a feed's real hostname via
socket.getaddrinfo before every fetch. Tests use made-up hostnames
like example.com/a.example.com purely as fixture data — they must
not depend on real DNS/network access to pass that check, and must
resolve deterministically regardless of this sandbox's network
access. Every lookup here resolves to a fixed, definitely-public IP;
tests that specifically exercise the "resolves to a non-public
address" rejection path patch socket.getaddrinfo themselves instead.
"""
def _fake_getaddrinfo(host, *args, **kwargs):
return [(socket.AF_INET, socket.SOCK_STREAM, 6, "", ("93.184.216.34", 0))]
monkeypatch.setattr(socket, "getaddrinfo", _fake_getaddrinfo)
+6 -1
View File
@@ -23,6 +23,8 @@ from django.utils import timezone
from apps.articles.models import Article
from apps.feeds.models import Feed
from .ssrf_guard import SSRFBlocked, safe_get
logger = logging.getLogger(__name__)
FETCH_TIMEOUT_SECONDS = 15
@@ -139,7 +141,10 @@ def fetch_and_ingest(feed: Feed) -> FetchResult:
headers["If-Modified-Since"] = feed.last_modified
try:
response = requests.get(feed.feed_url, headers=headers, timeout=FETCH_TIMEOUT_SECONDS)
response = safe_get(feed.feed_url, headers=headers, timeout=FETCH_TIMEOUT_SECONDS)
except SSRFBlocked as exc:
feed.last_fetched_at = timezone.now()
return _record_error(feed, f"Refused to fetch: {exc}")
except requests.RequestException as exc:
feed.last_fetched_at = timezone.now()
return _record_error(feed, f"Request failed: {exc}")
+121
View File
@@ -0,0 +1,121 @@
"""Rejects feed-fetch targets that could turn this server/worker's own
network access into an SSRF vector.
Feed URLs are entirely attacker-controlled (any authenticated user via
"add feed", or an uploaded OPML file — see apps.feeds.opml) and this app
fetches them from the server's own network position. fetch_and_ingest
(apps.feeds.services.fetcher) is the single chokepoint every feed URL
passes through before any HTTP request is made, regardless of which
path created the Feed row, so enforcing this here covers all of them.
This validates the resolved address before connecting and again before
following each redirect hop, denying by default anything that isn't a
globally routable address (ip.is_global) — covering loopback, RFC1918
private ranges, link-local (including the 169.254.169.254 cloud
metadata address), and so on, for both IPv4 and IPv6.
What this does *not* close: the TCP connection isn't pinned to the
specific IP that was validated, so a DNS answer that changes between
the validation lookup and the actual connect (classic DNS rebinding)
could still slip through. Closing that fully needs a custom transport
that connects to the exact validated IP while still verifying TLS
against the original hostname — more machinery than this self-hosted
app's threat model currently justifies; this stops the much more
common case of a feed URL (or a redirect) simply pointing straight at
an internal address.
"""
from __future__ import annotations
import ipaddress
import socket
from dataclasses import dataclass
from urllib.parse import urljoin, urlparse
import requests
from requests.structures import CaseInsensitiveDict
ALLOWED_SCHEMES = {"http", "https"}
MAX_REDIRECTS = 5
MAX_RESPONSE_BYTES = 10 * 1024 * 1024 # 10 MB — generous for an RSS/Atom feed
READ_CHUNK_BYTES = 65536
class SSRFBlocked(Exception):
"""Raised when a fetch target — or one of its redirects — isn't safe
to connect to on a user's behalf."""
@dataclass
class BoundedResponse:
status_code: int
# requests' own CaseInsensitiveDict — server header casing varies
# (ETag/Etag/etag) and fetcher.py looks these up by a fixed-case key.
headers: CaseInsensitiveDict
content: bytes
def validate_fetch_url(url: str) -> None:
"""Raises SSRFBlocked unless `url` has an http(s) scheme, a hostname,
and every address that hostname resolves to is globally routable."""
parsed = urlparse(url)
if parsed.scheme not in ALLOWED_SCHEMES:
raise SSRFBlocked(f"Unsupported URL scheme: {parsed.scheme!r}")
if not parsed.hostname:
raise SSRFBlocked("URL has no hostname")
try:
addrinfo = socket.getaddrinfo(parsed.hostname, None)
except OSError as exc:
raise SSRFBlocked(f"Could not resolve {parsed.hostname!r}: {exc}") from exc
for *_, sockaddr in addrinfo:
ip = ipaddress.ip_address(sockaddr[0])
if not ip.is_global:
raise SSRFBlocked(f"{parsed.hostname!r} resolves to a non-public address: {ip}")
def safe_get(url: str, *, headers: dict, timeout: float) -> BoundedResponse:
"""GETs `url` the way fetch_and_ingest needs it: every redirect hop is
validated and followed manually (never left to requests/urllib3, so
a redirect can't bypass validate_fetch_url), capped at MAX_REDIRECTS,
and the body is read in bounded chunks rather than trusting
Content-Length or buffering an unbounded response.
"""
current_url = url
with requests.Session() as session:
for _ in range(MAX_REDIRECTS + 1):
validate_fetch_url(current_url)
response = session.get(
current_url,
headers=headers,
timeout=timeout,
stream=True,
allow_redirects=False,
)
try:
if response.is_redirect or response.is_permanent_redirect:
location = response.headers.get("Location")
if not location:
raise SSRFBlocked("Redirect response had no Location header")
current_url = urljoin(current_url, location)
continue
return BoundedResponse(
status_code=response.status_code,
headers=response.headers,
content=_read_bounded(response),
)
finally:
response.close()
raise SSRFBlocked(f"Too many redirects (> {MAX_REDIRECTS})")
def _read_bounded(response: requests.Response) -> bytes:
chunks = []
total = 0
for chunk in response.iter_content(chunk_size=READ_CHUNK_BYTES):
total += len(chunk)
if total > MAX_RESPONSE_BYTES:
raise SSRFBlocked(f"Response exceeded {MAX_RESPONSE_BYTES} bytes")
chunks.append(chunk)
return b"".join(chunks)
+138
View File
@@ -0,0 +1,138 @@
import socket
import pytest
import responses
from apps.feeds.services.ssrf_guard import (
MAX_REDIRECTS,
MAX_RESPONSE_BYTES,
SSRFBlocked,
safe_get,
validate_fetch_url,
)
# apps/feeds/conftest.py's autouse _fake_public_dns fixture pins every
# hostname to this address unless a test below overrides it directly.
PUBLIC_IP = "93.184.216.34"
def _patch_dns(monkeypatch, ip: str):
def _fake_getaddrinfo(host, *args, **kwargs):
return [(socket.AF_INET, socket.SOCK_STREAM, 6, "", (ip, 0))]
monkeypatch.setattr(socket, "getaddrinfo", _fake_getaddrinfo)
class TestValidateFetchUrl:
def test_allows_a_public_https_url(self):
validate_fetch_url("https://example.com/feed.xml")
def test_rejects_a_non_http_scheme(self):
with pytest.raises(SSRFBlocked, match="scheme"):
validate_fetch_url("file:///etc/passwd")
def test_rejects_a_url_with_no_hostname(self):
with pytest.raises(SSRFBlocked, match="hostname"):
validate_fetch_url("https:///feed.xml")
@pytest.mark.parametrize(
"ip",
[
"127.0.0.1", # loopback
"10.0.0.5", # RFC1918 private
"192.168.1.1", # RFC1918 private
"169.254.169.254", # link-local / cloud metadata
"0.0.0.0", # unspecified
"::1", # IPv6 loopback
"fc00::1", # IPv6 unique local
],
)
def test_rejects_a_url_resolving_to_a_non_public_address(self, monkeypatch, ip):
_patch_dns(monkeypatch, ip)
with pytest.raises(SSRFBlocked, match="non-public"):
validate_fetch_url("https://internal.example.com/feed.xml")
def test_rejects_a_url_whose_hostname_does_not_resolve(self, monkeypatch):
def _raise(*args, **kwargs):
raise socket.gaierror("nope")
monkeypatch.setattr(socket, "getaddrinfo", _raise)
with pytest.raises(SSRFBlocked, match="Could not resolve"):
validate_fetch_url("https://nowhere.example.com/feed.xml")
class TestSafeGet:
@responses.activate
def test_returns_the_response_body_for_a_normal_feed(self):
responses.add(responses.GET, "https://example.com/feed.xml", body="<rss></rss>")
response = safe_get("https://example.com/feed.xml", headers={}, timeout=5)
assert response.status_code == 200
assert response.content == b"<rss></rss>"
@responses.activate
def test_follows_a_safe_redirect(self):
responses.add(
responses.GET,
"https://example.com/feed.xml",
status=302,
headers={"Location": "https://example.com/final.xml"},
)
responses.add(responses.GET, "https://example.com/final.xml", body="<rss>final</rss>")
response = safe_get("https://example.com/feed.xml", headers={}, timeout=5)
assert response.content == b"<rss>final</rss>"
@responses.activate
def test_blocks_a_redirect_to_a_private_address(self, monkeypatch):
responses.add(
responses.GET,
"https://example.com/feed.xml",
status=302,
headers={"Location": "http://internal.example.com/secret"},
)
real_getaddrinfo = socket.getaddrinfo
def _fake_getaddrinfo(host, *args, **kwargs):
if host == "internal.example.com":
return [(socket.AF_INET, socket.SOCK_STREAM, 6, "", ("127.0.0.1", 0))]
return real_getaddrinfo(host, *args, **kwargs)
monkeypatch.setattr(socket, "getaddrinfo", _fake_getaddrinfo)
with pytest.raises(SSRFBlocked, match="non-public"):
safe_get("https://example.com/feed.xml", headers={}, timeout=5)
@responses.activate
def test_blocks_a_redirect_loop_past_max_redirects(self):
for _ in range(MAX_REDIRECTS + 2):
responses.add(
responses.GET,
"https://example.com/feed.xml",
status=302,
headers={"Location": "https://example.com/feed.xml"},
)
with pytest.raises(SSRFBlocked, match="Too many redirects"):
safe_get("https://example.com/feed.xml", headers={}, timeout=5)
@responses.activate
def test_blocks_a_response_over_the_size_cap(self):
responses.add(
responses.GET,
"https://example.com/feed.xml",
body=b"x" * (MAX_RESPONSE_BYTES + 1),
)
with pytest.raises(SSRFBlocked, match="exceeded"):
safe_get("https://example.com/feed.xml", headers={}, timeout=5)
@responses.activate
def test_preserves_case_insensitive_header_lookup(self):
responses.add(
responses.GET,
"https://example.com/feed.xml",
body="<rss></rss>",
headers={"etag": "abc123"},
)
response = safe_get("https://example.com/feed.xml", headers={}, timeout=5)
assert response.headers.get("ETag") == "abc123"
+24
View File
@@ -164,3 +164,27 @@ def test_saved_filter_only_shows_saved_articles(client, alice, alices_article):
response = client.get(reverse("reader:stream"), {"filter": "starred"})
assert alices_article.title not in response.content.decode()
@pytest.mark.django_db
def test_sidebar_shows_a_tags_total_unread_count(client, alice):
from apps.articles.models import SubscriptionTag, Tag
feed_a = Feed.objects.create(feed_url="https://a.example.com/feed.xml")
feed_b = Feed.objects.create(feed_url="https://b.example.com/feed.xml")
sub_a = Subscription.objects.create(user=alice, feed=feed_a)
sub_b = Subscription.objects.create(user=alice, feed=feed_b)
tag = Tag.objects.create(user=alice, name="work")
SubscriptionTag.objects.create(user=alice, subscription=sub_a, tag=tag)
SubscriptionTag.objects.create(user=alice, subscription=sub_b, tag=tag)
for i in range(3):
Article.objects.create(feed=feed_a, guid=f"a-{i}", title=f"A{i}")
for i in range(4):
Article.objects.create(feed=feed_b, guid=f"b-{i}", title=f"B{i}")
client.login(username="alice", password="pw-alice-123")
response = client.get(reverse("reader:stream"))
# The only "7" on the page — the individual feeds show 3 and 4.
content = response.content.decode()
assert '<span class="unread-count">7</span>' in content
+68 -4
View File
@@ -9,6 +9,7 @@ from __future__ import annotations
import logging
import re
import signal
from django.utils import timezone
@@ -27,6 +28,42 @@ logger = logging.getLogger(__name__)
_TAG_RE = re.compile(r"<[^>]+>")
# Bounds how much CPU/time a single user-authored regex rule can burn
# against one article. re has no built-in match timeout, so this uses
# SIGALRM to interrupt a runaway match (catastrophic backtracking) —
# only effective on the main thread of a process with SIGALRM available,
# which covers gunicorn's sync workers and Celery's (default) prefork
# pool; elsewhere it's skipped and MAX_REGEX_HAYSTACK_CHARS is the only
# bound. reapply_rules_for_user (apps.rules.tasks) also carries a Celery
# task time_limit as a second, coarser backstop.
REGEX_TIMEOUT_SECONDS = 1
MAX_REGEX_HAYSTACK_CHARS = 20_000
class _RegexTimeout(Exception):
pass
def _raise_regex_timeout(signum, frame):
raise _RegexTimeout()
def _regex_search_with_timeout(pattern: str, haystack: str, flags: int):
if not hasattr(signal, "SIGALRM"):
return re.search(pattern, haystack, flags)
try:
previous_handler = signal.signal(signal.SIGALRM, _raise_regex_timeout)
except ValueError:
# Not the main thread (e.g. a threaded dev server) — signal
# timers only work there.
return re.search(pattern, haystack, flags)
signal.setitimer(signal.ITIMER_REAL, REGEX_TIMEOUT_SECONDS)
try:
return re.search(pattern, haystack, flags)
finally:
signal.setitimer(signal.ITIMER_REAL, 0)
signal.signal(signal.SIGALRM, previous_handler)
def _haystack(article: Article, match_field: str) -> str:
if match_field == FilterRule.FIELD_TITLE:
@@ -47,8 +84,12 @@ def rule_matches(rule: FilterRule, article: Article) -> bool:
if rule.match_type == FilterRule.MATCH_REGEX:
flags = 0 if rule.case_sensitive else re.IGNORECASE
haystack = raw_haystack[:MAX_REGEX_HAYSTACK_CHARS]
try:
return re.search(rule.match_value, raw_haystack, flags) is not None
return _regex_search_with_timeout(rule.match_value, haystack, flags) is not None
except _RegexTimeout:
logger.warning("Regex timed out for rule %s", rule.pk)
return False
except re.error as exc:
logger.warning("Invalid regex in rule %s: %s", rule.pk, exc)
return False
@@ -100,13 +141,36 @@ def apply_rule_action(rule: FilterRule, user, article: Article, state: UserArtic
return False
def evaluate_rules_for_article(user, article: Article, rules) -> None:
"""Evaluates an ordered iterable of rules (already scoped/enabled)
against one article for one user, applying actions as they match."""
def rule_in_scope(rule: FilterRule, feed_id, subscription_tag_ids) -> bool:
"""Whether `rule`'s own feed/tag scope (see FilterRule's docstring)
covers an article from `feed_id`, given the user's subscription-tag
ids for that feed.
Centralized here — rather than left for each caller to re-derive via
its own query filters — so ingest-time dispatch and backlog re-apply
can't drift into scoping rules differently (see apps.articles.tasks
and apps.rules.tasks).
"""
if rule.feed_id is not None and rule.feed_id != feed_id:
return False
if rule.tag_id is not None and rule.tag_id not in subscription_tag_ids:
return False
return True
def evaluate_rules_for_article(user, article: Article, rules, subscription_tag_ids=()) -> None:
"""Evaluates an ordered iterable of a user's enabled rules against one
article, applying actions as they match. `rules` doesn't need to be
pre-scoped to this article's feed/tags — out-of-scope rules (per
rule_in_scope) are skipped here, so callers can pass one user's whole
enabled ruleset regardless of which feed(s) the article(s) belong to.
"""
matched_any = False
state = None
for rule in rules:
if not rule_in_scope(rule, article.feed_id, subscription_tag_ids):
continue
if not rule_matches(rule, article):
continue
if state is None:
+24 -6
View File
@@ -1,6 +1,6 @@
from celery import shared_task
from apps.articles.models import Article
from apps.articles.models import Article, SubscriptionTag
from apps.feeds.models import Subscription
from .engine import evaluate_rules_for_article
@@ -12,9 +12,20 @@ def reapply_rules_for_user_backlog(rule_id: int) -> int:
rule, so priority/short-circuit ordering stays correct) against their
unread backlog, scoped by the triggering rule's own feed/tag scope.
Exists as a plain function so it can be called synchronously for
immediate feedback (e.g. from a "Re-apply" view) or wrapped in a task
for background use — see reapply_rules_for_user below.
The *backlog* is narrowed to the triggering rule's own feed/tag scope
(re-applying a feed-scoped rule has no reason to touch other feeds'
articles), but which of the user's *other* rules apply to each of
those articles is still per-article/per-feed — evaluate_rules_for_article
checks that via rule_in_scope, so a feed- or tag-scoped rule for a
different feed can't fire on this backlog just because it happened to
match (see apps.rules.engine.rule_in_scope).
Exists as a plain function, separate from the reapply_rules_for_user
task below, so it's unit-testable without Celery — but every caller
should go through that task (see apps.rules.views.reapply_rule), not
call this directly from a request: a large backlog or a pathological
regex (bounded but not eliminated by apps.rules.engine's timeout
guard) could otherwise tie up a web worker for a long time.
"""
try:
rule = FilterRule.objects.select_related("user").get(pk=rule_id)
@@ -45,13 +56,20 @@ def reapply_rules_for_user_backlog(rule_id: int) -> int:
rules = list(FilterRule.objects.filter(user=user, is_enabled=True).order_by("priority", "id"))
subscription_tag_ids_by_feed: dict[int, set[int]] = {}
for feed_id, tag_id in SubscriptionTag.objects.filter(
user=user, subscription__feed_id__in=feed_ids
).values_list("subscription__feed_id", "tag_id"):
subscription_tag_ids_by_feed.setdefault(feed_id, set()).add(tag_id)
count = 0
for article in backlog:
evaluate_rules_for_article(user, article, rules)
subscription_tag_ids = subscription_tag_ids_by_feed.get(article.feed_id, set())
evaluate_rules_for_article(user, article, rules, subscription_tag_ids)
count += 1
return count
@shared_task
@shared_task(time_limit=120, soft_time_limit=90)
def reapply_rules_for_user(rule_id: int):
return reapply_rules_for_user_backlog(rule_id)
+37 -1
View File
@@ -1,3 +1,5 @@
import time
import pytest
from django.contrib.auth import get_user_model
@@ -10,7 +12,7 @@ from apps.articles.models import (
)
from apps.feeds.models import Feed
from .engine import evaluate_rules_for_article, rule_matches
from .engine import evaluate_rules_for_article, rule_in_scope, rule_matches
from .models import FilterRule
User = get_user_model()
@@ -87,6 +89,22 @@ class TestRuleMatches:
rule = make_rule(user, match_type=FilterRule.MATCH_REGEX, match_value="(unclosed")
assert rule_matches(rule, article) is False
def test_catastrophic_backtracking_regex_times_out_instead_of_hanging(self, user, article):
# (a+)+$ against a long run of a's followed by a non-matching
# character is a textbook ReDoS pattern — left unguarded this
# takes on the order of a minute (confirmed locally); the
# REGEX_TIMEOUT_SECONDS guard in apps.rules.engine must interrupt
# it well before that instead of hanging the caller.
article.title = "a" * 30 + "!"
rule = make_rule(user, match_type=FilterRule.MATCH_REGEX, match_value=r"(a+)+$")
start = time.monotonic()
result = rule_matches(rule, article)
elapsed = time.monotonic() - start
assert result is False
assert elapsed < 5
def test_content_field_strips_html_tags(self, user, article):
rule = make_rule(user, match_field=FilterRule.FIELD_CONTENT, match_value="Lots of details")
assert rule_matches(rule, article) is True
@@ -104,6 +122,24 @@ class TestRuleMatches:
assert rule_matches(rule, article) is True
@pytest.mark.django_db
class TestRuleInScope:
def test_unscoped_rule_applies_to_any_feed(self, user):
rule = make_rule(user)
assert rule_in_scope(rule, feed_id=999, subscription_tag_ids=set()) is True
def test_feed_scoped_rule_only_applies_to_its_own_feed(self, user, feed):
rule = make_rule(user, feed=feed)
assert rule_in_scope(rule, feed_id=feed.id, subscription_tag_ids=set()) is True
assert rule_in_scope(rule, feed_id=feed.id + 1, subscription_tag_ids=set()) is False
def test_tag_scoped_rule_only_applies_when_the_feed_carries_that_tag(self, user):
tag = Tag.objects.create(user=user, name="work")
rule = make_rule(user, tag=tag)
assert rule_in_scope(rule, feed_id=1, subscription_tag_ids={tag.id}) is True
assert rule_in_scope(rule, feed_id=1, subscription_tag_ids=set()) is False
@pytest.mark.django_db
class TestApplyRuleAction:
def test_star_action(self, user, article):
+103 -1
View File
@@ -1,8 +1,17 @@
from unittest.mock import patch
import pytest
from django.contrib.auth import get_user_model
from django.urls import reverse
from apps.articles.models import STARRED_TAG_NAME, Article, ArticleTag, UserArticleState
from apps.articles.models import (
STARRED_TAG_NAME,
Article,
ArticleTag,
SubscriptionTag,
Tag,
UserArticleState,
)
from apps.feeds.models import Feed, Subscription
from .models import FilterRule
@@ -119,6 +128,28 @@ def test_reapply_rule_view_applies_to_unread_backlog(client, alice, feed):
assert is_starred(alice, article) is True
@pytest.mark.django_db
def test_reapply_rule_view_always_queues_through_celery_never_runs_inline(client, alice, feed):
# A large backlog or a pathological regex must not be able to tie up
# the request/response cycle — the view should dispatch a task, not
# call reapply_rules_for_user_backlog directly.
Subscription.objects.create(user=alice, feed=feed)
rule = FilterRule.objects.create(
user=alice,
match_field=FilterRule.FIELD_TITLE,
match_type=FilterRule.MATCH_CONTAINS,
match_value="Python",
action=FilterRule.ACTION_STAR,
)
client.login(username="alice", password="pw-alice-123")
with patch("apps.rules.views.reapply_rules_for_user.delay") as mock_delay:
response = client.post(reverse("rules:reapply_rule", args=[rule.pk]))
assert response.status_code == 302
mock_delay.assert_called_once_with(rule.pk)
@pytest.mark.django_db
def test_reapply_skips_already_read_articles(alice, feed):
article = Article.objects.create(feed=feed, guid="guid-1", title="Python 4 released")
@@ -160,3 +191,74 @@ def test_reapply_respects_feed_scope(alice):
assert is_starred(alice, article_a) is True
assert not UserArticleState.objects.filter(user=alice, article=article_b).exists()
@pytest.mark.django_db
def test_reapply_does_not_apply_a_different_feeds_scoped_rule_to_the_backlog():
# Regression test for a bug where re-applying one feed-scoped rule
# loaded *every* enabled rule and ran it against the whole backlog
# with no scope check — so a same-matching rule scoped to a feed
# the user isn't re-applying would still fire on this backlog.
feed_a = Feed.objects.create(feed_url="https://a.example.com/feed.xml")
feed_b = Feed.objects.create(feed_url="https://b.example.com/feed.xml")
alice = User.objects.create_user("alice", password="pw-alice-123")
Subscription.objects.create(user=alice, feed=feed_a)
Subscription.objects.create(user=alice, feed=feed_b)
article_a = Article.objects.create(feed=feed_a, guid="guid-1", title="Python in A")
triggering_rule = FilterRule.objects.create(
user=alice,
feed=feed_a,
match_field=FilterRule.FIELD_TITLE,
match_type=FilterRule.MATCH_CONTAINS,
match_value="Python",
action=FilterRule.ACTION_STAR,
)
FilterRule.objects.create(
user=alice,
feed=feed_b,
match_field=FilterRule.FIELD_TITLE,
match_type=FilterRule.MATCH_CONTAINS,
match_value="Python",
action=FilterRule.ACTION_TAG,
action_value="feed-b-only",
)
reapply_rules_for_user_backlog(triggering_rule.pk)
assert is_starred(alice, article_a) is True
assert not Tag.objects.filter(user=alice, name="feed-b-only").exists()
@pytest.mark.django_db
def test_reapply_applies_a_tag_scoped_rule_only_to_that_tags_feeds():
feed_a = Feed.objects.create(feed_url="https://a.example.com/feed.xml")
feed_b = Feed.objects.create(feed_url="https://b.example.com/feed.xml")
alice = User.objects.create_user("alice", password="pw-alice-123")
sub_a = Subscription.objects.create(user=alice, feed=feed_a)
Subscription.objects.create(user=alice, feed=feed_b)
tag = Tag.objects.create(user=alice, name="work")
SubscriptionTag.objects.create(user=alice, subscription=sub_a, tag=tag)
article_a = Article.objects.create(feed=feed_a, guid="guid-1", title="Python in A")
article_b = Article.objects.create(feed=feed_b, guid="guid-2", title="Python in B")
unscoped_trigger = FilterRule.objects.create(
user=alice,
match_field=FilterRule.FIELD_TITLE,
match_type=FilterRule.MATCH_CONTAINS,
match_value="unrelated-trigger-value",
action=FilterRule.ACTION_MARK_READ,
)
FilterRule.objects.create(
user=alice,
tag=tag,
match_field=FilterRule.FIELD_TITLE,
match_type=FilterRule.MATCH_CONTAINS,
match_value="Python",
action=FilterRule.ACTION_STAR,
)
reapply_rules_for_user_backlog(unscoped_trigger.pk)
assert is_starred(alice, article_a) is True
assert is_starred(alice, article_b) is False
+6 -5
View File
@@ -5,7 +5,7 @@ from django.views.decorators.http import require_POST
from .forms import FilterRuleForm
from .models import FilterRule
from .tasks import reapply_rules_for_user_backlog
from .tasks import reapply_rules_for_user
@login_required
@@ -48,8 +48,9 @@ def delete_rule(request, rule_id):
@require_POST
def reapply_rule(request, rule_id):
rule = get_object_or_404(FilterRule, user=request.user, pk=rule_id)
# Run synchronously for immediate feedback, same pattern as add_feed —
# a Celery worker isn't required for this to work.
count = reapply_rules_for_user_backlog(rule.pk)
messages.success(request, f"Re-applied your rules to {count} unread article(s).")
# Always queued, never run inline from the request/response cycle —
# a large backlog or a pathological regex could otherwise tie up a
# web worker for a long time (see apps.rules.tasks).
reapply_rules_for_user.delay(rule.pk)
messages.success(request, "Re-applying your rules to your unread backlog…")
return redirect("rules:rule_list")
+10
View File
@@ -299,6 +299,16 @@ form p {
letter-spacing: 0.04em;
}
/* An inner wrapper, not `summary` itself — `summary`'s native disclosure
triangle only renders with its default `display: list-item`, so the
flex layout for the tag name + unread count lives one level in. */
.sidebar-tag-row {
display: flex;
align-items: center;
justify-content: space-between;
gap: 0.4rem;
}
.sidebar-tag-link {
display: inline-block;
padding: 0.15rem 0.4rem;
+46
View File
@@ -0,0 +1,46 @@
(function () {
// Remembers which sidebar <details> sections (one per tag, plus
// "Untagged" — see templates/core/_sidebar.html's data-sidebar-key)
// the user has collapsed, so a full page navigation (this isn't an
// SPA — every click re-renders the sidebar server-side) doesn't keep
// snapping them back open. Per-browser via localStorage, same pattern
// as theme.js/theme-init.js — not synced across devices/accounts.
var STORAGE_KEY = "evection-sidebar-collapsed";
function readCollapsed() {
try {
var raw = localStorage.getItem(STORAGE_KEY);
return raw ? JSON.parse(raw) : [];
} catch (e) {
return [];
}
}
function writeCollapsed(keys) {
try {
localStorage.setItem(STORAGE_KEY, JSON.stringify(keys));
} catch (e) {
/* localStorage unavailable (private mode, etc.) — state just won't persist */
}
}
var collapsed = readCollapsed();
document.querySelectorAll(".sidebar-section[data-sidebar-key]").forEach(function (details) {
var key = details.dataset.sidebarKey;
if (collapsed.indexOf(key) !== -1) {
details.open = false;
}
details.addEventListener("toggle", function () {
var current = readCollapsed();
var index = current.indexOf(key);
if (details.open) {
if (index !== -1) current.splice(index, 1);
} else if (index === -1) {
current.push(key);
}
writeCollapsed(current);
});
});
})();
+4 -1
View File
@@ -53,7 +53,10 @@
{% endif %}
<script src="{% static 'js/stream.js' %}" defer></script>
{% if user.is_authenticated %}<script src="{% static 'js/theme.js' %}" defer></script>{% endif %}
{% if user.is_authenticated %}
<script src="{% static 'js/theme.js' %}" defer></script>
<script src="{% static 'js/sidebar-state.js' %}" defer></script>
{% endif %}
<script src="{% static 'js/register-sw.js' %}" defer></script>
</body>
</html>
+7 -4
View File
@@ -31,10 +31,13 @@
</ul>
{% for entry in sidebar_tag_groups %}
<details class="sidebar-section" open>
<details class="sidebar-section" data-sidebar-key="tag-{{ entry.tag.pk }}" open>
<summary>
<a href="{% url 'reader:stream' %}?tag={{ entry.tag.pk }}"
class="sidebar-tag-link{% if request.GET.tag == entry.tag.pk|stringformat:'s' %} active{% endif %}">{{ entry.tag.name }}</a>
<span class="sidebar-tag-row">
<a href="{% url 'reader:stream' %}?tag={{ entry.tag.pk }}"
class="sidebar-tag-link{% if request.GET.tag == entry.tag.pk|stringformat:'s' %} active{% endif %}">{{ entry.tag.name }}</a>
{% if entry.unread_count %}<span class="unread-count">{{ entry.unread_count }}</span>{% endif %}
</span>
</summary>
{% if entry.subscriptions %}
<ul class="sidebar-item-list">
@@ -54,7 +57,7 @@
{% endfor %}
{% if sidebar_untagged %}
<details class="sidebar-section" open>
<details class="sidebar-section" data-sidebar-key="untagged" open>
<summary>Untagged</summary>
<ul class="sidebar-item-list">
{% for subscription in sidebar_untagged %}