Compare commits

..
Author SHA1 Message Date
ksmithandClaude Sonnet 5 90837b339c Make the Site settings admin page go straight to the edit form
It's a true singleton, not a collection, so the normal changelist (an
"Add" link, a list with 0 or 1 rows, an extra click to get to the one
row) was just friction — and visually inconsistent with every other
app's admin section, which do show an "Add" link. The changelist view
now redirects straight to the single row's change form (creating it
first if this is a fresh install), and "Add" is never offered at all.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-10-31 13:23:47 -06:00
ksmithandClaude Sonnet 5 dc8b9ff83b Make the feed-poll interval site-wide instead of per-feed
Replaces Feed.fetch_interval_minutes with a single admin-editable
SiteSettings.fetch_interval_minutes (apps.core), following the same
singleton-row pattern as accounts.SetupLock. Per-feed error backoff is
unchanged — only the configured baseline interval is now shared across
every feed, loaded once per poll_due_feeds run rather than queried per
feed. EVECTION_MIN_FETCH_INTERVAL_MINUTES keeps working as the floor
underneath it.

SiteSettings is registered in Django admin as a singleton: "Add" is
only offered when no row exists yet, and delete is always refused.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-10-31 12:59:47 -06:00
ksmithandClaude Sonnet 5 a1ea2e5662 Add login rate limiting (S6)
Django's default LoginView had no throttle — an internet-facing
instance was exposed to unlimited password-guessing/credential-
stuffing attempts. apps.accounts.views.LoginView adds two independent,
Redis-cache-backed throttles (shared across all gunicorn workers,
unlike the per-process default LocMemCache): by client address (catches
one attacker spraying many usernames) and by the submitted username
(catches a distributed attack on one specific account). Either blocks
further attempts for a 5-minute window; both are generous enough not
to lock out a user who's just mistyped their own password a few times.

Client address prefers X-Forwarded-For's first hop, falling back to
REMOTE_ADDR — the same reverse-proxy trust boundary config.settings.prod
already relies on for X-Forwarded-Proto (gunicorn is never meant to be
reachable except through that proxy).

CACHES now points at Redis (already a hard install requirement for
Celery) instead of Django's implicit per-process default, since the
whole point is a shared counter across every gunicorn worker; test
settings override it back to LocMemCache so the suite doesn't need a
real Redis instance.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-10-31 12:46:49 -06:00
ksmithandClaude Sonnet 5 9241a54958 Cap entries per poll and stored content size (T4)
ssrf_guard.MAX_RESPONSE_BYTES (S1) already bounds the raw download, but
a feed within that byte cap can still carry an unreasonable number of
entries, or one entry with an unreasonably large body — both cost real
CPU/DB time (nh3 sanitization, an Article row/UPDATE) per entry.
fetch_and_ingest now processes at most MAX_ENTRIES_PER_POLL (200)
entries per poll (logging a warning when truncated) and sanitize_html
bounds both its input (sanitizer cost) and output (what's actually
stored/rendered) length.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-10-31 12:46:40 -06:00
ksmithandClaude Sonnet 5 64f25789f5 Isolate unexpected fetch failures and harden task durability (T2)
fetch_feed had no time limit and no handling for anything beyond the
network/parse errors fetch_and_ingest already catches internally — an
unexpected exception (a parser/sanitizer bug, a DB error) crashed the
task with no error recorded on the Feed, the claim left held for the
full CLAIM_STALE_MINUTES, and no visibility beyond a bare traceback.
fetch_feed now catches that, records it the same way a handled failure
would be (reusing fetcher.record_fetch_error, promoted from a private
helper), clears the claim, and re-raises so Celery still marks the task
FAILED. soft_time_limit/time_limit bound a single fetch that somehow
outruns FETCH_TIMEOUT_SECONDS + ssrf_guard's response cap during
parsing/ingestion.

Also: a worker process killed mid-task (OOM, deploy restart, crash)
previously lost that task silently — Celery's default is to ack a
message the moment a worker *starts* it. CELERY_TASK_ACKS_LATE +
CELERY_TASK_REJECT_ON_WORKER_LOST (config/settings/base.py) make the
message redeliverable instead, and CELERY_WORKER_PREFETCH_MULTIPLIER=1
keeps a worker from grabbing a batch of tasks it could lose all at once.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-10-31 12:46:29 -06:00
ksmithandClaude Sonnet 5 0466ccc334 Fix sidebar disclosure triangle rendering above the tag name
summary's native ::marker triangle is positioned for inline content —
once the tag-name + unread-count row needed to be a flex box (for
v2026.10.31-15's tag unread counts), the marker got pushed onto its
own line above instead of sitting beside it, confirmed via a
screenshot after deploying that release.

Replaces the native marker with a plain ::before triangle (hiding
-webkit-details-marker too) so `summary` itself can safely be the flex
row, with the triangle as a normal flex item that stays inline no
matter what the rest of the row contains. Verified in the browser,
including that collapse/expand still works and the triangle still
rotates on open.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-10-31 12:39:55 -06:00
17 changed files with 534 additions and 39 deletions

No files matched your search

+74
View File
@@ -0,0 +1,74 @@
import pytest
from django.contrib.auth import get_user_model
from django.core.cache import cache
from django.urls import reverse
from .views import (
LOGIN_RATE_LIMIT_MAX_ATTEMPTS_PER_ADDRESS,
LOGIN_RATE_LIMIT_MAX_ATTEMPTS_PER_USERNAME,
)
User = get_user_model()
@pytest.fixture(autouse=True)
def _clear_rate_limit_cache():
cache.clear()
yield
cache.clear()
@pytest.fixture
def alice(db):
return User.objects.create_user("alice", password="pw-alice-123")
def _bad_login(client, username="alice"):
return client.post(reverse("accounts:login"), {"username": username, "password": "wrong"})
@pytest.mark.django_db
def test_a_single_failed_login_still_shows_the_normal_error(client, alice):
response = _bad_login(client)
assert response.status_code == 200
assert b"Please enter a correct" in response.content or response.context["form"].errors
@pytest.mark.django_db
def test_correct_credentials_succeed_before_the_limit_is_hit(client, alice):
for _ in range(3):
_bad_login(client)
response = client.post(
reverse("accounts:login"), {"username": "alice", "password": "pw-alice-123"}
)
assert response.status_code == 302
@pytest.mark.django_db
def test_too_many_failed_attempts_from_one_address_blocks_further_tries(client, alice):
for _ in range(LOGIN_RATE_LIMIT_MAX_ATTEMPTS_PER_ADDRESS):
_bad_login(client)
# Even the correct password is now refused — the throttle runs
# before the form is ever validated.
response = client.post(
reverse("accounts:login"), {"username": "alice", "password": "pw-alice-123"}
)
assert response.status_code == 302
assert response.url == reverse("accounts:login")
assert not response.wsgi_request.user.is_authenticated
@pytest.mark.django_db
def test_too_many_failed_attempts_against_one_username_blocks_that_username(client, alice):
bob = User.objects.create_user("bob", password="pw-bob-123")
for _ in range(LOGIN_RATE_LIMIT_MAX_ATTEMPTS_PER_USERNAME):
_bad_login(client, username="alice")
# Same client address, different username — the per-username
# throttle for "alice" shouldn't affect bob logging in.
response = client.post(reverse("accounts:login"), {"username": "bob", "password": "pw-bob-123"})
assert response.status_code == 302
assert response.wsgi_request.user == bob
+1 -5
View File
@@ -7,11 +7,7 @@ app_name = "accounts"
urlpatterns = [
path("setup/", views.setup, name="setup"),
path(
"login/",
auth_views.LoginView.as_view(template_name="accounts/login.html"),
name="login",
),
path("login/", views.LoginView.as_view(), name="login"),
path("logout/", auth_views.LogoutView.as_view(), name="logout"),
path("users/", views.manage_users, name="manage_users"),
path("users/create/", views.create_user, name="create_user"),
+64
View File
@@ -1,6 +1,8 @@
from django.contrib import messages
from django.contrib.auth import login
from django.contrib.auth.decorators import user_passes_test
from django.contrib.auth.views import LoginView as DjangoLoginView
from django.core.cache import cache
from django.db import IntegrityError, transaction
from django.shortcuts import redirect, render
@@ -10,6 +12,68 @@ from .models import SetupLock, User
is_admin = user_passes_test(lambda u: u.is_authenticated and u.is_staff)
# Two independent throttles, either of which can block an attempt: by
# client address (catches one attacker spraying many usernames) and by
# the submitted username (catches a distributed/low-and-slow attack on
# one specific account from many addresses). Both are generous enough
# not to lock out a user who's just fat-fingered their own password a
# few times.
LOGIN_RATE_LIMIT_WINDOW_SECONDS = 5 * 60
LOGIN_RATE_LIMIT_MAX_ATTEMPTS_PER_ADDRESS = 15
LOGIN_RATE_LIMIT_MAX_ATTEMPTS_PER_USERNAME = 8
def _client_address(request) -> str:
# gunicorn is always run behind a reverse proxy in every deployment
# (see docs/install.md) and its bind address is never meant to be
# reachable from outside that proxy — the same trust boundary
# config.settings.prod already relies on for X-Forwarded-Proto. A
# direct request (e.g. local dev) has no X-Forwarded-For and falls
# back to REMOTE_ADDR.
forwarded_for = request.META.get("HTTP_X_FORWARDED_FOR", "")
if forwarded_for:
return forwarded_for.split(",")[0].strip()
return request.META.get("REMOTE_ADDR", "unknown")
def _attempts_key(kind: str, value: str) -> str:
return f"login-attempts:{kind}:{value}"
def _too_many_attempts(key: str, limit: int) -> bool:
return cache.get(key, 0) >= limit
def _record_failed_attempt(key: str) -> None:
try:
cache.incr(key)
except ValueError:
cache.set(key, 1, LOGIN_RATE_LIMIT_WINDOW_SECONDS)
class LoginView(DjangoLoginView):
"""Wraps Django's own LoginView with the throttle above — everything
else (form, template, redirect behavior) is unchanged."""
template_name = "accounts/login.html"
def post(self, request, *args, **kwargs):
address_key = _attempts_key("addr", _client_address(request))
username_key = _attempts_key("user", request.POST.get("username", "")[:150])
if _too_many_attempts(
address_key, LOGIN_RATE_LIMIT_MAX_ATTEMPTS_PER_ADDRESS
) or _too_many_attempts(username_key, LOGIN_RATE_LIMIT_MAX_ATTEMPTS_PER_USERNAME):
messages.error(
request, "Too many login attempts. Please wait a few minutes and try again."
)
return redirect("accounts:login")
return super().post(request, *args, **kwargs)
def form_invalid(self, form):
_record_failed_attempt(_attempts_key("addr", _client_address(self.request)))
_record_failed_attempt(_attempts_key("user", self.request.POST.get("username", "")[:150]))
return super().form_invalid(form)
def setup(request):
if User.objects.exists():
+28 -1
View File
@@ -1 +1,28 @@
# Register your models here.
from django.contrib import admin
from django.shortcuts import redirect
from django.urls import reverse
from .models import SiteSettings
@admin.register(SiteSettings)
class SiteSettingsAdmin(admin.ModelAdmin):
"""A true singleton, not a collection — browsing a changelist to pick
the one row and click into it is an extra, pointless step, so
"Core > Site settings" in the sidebar goes straight to the (only)
change form instead, creating the row first if this is a fresh
install. No changelist/add/delete views exist at all here.
"""
fields = ["fetch_interval_minutes"]
def has_add_permission(self, request):
return False
def has_delete_permission(self, request, obj=None):
return False
def changelist_view(self, request, extra_context=None):
settings_row = SiteSettings.load()
url = reverse("admin:core_sitesettings_change", args=[settings_row.pk])
return redirect(url)
+33
View File
@@ -0,0 +1,33 @@
# Generated by Django 5.2.18 on 2026-10-31 18:56
from django.db import migrations, models
class Migration(migrations.Migration):
initial = True
dependencies = []
operations = [
migrations.CreateModel(
name="SiteSettings",
fields=[
(
"id",
models.PositiveSmallIntegerField(default=1, primary_key=True, serialize=False),
),
(
"fetch_interval_minutes",
models.PositiveIntegerField(
default=60,
help_text="How often, in minutes, every feed is polled for new articles. Can't be set lower than EVECTION_MIN_FETCH_INTERVAL_MINUTES (see .env).",
),
),
],
options={
"verbose_name": "Site settings",
"verbose_name_plural": "Site settings",
},
),
]
+35 -1
View File
@@ -1 +1,35 @@
# Create your models here.
from django.db import models
class SiteSettings(models.Model):
"""Singleton row (pk=1, same pattern as accounts.SetupLock) holding
instance-wide configuration an operator reasonably wants to change
without touching the server or .env — editable via Django admin.
Currently just the feed-poll interval, which used to be a per-feed
field; it's site-wide now (see apps.feeds.tasks.poll_due_feeds).
"""
id = models.PositiveSmallIntegerField(primary_key=True, default=1)
fetch_interval_minutes = models.PositiveIntegerField(
default=60,
help_text=(
"How often, in minutes, every feed is polled for new articles. "
"Can't be set lower than EVECTION_MIN_FETCH_INTERVAL_MINUTES (see .env)."
),
)
class Meta:
verbose_name = "Site settings"
verbose_name_plural = "Site settings"
def __str__(self):
return "Site settings"
@classmethod
def load(cls):
"""Returns the singleton row, creating it with defaults on first
use so callers never need to special-case "no row exists yet"
(e.g. a fresh install whose first poll_due_feeds run hasn't
created it, and whose admin hasn't visited the settings page)."""
obj, _ = cls.objects.get_or_create(pk=1)
return obj
+84
View File
@@ -0,0 +1,84 @@
from datetime import timedelta
from unittest.mock import patch
import pytest
from django.contrib.auth import get_user_model
from django.urls import reverse
from django.utils import timezone
from apps.feeds.models import Feed
from apps.feeds.tasks import poll_due_feeds
from .models import SiteSettings
User = get_user_model()
@pytest.mark.django_db
def test_load_creates_the_singleton_row_with_defaults():
assert not SiteSettings.objects.exists()
settings_row = SiteSettings.load()
assert settings_row.fetch_interval_minutes == 60
assert SiteSettings.objects.count() == 1
@pytest.mark.django_db
def test_load_returns_the_existing_row_without_creating_a_second_one():
SiteSettings.objects.create(fetch_interval_minutes=30)
settings_row = SiteSettings.load()
assert settings_row.fetch_interval_minutes == 30
assert SiteSettings.objects.count() == 1
@pytest.mark.django_db
def test_poll_due_feeds_uses_the_site_wide_interval_for_every_feed():
SiteSettings.objects.create(fetch_interval_minutes=120)
due = Feed.objects.create(
feed_url="https://a.example.com/feed.xml",
last_fetched_at=timezone.now() - timedelta(minutes=150),
)
not_due = Feed.objects.create(
feed_url="https://b.example.com/feed.xml",
last_fetched_at=timezone.now() - timedelta(minutes=90),
)
with patch("apps.feeds.tasks.fetch_feed.delay") as mock_delay:
poll_due_feeds()
dispatched_ids = {call.args[0] for call in mock_delay.call_args_list}
assert dispatched_ids == {due.id}
assert not_due.id not in dispatched_ids
@pytest.mark.django_db
def test_admin_changelist_redirects_straight_to_the_change_form(client):
assert not SiteSettings.objects.exists()
User.objects.create_superuser("admin", password="pw-admin-123")
client.login(username="admin", password="pw-admin-123")
changelist_url = reverse("admin:core_sitesettings_changelist")
response = client.get(changelist_url)
settings_row = SiteSettings.objects.get() # created by the redirect itself
assert response.status_code == 302
assert response.url == reverse("admin:core_sitesettings_change", args=[settings_row.pk])
@pytest.mark.django_db
def test_admin_never_offers_add_or_delete(client):
# ModelAdmin.has_add_permission/has_delete_permission returning False
# makes Django raise PermissionDenied rather than build a 403 response
# directly — the test client's default behavior re-raises that for
# debugging convenience instead of returning it as a response.
client.raise_request_exception = False
User.objects.create_superuser("admin", password="pw-admin-123")
client.login(username="admin", password="pw-admin-123")
response = client.get(reverse("admin:core_sitesettings_add"))
assert response.status_code == 403
settings_row = SiteSettings.load()
delete_url = reverse("admin:core_sitesettings_delete", args=[settings_row.pk])
response = client.get(delete_url)
assert response.status_code == 403
@@ -0,0 +1,17 @@
# Generated by Django 5.2.18 on 2026-10-31 18:56
from django.db import migrations
class Migration(migrations.Migration):
dependencies = [
("feeds", "0003_feed_claimed_at"),
]
operations = [
migrations.RemoveField(
model_name="feed",
name="fetch_interval_minutes",
),
]
+4 -2
View File
@@ -35,8 +35,10 @@ class Feed(models.Model):
)
last_error_message = models.TextField(blank=True)
# Instance-wide, not a per-user preference — see class docstring.
fetch_interval_minutes = models.PositiveIntegerField(default=60)
# The poll interval itself is site-wide now (apps.core.models.
# SiteSettings), not per-feed — only the error-driven backoff on top
# of it still varies per feed (see apps.feeds.services.fetcher.
# effective_interval_minutes).
fetch_error_count = models.PositiveIntegerField(default=0)
is_active = models.BooleanField(default=True)
+37 -13
View File
@@ -31,6 +31,15 @@ FETCH_TIMEOUT_SECONDS = 15
MAX_BACKOFF_MINUTES = 24 * 60
USER_AGENT = "Evection/0.1 (+https://github.com/; self-hosted RSS reader)"
# ssrf_guard.MAX_RESPONSE_BYTES already bounds the raw download; these
# bound what a single poll does *with* that response — a feed within the
# byte cap can still carry an unreasonable number of entries, or one
# entry with an unreasonably large body, and both cost real CPU/DB time
# (nh3 sanitization, an Article row/UPDATE) per entry processed.
MAX_ENTRIES_PER_POLL = 200
MAX_CONTENT_HTML_INPUT_CHARS = 200_000 # bounds sanitizer cost, not just storage
MAX_CONTENT_HTML_CHARS = 50_000 # bounds what's actually stored/rendered
ALLOWED_TAGS = {
"a",
"p",
@@ -83,7 +92,9 @@ class FetchResult:
def sanitize_html(raw_html: str) -> str:
if not raw_html:
return ""
return nh3.clean(raw_html, tags=ALLOWED_TAGS, attributes=ALLOWED_ATTRIBUTES, link_rel=None)
bounded = raw_html[:MAX_CONTENT_HTML_INPUT_CHARS]
cleaned = nh3.clean(bounded, tags=ALLOWED_TAGS, attributes=ALLOWED_ATTRIBUTES, link_rel=None)
return cleaned[:MAX_CONTENT_HTML_CHARS]
def _content_hash(title: str, content_html: str) -> str:
@@ -113,14 +124,17 @@ def _entry_content_html(entry) -> str:
return getattr(entry, "summary", "") or ""
def effective_interval_minutes(feed: Feed) -> int:
"""The actual polling interval to use, applying exponential backoff on
top of the admin-configured ``fetch_interval_minutes`` for feeds that
are currently erroring. ``fetch_interval_minutes`` itself is never
mutated by the fetcher — it's the admin's chosen baseline.
def effective_interval_minutes(feed: Feed, configured_interval_minutes: int) -> int:
"""The actual polling interval to use for `feed`, applying exponential
backoff on top of the site-wide `configured_interval_minutes` (see
apps.core.models.SiteSettings) for a feed that's currently erroring.
Takes the configured value as a parameter rather than querying
SiteSettings itself, since this runs once per feed (apps.feeds.tasks
calls it from a list comprehension over every active feed) and the
site-wide value doesn't vary by feed.
"""
floor = getattr(settings, "EVECTION_MIN_FETCH_INTERVAL_MINUTES", 15)
base = max(feed.fetch_interval_minutes, floor)
base = max(configured_interval_minutes, floor)
if feed.fetch_error_count <= 0:
return base
backoff = base * (2**feed.fetch_error_count)
@@ -144,10 +158,10 @@ def fetch_and_ingest(feed: Feed) -> FetchResult:
response = safe_get(feed.feed_url, headers=headers, timeout=FETCH_TIMEOUT_SECONDS)
except SSRFBlocked as exc:
feed.last_fetched_at = timezone.now()
return _record_error(feed, f"Refused to fetch: {exc}")
return record_fetch_error(feed, f"Refused to fetch: {exc}")
except requests.RequestException as exc:
feed.last_fetched_at = timezone.now()
return _record_error(feed, f"Request failed: {exc}")
return record_fetch_error(feed, f"Request failed: {exc}")
feed.last_fetched_at = timezone.now()
@@ -159,12 +173,12 @@ def fetch_and_ingest(feed: Feed) -> FetchResult:
)
if response.status_code >= 400:
return _record_error(feed, f"HTTP {response.status_code}")
return record_fetch_error(feed, f"HTTP {response.status_code}")
parsed = feedparser.parse(response.content)
if parsed.bozo and not parsed.entries:
return _record_error(feed, f"Unparseable feed: {parsed.get('bozo_exception')}")
return record_fetch_error(feed, f"Unparseable feed: {parsed.get('bozo_exception')}")
if parsed.bozo:
logger.warning(
@@ -186,7 +200,17 @@ def fetch_and_ingest(feed: Feed) -> FetchResult:
new_ids: list[int] = []
updated_ids: list[int] = []
for entry in parsed.entries:
entries = parsed.entries
if len(entries) > MAX_ENTRIES_PER_POLL:
logger.warning(
"Feed %s: %d entries in one poll, only processing the first %d",
feed.feed_url,
len(entries),
MAX_ENTRIES_PER_POLL,
)
entries = entries[:MAX_ENTRIES_PER_POLL]
for entry in entries:
guid = _entry_guid(entry)
title = (getattr(entry, "title", "") or "")[:1000]
content_html = sanitize_html(_entry_content_html(entry))
@@ -220,7 +244,7 @@ def fetch_and_ingest(feed: Feed) -> FetchResult:
)
def _record_error(feed: Feed, message: str) -> FetchResult:
def record_fetch_error(feed: Feed, message: str) -> FetchResult:
feed.last_fetch_status = Feed.STATUS_ERROR
feed.last_error_message = message[:2000]
feed.fetch_error_count += 1
+39 -6
View File
@@ -8,10 +8,11 @@ from django.db.models import Q
from django.utils import timezone
from apps.articles.tasks import apply_filter_rules_for_feed
from apps.core.models import SiteSettings
from .models import Feed
from .opml import import_opml
from .services.fetcher import effective_interval_minutes, fetch_and_ingest
from .services.fetcher import effective_interval_minutes, fetch_and_ingest, record_fetch_error
logger = logging.getLogger(__name__)
@@ -32,12 +33,15 @@ def poll_due_feeds():
"""
now = timezone.now()
claim_stale_before = now - timedelta(minutes=CLAIM_STALE_MINUTES)
# Loaded once per call, not per feed — the interval is site-wide
# (apps.core.models.SiteSettings), so every feed below shares it.
configured_interval = SiteSettings.load().fetch_interval_minutes
due_feed_ids = [
feed.id
for feed in Feed.objects.filter(is_active=True).only(
"id", "last_fetched_at", "fetch_interval_minutes", "fetch_error_count"
"id", "last_fetched_at", "fetch_error_count"
)
if _is_due(feed)
if _is_due(feed, configured_interval)
]
claimed_ids = []
for feed_id in due_feed_ids:
@@ -57,21 +61,50 @@ def poll_due_feeds():
return len(claimed_ids)
def _is_due(feed: Feed) -> bool:
def _is_due(feed: Feed, configured_interval_minutes: int) -> bool:
if feed.last_fetched_at is None:
return True
interval = effective_interval_minutes(feed)
interval = effective_interval_minutes(feed, configured_interval_minutes)
return feed.last_fetched_at <= timezone.now() - timedelta(minutes=interval)
@shared_task
# Bounds a single feed's fetch+parse+ingest, independent of
# FETCH_TIMEOUT_SECONDS (apps.feeds.services.fetcher) which only covers
# the HTTP request itself: parsing/sanitizing/writing a large-but-under-
# the-byte-cap feed could still run long. soft_time_limit lets the
# except-Exception handler below record a normal feed error (and clear
# the claim) before the hard time_limit kills the worker process outright.
FETCH_FEED_SOFT_TIME_LIMIT = 60
FETCH_FEED_TIME_LIMIT = 90
@shared_task(time_limit=FETCH_FEED_TIME_LIMIT, soft_time_limit=FETCH_FEED_SOFT_TIME_LIMIT)
def fetch_feed(feed_id: int):
try:
feed = Feed.objects.get(pk=feed_id, is_active=True)
except Feed.DoesNotExist:
return None
try:
result = fetch_and_ingest(feed)
except Exception:
# fetch_and_ingest already handles the expected failure modes
# (network errors, SSRF-blocked targets, unparseable feeds) by
# returning an error FetchResult rather than raising — anything
# that reaches here is unexpected (a parser/sanitizer bug, a DB
# error, this task's own soft time limit), so it's recorded the
# same way a handled failure would be rather than silently
# leaving the feed's status stale, the claim held for the full
# CLAIM_STALE_MINUTES, and no record of what happened. Re-raised
# afterward so Celery still marks the task FAILED (visible in
# worker logs/monitoring) instead of swallowing it.
logger.exception("Feed %s: unexpected error during fetch", feed.feed_url)
record_fetch_error(feed, "Unexpected error — see worker logs")
feed.last_fetched_at = timezone.now()
feed.claimed_at = None
feed.save()
raise
feed.claimed_at = None
feed.save()
+34 -2
View File
@@ -5,7 +5,13 @@ from django.utils import timezone
from apps.articles.models import Article, UserArticleState
from apps.feeds.models import Feed
from apps.feeds.services.fetcher import effective_interval_minutes, fetch_and_ingest
from apps.feeds.services.fetcher import (
MAX_CONTENT_HTML_CHARS,
MAX_ENTRIES_PER_POLL,
effective_interval_minutes,
fetch_and_ingest,
sanitize_html,
)
FEED_URL = "https://example.com/feed.xml"
@@ -130,7 +136,7 @@ def test_unparseable_feed_records_error_and_backs_off(feed):
assert result.status == Feed.STATUS_ERROR
assert feed.fetch_error_count == 1
assert effective_interval_minutes(feed) > feed.fetch_interval_minutes
assert effective_interval_minutes(feed, 60) > 60
@pytest.mark.django_db
@@ -196,3 +202,29 @@ def test_refetching_updated_content_does_not_reset_user_state():
state.refresh_from_db()
assert state.is_read is True
assert ArticleTag.objects.filter(user=user, article=article, tag=starred_tag).exists()
def test_sanitize_html_caps_stored_output_length():
huge = "<p>" + ("a" * (MAX_CONTENT_HTML_CHARS + 10_000)) + "</p>"
cleaned = sanitize_html(huge)
assert len(cleaned) <= MAX_CONTENT_HTML_CHARS
@pytest.mark.django_db
@responses.activate
def test_fetch_only_processes_the_first_max_entries_per_poll(feed):
items = "".join(
f"<item><title>Item {i}</title><guid>guid-{i}</guid></item>"
for i in range(MAX_ENTRIES_PER_POLL + 5)
)
xml = f"""<?xml version="1.0"?>
<rss version="2.0"><channel><title>Big feed</title>{items}</channel></rss>
"""
responses.add(responses.GET, FEED_URL, body=xml, status=200)
result = fetch_and_ingest(feed)
feed.save()
assert len(result.new_article_ids) == MAX_ENTRIES_PER_POLL
assert Article.objects.filter(feed=feed).count() == MAX_ENTRIES_PER_POLL
assert not Article.objects.filter(feed=feed, guid=f"guid-{MAX_ENTRIES_PER_POLL}").exists()
+19 -5
View File
@@ -16,17 +16,14 @@ def test_poll_due_feeds_dispatches_only_due_feeds():
recently_fetched = Feed.objects.create(
feed_url="https://b.example.com/feed.xml",
last_fetched_at=timezone.now(),
fetch_interval_minutes=60,
)
overdue = Feed.objects.create(
feed_url="https://c.example.com/feed.xml",
last_fetched_at=timezone.now() - timedelta(hours=2),
fetch_interval_minutes=60,
)
inactive_but_overdue = Feed.objects.create(
feed_url="https://d.example.com/feed.xml",
last_fetched_at=timezone.now() - timedelta(hours=2),
fetch_interval_minutes=60,
is_active=False,
)
@@ -48,7 +45,6 @@ def test_poll_due_feeds_does_not_reclaim_a_feed_it_just_claimed():
overdue = Feed.objects.create(
feed_url="https://c.example.com/feed.xml",
last_fetched_at=timezone.now() - timedelta(hours=2),
fetch_interval_minutes=60,
)
with patch("apps.feeds.tasks.fetch_feed.delay") as mock_delay:
@@ -69,7 +65,6 @@ def test_poll_due_feeds_reclaims_a_stale_claim():
overdue = Feed.objects.create(
feed_url="https://c.example.com/feed.xml",
last_fetched_at=timezone.now() - timedelta(hours=2),
fetch_interval_minutes=60,
claimed_at=timezone.now() - timedelta(minutes=CLAIM_STALE_MINUTES + 1),
)
@@ -127,3 +122,22 @@ def test_fetch_feed_task_skips_inactive_feed():
@pytest.mark.django_db
def test_fetch_feed_task_skips_missing_feed():
assert fetch_feed(999999) is None
@pytest.mark.django_db
def test_fetch_feed_task_records_error_and_clears_claim_on_unexpected_exception():
feed = Feed.objects.create(
feed_url="https://example.com/feed.xml",
claimed_at=timezone.now(),
fetch_error_count=0,
)
with patch("apps.feeds.tasks.fetch_and_ingest", side_effect=RuntimeError("boom")):
with pytest.raises(RuntimeError, match="boom"):
fetch_feed(feed.id)
feed.refresh_from_db()
assert feed.last_fetch_status == Feed.STATUS_ERROR
assert feed.fetch_error_count == 1
assert feed.claimed_at is None
assert feed.last_fetched_at is not None
+25
View File
@@ -106,9 +106,34 @@ EVECTION_SIGNUP_ENABLED = env.bool("EVECTION_SIGNUP_ENABLED", default=False)
EVECTION_MIN_FETCH_INTERVAL_MINUTES = env.int("EVECTION_MIN_FETCH_INTERVAL_MINUTES", default=15)
EVECTION_PWA_CACHE_VERSION = env.str("EVECTION_PWA_CACHE_VERSION", default="v1")
# Redis-backed rather than Django's default per-process LocMemCache: login
# rate limiting (apps.accounts.views) must share counters across every
# gunicorn worker, not just whichever one happened to handle a given
# request. Redis is already a hard install requirement for Celery, so
# this adds no new infrastructure — just a distinct key prefix so cache
# keys can't collide with anything else sharing that Redis instance.
CACHES = {
"default": {
"BACKEND": "django.core.cache.backends.redis.RedisCache",
"LOCATION": env.str("REDIS_URL", default="redis://localhost:6379/0"),
"KEY_PREFIX": "evection-cache",
}
}
# Celery
CELERY_BROKER_URL = env.str("REDIS_URL", default="redis://localhost:6379/0")
CELERY_RESULT_BACKEND = None
CELERY_TASK_SERIALIZER = "json"
CELERY_ACCEPT_CONTENT = ["json"]
CELERY_TIMEZONE = TIME_ZONE
# Without these, a worker process killed mid-task (OOM, deploy restart,
# crash) silently loses that task: the default is to ack a message the
# moment a worker *starts* it, not when it finishes. acks_late + reject-
# on-worker-lost means the message is only consumed on completion (success
# or a handled failure) and is redelivered to another worker otherwise.
# prefetch_multiplier=1 keeps a worker from grabbing a batch of tasks it
# could similarly lose all at once.
CELERY_TASK_ACKS_LATE = True
CELERY_TASK_REJECT_ON_WORKER_LOST = True
CELERY_WORKER_PREFETCH_MULTIPLIER = 1
+7
View File
@@ -5,3 +5,10 @@ CELERY_TASK_ALWAYS_EAGER = True
CELERY_TASK_EAGER_PROPAGATES = True
PASSWORD_HASHERS = ["django.contrib.auth.hashers.MD5PasswordHasher"]
# base.py's CACHES needs a real Redis instance — tests must not depend on
# one being available, and a plain per-process cache is also what keeps
# login-rate-limit tests isolated from each other (LocMemCache's default
# location is per-process but shared across tests in that process; see
# apps.accounts.test_rate_limit's cache.clear() fixture).
CACHES = {"default": {"BACKEND": "django.core.cache.backends.locmem.LocMemCache"}}
+5 -1
View File
@@ -66,7 +66,7 @@ and 4 below first — prerequisites and what `.env` controls.
- Python 3.11 or 3.12, with its `venv` module (Debian/Ubuntu split this into a
separate `python3.X-venv` package — see below)
- Redis (Celery's broker)
- Redis (Celery's broker, and the app's cache — e.g. login rate limiting)
- **Postgres only**: a PostgreSQL server (12+) and its client headers if you're not
using the bundled `psycopg[binary]` wheel (most platforms are covered by the wheel,
so this is rarely needed)
@@ -143,6 +143,10 @@ At minimum, set the following in `.env`:
- `EVECTION_GUNICORN_BIND` — the `address:port` gunicorn listens on; see
"Reverse proxy" below for how to pick this
How often every feed is polled is a site-wide setting (Django admin → Site
settings), not a `.env` value — `EVECTION_MIN_FETCH_INTERVAL_MINUTES` only sets
a floor under it, so it can't be configured lower than that without editing `.env`.
Evection reads `config.settings.prod` when run via the commands below (see each
systemd unit's `DJANGO_SETTINGS_MODULE`). That module turns on HTTPS-only cookies,
HSTS, and a Content-Security-Policy header on top of the base settings, and
+28 -3
View File
@@ -290,23 +290,48 @@ form p {
padding: 0 0.5rem;
}
/* `summary`'s native disclosure triangle is a ::marker, positioned for
list-item/inline content — once the tag-name + unread-count row below
needed to be a flex box (block-level), the marker got pushed onto its
own line above instead of sitting beside it. Replacing it with a
plain ::before triangle lets `summary` itself be a flex row with that
triangle as a normal flex item, so it stays inline no matter what the
rest of the row contains. */
.sidebar-section summary {
display: flex;
align-items: center;
gap: 0.4rem;
padding: 0.4rem 0.75rem;
cursor: pointer;
color: var(--text-dim);
font-size: 0.75rem;
text-transform: uppercase;
letter-spacing: 0.04em;
list-style: none;
}
.sidebar-section summary::-webkit-details-marker {
display: none;
}
.sidebar-section summary::before {
content: "▶";
flex: 0 0 auto;
font-size: 0.6rem;
transition: transform 0.1s ease;
}
.sidebar-section[open] > summary::before {
transform: rotate(90deg);
}
/* An inner wrapper, not `summary` itself — `summary`'s native disclosure
triangle only renders with its default `display: list-item`, so the
flex layout for the tag name + unread count lives one level in. */
.sidebar-tag-row {
display: flex;
align-items: center;
justify-content: space-between;
gap: 0.4rem;
flex: 1 1 auto;
min-width: 0;
}
.sidebar-tag-link {