Compare commits

...
Author SHA1 Message Date
ksmithandClaude Sonnet 5 dc8b9ff83b Make the feed-poll interval site-wide instead of per-feed
Replaces Feed.fetch_interval_minutes with a single admin-editable
SiteSettings.fetch_interval_minutes (apps.core), following the same
singleton-row pattern as accounts.SetupLock. Per-feed error backoff is
unchanged — only the configured baseline interval is now shared across
every feed, loaded once per poll_due_feeds run rather than queried per
feed. EVECTION_MIN_FETCH_INTERVAL_MINUTES keeps working as the floor
underneath it.

SiteSettings is registered in Django admin as a singleton: "Add" is
only offered when no row exists yet, and delete is always refused.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-10-31 12:59:47 -06:00
ksmithandClaude Sonnet 5 a1ea2e5662 Add login rate limiting (S6)
Django's default LoginView had no throttle — an internet-facing
instance was exposed to unlimited password-guessing/credential-
stuffing attempts. apps.accounts.views.LoginView adds two independent,
Redis-cache-backed throttles (shared across all gunicorn workers,
unlike the per-process default LocMemCache): by client address (catches
one attacker spraying many usernames) and by the submitted username
(catches a distributed attack on one specific account). Either blocks
further attempts for a 5-minute window; both are generous enough not
to lock out a user who's just mistyped their own password a few times.

Client address prefers X-Forwarded-For's first hop, falling back to
REMOTE_ADDR — the same reverse-proxy trust boundary config.settings.prod
already relies on for X-Forwarded-Proto (gunicorn is never meant to be
reachable except through that proxy).

CACHES now points at Redis (already a hard install requirement for
Celery) instead of Django's implicit per-process default, since the
whole point is a shared counter across every gunicorn worker; test
settings override it back to LocMemCache so the suite doesn't need a
real Redis instance.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-10-31 12:46:49 -06:00
ksmithandClaude Sonnet 5 9241a54958 Cap entries per poll and stored content size (T4)
ssrf_guard.MAX_RESPONSE_BYTES (S1) already bounds the raw download, but
a feed within that byte cap can still carry an unreasonable number of
entries, or one entry with an unreasonably large body — both cost real
CPU/DB time (nh3 sanitization, an Article row/UPDATE) per entry.
fetch_and_ingest now processes at most MAX_ENTRIES_PER_POLL (200)
entries per poll (logging a warning when truncated) and sanitize_html
bounds both its input (sanitizer cost) and output (what's actually
stored/rendered) length.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-10-31 12:46:40 -06:00
ksmithandClaude Sonnet 5 64f25789f5 Isolate unexpected fetch failures and harden task durability (T2)
fetch_feed had no time limit and no handling for anything beyond the
network/parse errors fetch_and_ingest already catches internally — an
unexpected exception (a parser/sanitizer bug, a DB error) crashed the
task with no error recorded on the Feed, the claim left held for the
full CLAIM_STALE_MINUTES, and no visibility beyond a bare traceback.
fetch_feed now catches that, records it the same way a handled failure
would be (reusing fetcher.record_fetch_error, promoted from a private
helper), clears the claim, and re-raises so Celery still marks the task
FAILED. soft_time_limit/time_limit bound a single fetch that somehow
outruns FETCH_TIMEOUT_SECONDS + ssrf_guard's response cap during
parsing/ingestion.

Also: a worker process killed mid-task (OOM, deploy restart, crash)
previously lost that task silently — Celery's default is to ack a
message the moment a worker *starts* it. CELERY_TASK_ACKS_LATE +
CELERY_TASK_REJECT_ON_WORKER_LOST (config/settings/base.py) make the
message redeliverable instead, and CELERY_WORKER_PREFETCH_MULTIPLIER=1
keeps a worker from grabbing a batch of tasks it could lose all at once.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-10-31 12:46:29 -06:00
16 changed files with 486 additions and 36 deletions

No files matched your search

+74
View File
@@ -0,0 +1,74 @@
import pytest
from django.contrib.auth import get_user_model
from django.core.cache import cache
from django.urls import reverse
from .views import (
LOGIN_RATE_LIMIT_MAX_ATTEMPTS_PER_ADDRESS,
LOGIN_RATE_LIMIT_MAX_ATTEMPTS_PER_USERNAME,
)
User = get_user_model()
@pytest.fixture(autouse=True)
def _clear_rate_limit_cache():
cache.clear()
yield
cache.clear()
@pytest.fixture
def alice(db):
return User.objects.create_user("alice", password="pw-alice-123")
def _bad_login(client, username="alice"):
return client.post(reverse("accounts:login"), {"username": username, "password": "wrong"})
@pytest.mark.django_db
def test_a_single_failed_login_still_shows_the_normal_error(client, alice):
response = _bad_login(client)
assert response.status_code == 200
assert b"Please enter a correct" in response.content or response.context["form"].errors
@pytest.mark.django_db
def test_correct_credentials_succeed_before_the_limit_is_hit(client, alice):
for _ in range(3):
_bad_login(client)
response = client.post(
reverse("accounts:login"), {"username": "alice", "password": "pw-alice-123"}
)
assert response.status_code == 302
@pytest.mark.django_db
def test_too_many_failed_attempts_from_one_address_blocks_further_tries(client, alice):
for _ in range(LOGIN_RATE_LIMIT_MAX_ATTEMPTS_PER_ADDRESS):
_bad_login(client)
# Even the correct password is now refused — the throttle runs
# before the form is ever validated.
response = client.post(
reverse("accounts:login"), {"username": "alice", "password": "pw-alice-123"}
)
assert response.status_code == 302
assert response.url == reverse("accounts:login")
assert not response.wsgi_request.user.is_authenticated
@pytest.mark.django_db
def test_too_many_failed_attempts_against_one_username_blocks_that_username(client, alice):
bob = User.objects.create_user("bob", password="pw-bob-123")
for _ in range(LOGIN_RATE_LIMIT_MAX_ATTEMPTS_PER_USERNAME):
_bad_login(client, username="alice")
# Same client address, different username — the per-username
# throttle for "alice" shouldn't affect bob logging in.
response = client.post(reverse("accounts:login"), {"username": "bob", "password": "pw-bob-123"})
assert response.status_code == 302
assert response.wsgi_request.user == bob
+1 -5
View File
@@ -7,11 +7,7 @@ app_name = "accounts"
urlpatterns = [ urlpatterns = [
path("setup/", views.setup, name="setup"), path("setup/", views.setup, name="setup"),
path( path("login/", views.LoginView.as_view(), name="login"),
"login/",
auth_views.LoginView.as_view(template_name="accounts/login.html"),
name="login",
),
path("logout/", auth_views.LogoutView.as_view(), name="logout"), path("logout/", auth_views.LogoutView.as_view(), name="logout"),
path("users/", views.manage_users, name="manage_users"), path("users/", views.manage_users, name="manage_users"),
path("users/create/", views.create_user, name="create_user"), path("users/create/", views.create_user, name="create_user"),
+64
View File
@@ -1,6 +1,8 @@
from django.contrib import messages from django.contrib import messages
from django.contrib.auth import login from django.contrib.auth import login
from django.contrib.auth.decorators import user_passes_test from django.contrib.auth.decorators import user_passes_test
from django.contrib.auth.views import LoginView as DjangoLoginView
from django.core.cache import cache
from django.db import IntegrityError, transaction from django.db import IntegrityError, transaction
from django.shortcuts import redirect, render from django.shortcuts import redirect, render
@@ -10,6 +12,68 @@ from .models import SetupLock, User
is_admin = user_passes_test(lambda u: u.is_authenticated and u.is_staff) is_admin = user_passes_test(lambda u: u.is_authenticated and u.is_staff)
# Two independent throttles, either of which can block an attempt: by
# client address (catches one attacker spraying many usernames) and by
# the submitted username (catches a distributed/low-and-slow attack on
# one specific account from many addresses). Both are generous enough
# not to lock out a user who's just fat-fingered their own password a
# few times.
LOGIN_RATE_LIMIT_WINDOW_SECONDS = 5 * 60
LOGIN_RATE_LIMIT_MAX_ATTEMPTS_PER_ADDRESS = 15
LOGIN_RATE_LIMIT_MAX_ATTEMPTS_PER_USERNAME = 8
def _client_address(request) -> str:
# gunicorn is always run behind a reverse proxy in every deployment
# (see docs/install.md) and its bind address is never meant to be
# reachable from outside that proxy — the same trust boundary
# config.settings.prod already relies on for X-Forwarded-Proto. A
# direct request (e.g. local dev) has no X-Forwarded-For and falls
# back to REMOTE_ADDR.
forwarded_for = request.META.get("HTTP_X_FORWARDED_FOR", "")
if forwarded_for:
return forwarded_for.split(",")[0].strip()
return request.META.get("REMOTE_ADDR", "unknown")
def _attempts_key(kind: str, value: str) -> str:
return f"login-attempts:{kind}:{value}"
def _too_many_attempts(key: str, limit: int) -> bool:
return cache.get(key, 0) >= limit
def _record_failed_attempt(key: str) -> None:
try:
cache.incr(key)
except ValueError:
cache.set(key, 1, LOGIN_RATE_LIMIT_WINDOW_SECONDS)
class LoginView(DjangoLoginView):
"""Wraps Django's own LoginView with the throttle above — everything
else (form, template, redirect behavior) is unchanged."""
template_name = "accounts/login.html"
def post(self, request, *args, **kwargs):
address_key = _attempts_key("addr", _client_address(request))
username_key = _attempts_key("user", request.POST.get("username", "")[:150])
if _too_many_attempts(
address_key, LOGIN_RATE_LIMIT_MAX_ATTEMPTS_PER_ADDRESS
) or _too_many_attempts(username_key, LOGIN_RATE_LIMIT_MAX_ATTEMPTS_PER_USERNAME):
messages.error(
request, "Too many login attempts. Please wait a few minutes and try again."
)
return redirect("accounts:login")
return super().post(request, *args, **kwargs)
def form_invalid(self, form):
_record_failed_attempt(_attempts_key("addr", _client_address(self.request)))
_record_failed_attempt(_attempts_key("user", self.request.POST.get("username", "")[:150]))
return super().form_invalid(form)
def setup(request): def setup(request):
if User.objects.exists(): if User.objects.exists():
+18 -1
View File
@@ -1 +1,18 @@
# Register your models here. from django.contrib import admin
from .models import SiteSettings
@admin.register(SiteSettings)
class SiteSettingsAdmin(admin.ModelAdmin):
list_display = ["fetch_interval_minutes"]
fields = ["fetch_interval_minutes"]
def has_add_permission(self, request):
# Singleton — only offer "Add" when the row doesn't exist yet
# (a fresh install before the first poll_due_feeds run or admin
# visit has created it via SiteSettings.load()).
return not SiteSettings.objects.exists()
def has_delete_permission(self, request, obj=None):
return False
+33
View File
@@ -0,0 +1,33 @@
# Generated by Django 5.2.18 on 2026-10-31 18:56
from django.db import migrations, models
class Migration(migrations.Migration):
initial = True
dependencies = []
operations = [
migrations.CreateModel(
name="SiteSettings",
fields=[
(
"id",
models.PositiveSmallIntegerField(default=1, primary_key=True, serialize=False),
),
(
"fetch_interval_minutes",
models.PositiveIntegerField(
default=60,
help_text="How often, in minutes, every feed is polled for new articles. Can't be set lower than EVECTION_MIN_FETCH_INTERVAL_MINUTES (see .env).",
),
),
],
options={
"verbose_name": "Site settings",
"verbose_name_plural": "Site settings",
},
),
]
+35 -1
View File
@@ -1 +1,35 @@
# Create your models here. from django.db import models
class SiteSettings(models.Model):
"""Singleton row (pk=1, same pattern as accounts.SetupLock) holding
instance-wide configuration an operator reasonably wants to change
without touching the server or .env — editable via Django admin.
Currently just the feed-poll interval, which used to be a per-feed
field; it's site-wide now (see apps.feeds.tasks.poll_due_feeds).
"""
id = models.PositiveSmallIntegerField(primary_key=True, default=1)
fetch_interval_minutes = models.PositiveIntegerField(
default=60,
help_text=(
"How often, in minutes, every feed is polled for new articles. "
"Can't be set lower than EVECTION_MIN_FETCH_INTERVAL_MINUTES (see .env)."
),
)
class Meta:
verbose_name = "Site settings"
verbose_name_plural = "Site settings"
def __str__(self):
return "Site settings"
@classmethod
def load(cls):
"""Returns the singleton row, creating it with defaults on first
use so callers never need to special-case "no row exists yet"
(e.g. a fresh install whose first poll_due_feeds run hasn't
created it, and whose admin hasn't visited the settings page)."""
obj, _ = cls.objects.get_or_create(pk=1)
return obj
+74
View File
@@ -0,0 +1,74 @@
from datetime import timedelta
from unittest.mock import patch
import pytest
from django.contrib.auth import get_user_model
from django.urls import reverse
from django.utils import timezone
from apps.feeds.models import Feed
from apps.feeds.tasks import poll_due_feeds
from .models import SiteSettings
User = get_user_model()
@pytest.mark.django_db
def test_load_creates_the_singleton_row_with_defaults():
assert not SiteSettings.objects.exists()
settings_row = SiteSettings.load()
assert settings_row.fetch_interval_minutes == 60
assert SiteSettings.objects.count() == 1
@pytest.mark.django_db
def test_load_returns_the_existing_row_without_creating_a_second_one():
SiteSettings.objects.create(fetch_interval_minutes=30)
settings_row = SiteSettings.load()
assert settings_row.fetch_interval_minutes == 30
assert SiteSettings.objects.count() == 1
@pytest.mark.django_db
def test_poll_due_feeds_uses_the_site_wide_interval_for_every_feed():
SiteSettings.objects.create(fetch_interval_minutes=120)
due = Feed.objects.create(
feed_url="https://a.example.com/feed.xml",
last_fetched_at=timezone.now() - timedelta(minutes=150),
)
not_due = Feed.objects.create(
feed_url="https://b.example.com/feed.xml",
last_fetched_at=timezone.now() - timedelta(minutes=90),
)
with patch("apps.feeds.tasks.fetch_feed.delay") as mock_delay:
poll_due_feeds()
dispatched_ids = {call.args[0] for call in mock_delay.call_args_list}
assert dispatched_ids == {due.id}
assert not_due.id not in dispatched_ids
@pytest.mark.django_db
def test_admin_offers_add_only_once_and_never_offers_delete(client):
# ModelAdmin.has_add_permission/has_delete_permission returning False
# makes Django raise PermissionDenied rather than build a 403 response
# directly — the test client's default behavior re-raises that for
# debugging convenience instead of returning it as a response.
client.raise_request_exception = False
User.objects.create_superuser("admin", password="pw-admin-123")
client.login(username="admin", password="pw-admin-123")
add_url = reverse("admin:core_sitesettings_add")
response = client.get(add_url)
assert response.status_code == 200 # no row yet — "Add" is offered
settings_row = SiteSettings.load()
response = client.get(add_url)
assert response.status_code == 403 # a row exists now — add is refused
delete_url = reverse("admin:core_sitesettings_delete", args=[settings_row.pk])
response = client.get(delete_url)
assert response.status_code == 403
@@ -0,0 +1,17 @@
# Generated by Django 5.2.18 on 2026-10-31 18:56
from django.db import migrations
class Migration(migrations.Migration):
dependencies = [
("feeds", "0003_feed_claimed_at"),
]
operations = [
migrations.RemoveField(
model_name="feed",
name="fetch_interval_minutes",
),
]
+4 -2
View File
@@ -35,8 +35,10 @@ class Feed(models.Model):
) )
last_error_message = models.TextField(blank=True) last_error_message = models.TextField(blank=True)
# Instance-wide, not a per-user preference — see class docstring. # The poll interval itself is site-wide now (apps.core.models.
fetch_interval_minutes = models.PositiveIntegerField(default=60) # SiteSettings), not per-feed — only the error-driven backoff on top
# of it still varies per feed (see apps.feeds.services.fetcher.
# effective_interval_minutes).
fetch_error_count = models.PositiveIntegerField(default=0) fetch_error_count = models.PositiveIntegerField(default=0)
is_active = models.BooleanField(default=True) is_active = models.BooleanField(default=True)
+37 -13
View File
@@ -31,6 +31,15 @@ FETCH_TIMEOUT_SECONDS = 15
MAX_BACKOFF_MINUTES = 24 * 60 MAX_BACKOFF_MINUTES = 24 * 60
USER_AGENT = "Evection/0.1 (+https://github.com/; self-hosted RSS reader)" USER_AGENT = "Evection/0.1 (+https://github.com/; self-hosted RSS reader)"
# ssrf_guard.MAX_RESPONSE_BYTES already bounds the raw download; these
# bound what a single poll does *with* that response — a feed within the
# byte cap can still carry an unreasonable number of entries, or one
# entry with an unreasonably large body, and both cost real CPU/DB time
# (nh3 sanitization, an Article row/UPDATE) per entry processed.
MAX_ENTRIES_PER_POLL = 200
MAX_CONTENT_HTML_INPUT_CHARS = 200_000 # bounds sanitizer cost, not just storage
MAX_CONTENT_HTML_CHARS = 50_000 # bounds what's actually stored/rendered
ALLOWED_TAGS = { ALLOWED_TAGS = {
"a", "a",
"p", "p",
@@ -83,7 +92,9 @@ class FetchResult:
def sanitize_html(raw_html: str) -> str: def sanitize_html(raw_html: str) -> str:
if not raw_html: if not raw_html:
return "" return ""
return nh3.clean(raw_html, tags=ALLOWED_TAGS, attributes=ALLOWED_ATTRIBUTES, link_rel=None) bounded = raw_html[:MAX_CONTENT_HTML_INPUT_CHARS]
cleaned = nh3.clean(bounded, tags=ALLOWED_TAGS, attributes=ALLOWED_ATTRIBUTES, link_rel=None)
return cleaned[:MAX_CONTENT_HTML_CHARS]
def _content_hash(title: str, content_html: str) -> str: def _content_hash(title: str, content_html: str) -> str:
@@ -113,14 +124,17 @@ def _entry_content_html(entry) -> str:
return getattr(entry, "summary", "") or "" return getattr(entry, "summary", "") or ""
def effective_interval_minutes(feed: Feed) -> int: def effective_interval_minutes(feed: Feed, configured_interval_minutes: int) -> int:
"""The actual polling interval to use, applying exponential backoff on """The actual polling interval to use for `feed`, applying exponential
top of the admin-configured ``fetch_interval_minutes`` for feeds that backoff on top of the site-wide `configured_interval_minutes` (see
are currently erroring. ``fetch_interval_minutes`` itself is never apps.core.models.SiteSettings) for a feed that's currently erroring.
mutated by the fetcher — it's the admin's chosen baseline. Takes the configured value as a parameter rather than querying
SiteSettings itself, since this runs once per feed (apps.feeds.tasks
calls it from a list comprehension over every active feed) and the
site-wide value doesn't vary by feed.
""" """
floor = getattr(settings, "EVECTION_MIN_FETCH_INTERVAL_MINUTES", 15) floor = getattr(settings, "EVECTION_MIN_FETCH_INTERVAL_MINUTES", 15)
base = max(feed.fetch_interval_minutes, floor) base = max(configured_interval_minutes, floor)
if feed.fetch_error_count <= 0: if feed.fetch_error_count <= 0:
return base return base
backoff = base * (2**feed.fetch_error_count) backoff = base * (2**feed.fetch_error_count)
@@ -144,10 +158,10 @@ def fetch_and_ingest(feed: Feed) -> FetchResult:
response = safe_get(feed.feed_url, headers=headers, timeout=FETCH_TIMEOUT_SECONDS) response = safe_get(feed.feed_url, headers=headers, timeout=FETCH_TIMEOUT_SECONDS)
except SSRFBlocked as exc: except SSRFBlocked as exc:
feed.last_fetched_at = timezone.now() feed.last_fetched_at = timezone.now()
return _record_error(feed, f"Refused to fetch: {exc}") return record_fetch_error(feed, f"Refused to fetch: {exc}")
except requests.RequestException as exc: except requests.RequestException as exc:
feed.last_fetched_at = timezone.now() feed.last_fetched_at = timezone.now()
return _record_error(feed, f"Request failed: {exc}") return record_fetch_error(feed, f"Request failed: {exc}")
feed.last_fetched_at = timezone.now() feed.last_fetched_at = timezone.now()
@@ -159,12 +173,12 @@ def fetch_and_ingest(feed: Feed) -> FetchResult:
) )
if response.status_code >= 400: if response.status_code >= 400:
return _record_error(feed, f"HTTP {response.status_code}") return record_fetch_error(feed, f"HTTP {response.status_code}")
parsed = feedparser.parse(response.content) parsed = feedparser.parse(response.content)
if parsed.bozo and not parsed.entries: if parsed.bozo and not parsed.entries:
return _record_error(feed, f"Unparseable feed: {parsed.get('bozo_exception')}") return record_fetch_error(feed, f"Unparseable feed: {parsed.get('bozo_exception')}")
if parsed.bozo: if parsed.bozo:
logger.warning( logger.warning(
@@ -186,7 +200,17 @@ def fetch_and_ingest(feed: Feed) -> FetchResult:
new_ids: list[int] = [] new_ids: list[int] = []
updated_ids: list[int] = [] updated_ids: list[int] = []
for entry in parsed.entries: entries = parsed.entries
if len(entries) > MAX_ENTRIES_PER_POLL:
logger.warning(
"Feed %s: %d entries in one poll, only processing the first %d",
feed.feed_url,
len(entries),
MAX_ENTRIES_PER_POLL,
)
entries = entries[:MAX_ENTRIES_PER_POLL]
for entry in entries:
guid = _entry_guid(entry) guid = _entry_guid(entry)
title = (getattr(entry, "title", "") or "")[:1000] title = (getattr(entry, "title", "") or "")[:1000]
content_html = sanitize_html(_entry_content_html(entry)) content_html = sanitize_html(_entry_content_html(entry))
@@ -220,7 +244,7 @@ def fetch_and_ingest(feed: Feed) -> FetchResult:
) )
def _record_error(feed: Feed, message: str) -> FetchResult: def record_fetch_error(feed: Feed, message: str) -> FetchResult:
feed.last_fetch_status = Feed.STATUS_ERROR feed.last_fetch_status = Feed.STATUS_ERROR
feed.last_error_message = message[:2000] feed.last_error_message = message[:2000]
feed.fetch_error_count += 1 feed.fetch_error_count += 1
+39 -6
View File
@@ -8,10 +8,11 @@ from django.db.models import Q
from django.utils import timezone from django.utils import timezone
from apps.articles.tasks import apply_filter_rules_for_feed from apps.articles.tasks import apply_filter_rules_for_feed
from apps.core.models import SiteSettings
from .models import Feed from .models import Feed
from .opml import import_opml from .opml import import_opml
from .services.fetcher import effective_interval_minutes, fetch_and_ingest from .services.fetcher import effective_interval_minutes, fetch_and_ingest, record_fetch_error
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
@@ -32,12 +33,15 @@ def poll_due_feeds():
""" """
now = timezone.now() now = timezone.now()
claim_stale_before = now - timedelta(minutes=CLAIM_STALE_MINUTES) claim_stale_before = now - timedelta(minutes=CLAIM_STALE_MINUTES)
# Loaded once per call, not per feed — the interval is site-wide
# (apps.core.models.SiteSettings), so every feed below shares it.
configured_interval = SiteSettings.load().fetch_interval_minutes
due_feed_ids = [ due_feed_ids = [
feed.id feed.id
for feed in Feed.objects.filter(is_active=True).only( for feed in Feed.objects.filter(is_active=True).only(
"id", "last_fetched_at", "fetch_interval_minutes", "fetch_error_count" "id", "last_fetched_at", "fetch_error_count"
) )
if _is_due(feed) if _is_due(feed, configured_interval)
] ]
claimed_ids = [] claimed_ids = []
for feed_id in due_feed_ids: for feed_id in due_feed_ids:
@@ -57,21 +61,50 @@ def poll_due_feeds():
return len(claimed_ids) return len(claimed_ids)
def _is_due(feed: Feed) -> bool: def _is_due(feed: Feed, configured_interval_minutes: int) -> bool:
if feed.last_fetched_at is None: if feed.last_fetched_at is None:
return True return True
interval = effective_interval_minutes(feed) interval = effective_interval_minutes(feed, configured_interval_minutes)
return feed.last_fetched_at <= timezone.now() - timedelta(minutes=interval) return feed.last_fetched_at <= timezone.now() - timedelta(minutes=interval)
@shared_task # Bounds a single feed's fetch+parse+ingest, independent of
# FETCH_TIMEOUT_SECONDS (apps.feeds.services.fetcher) which only covers
# the HTTP request itself: parsing/sanitizing/writing a large-but-under-
# the-byte-cap feed could still run long. soft_time_limit lets the
# except-Exception handler below record a normal feed error (and clear
# the claim) before the hard time_limit kills the worker process outright.
FETCH_FEED_SOFT_TIME_LIMIT = 60
FETCH_FEED_TIME_LIMIT = 90
@shared_task(time_limit=FETCH_FEED_TIME_LIMIT, soft_time_limit=FETCH_FEED_SOFT_TIME_LIMIT)
def fetch_feed(feed_id: int): def fetch_feed(feed_id: int):
try: try:
feed = Feed.objects.get(pk=feed_id, is_active=True) feed = Feed.objects.get(pk=feed_id, is_active=True)
except Feed.DoesNotExist: except Feed.DoesNotExist:
return None return None
try:
result = fetch_and_ingest(feed) result = fetch_and_ingest(feed)
except Exception:
# fetch_and_ingest already handles the expected failure modes
# (network errors, SSRF-blocked targets, unparseable feeds) by
# returning an error FetchResult rather than raising — anything
# that reaches here is unexpected (a parser/sanitizer bug, a DB
# error, this task's own soft time limit), so it's recorded the
# same way a handled failure would be rather than silently
# leaving the feed's status stale, the claim held for the full
# CLAIM_STALE_MINUTES, and no record of what happened. Re-raised
# afterward so Celery still marks the task FAILED (visible in
# worker logs/monitoring) instead of swallowing it.
logger.exception("Feed %s: unexpected error during fetch", feed.feed_url)
record_fetch_error(feed, "Unexpected error — see worker logs")
feed.last_fetched_at = timezone.now()
feed.claimed_at = None
feed.save()
raise
feed.claimed_at = None feed.claimed_at = None
feed.save() feed.save()
+34 -2
View File
@@ -5,7 +5,13 @@ from django.utils import timezone
from apps.articles.models import Article, UserArticleState from apps.articles.models import Article, UserArticleState
from apps.feeds.models import Feed from apps.feeds.models import Feed
from apps.feeds.services.fetcher import effective_interval_minutes, fetch_and_ingest from apps.feeds.services.fetcher import (
MAX_CONTENT_HTML_CHARS,
MAX_ENTRIES_PER_POLL,
effective_interval_minutes,
fetch_and_ingest,
sanitize_html,
)
FEED_URL = "https://example.com/feed.xml" FEED_URL = "https://example.com/feed.xml"
@@ -130,7 +136,7 @@ def test_unparseable_feed_records_error_and_backs_off(feed):
assert result.status == Feed.STATUS_ERROR assert result.status == Feed.STATUS_ERROR
assert feed.fetch_error_count == 1 assert feed.fetch_error_count == 1
assert effective_interval_minutes(feed) > feed.fetch_interval_minutes assert effective_interval_minutes(feed, 60) > 60
@pytest.mark.django_db @pytest.mark.django_db
@@ -196,3 +202,29 @@ def test_refetching_updated_content_does_not_reset_user_state():
state.refresh_from_db() state.refresh_from_db()
assert state.is_read is True assert state.is_read is True
assert ArticleTag.objects.filter(user=user, article=article, tag=starred_tag).exists() assert ArticleTag.objects.filter(user=user, article=article, tag=starred_tag).exists()
def test_sanitize_html_caps_stored_output_length():
huge = "<p>" + ("a" * (MAX_CONTENT_HTML_CHARS + 10_000)) + "</p>"
cleaned = sanitize_html(huge)
assert len(cleaned) <= MAX_CONTENT_HTML_CHARS
@pytest.mark.django_db
@responses.activate
def test_fetch_only_processes_the_first_max_entries_per_poll(feed):
items = "".join(
f"<item><title>Item {i}</title><guid>guid-{i}</guid></item>"
for i in range(MAX_ENTRIES_PER_POLL + 5)
)
xml = f"""<?xml version="1.0"?>
<rss version="2.0"><channel><title>Big feed</title>{items}</channel></rss>
"""
responses.add(responses.GET, FEED_URL, body=xml, status=200)
result = fetch_and_ingest(feed)
feed.save()
assert len(result.new_article_ids) == MAX_ENTRIES_PER_POLL
assert Article.objects.filter(feed=feed).count() == MAX_ENTRIES_PER_POLL
assert not Article.objects.filter(feed=feed, guid=f"guid-{MAX_ENTRIES_PER_POLL}").exists()
+19 -5
View File
@@ -16,17 +16,14 @@ def test_poll_due_feeds_dispatches_only_due_feeds():
recently_fetched = Feed.objects.create( recently_fetched = Feed.objects.create(
feed_url="https://b.example.com/feed.xml", feed_url="https://b.example.com/feed.xml",
last_fetched_at=timezone.now(), last_fetched_at=timezone.now(),
fetch_interval_minutes=60,
) )
overdue = Feed.objects.create( overdue = Feed.objects.create(
feed_url="https://c.example.com/feed.xml", feed_url="https://c.example.com/feed.xml",
last_fetched_at=timezone.now() - timedelta(hours=2), last_fetched_at=timezone.now() - timedelta(hours=2),
fetch_interval_minutes=60,
) )
inactive_but_overdue = Feed.objects.create( inactive_but_overdue = Feed.objects.create(
feed_url="https://d.example.com/feed.xml", feed_url="https://d.example.com/feed.xml",
last_fetched_at=timezone.now() - timedelta(hours=2), last_fetched_at=timezone.now() - timedelta(hours=2),
fetch_interval_minutes=60,
is_active=False, is_active=False,
) )
@@ -48,7 +45,6 @@ def test_poll_due_feeds_does_not_reclaim_a_feed_it_just_claimed():
overdue = Feed.objects.create( overdue = Feed.objects.create(
feed_url="https://c.example.com/feed.xml", feed_url="https://c.example.com/feed.xml",
last_fetched_at=timezone.now() - timedelta(hours=2), last_fetched_at=timezone.now() - timedelta(hours=2),
fetch_interval_minutes=60,
) )
with patch("apps.feeds.tasks.fetch_feed.delay") as mock_delay: with patch("apps.feeds.tasks.fetch_feed.delay") as mock_delay:
@@ -69,7 +65,6 @@ def test_poll_due_feeds_reclaims_a_stale_claim():
overdue = Feed.objects.create( overdue = Feed.objects.create(
feed_url="https://c.example.com/feed.xml", feed_url="https://c.example.com/feed.xml",
last_fetched_at=timezone.now() - timedelta(hours=2), last_fetched_at=timezone.now() - timedelta(hours=2),
fetch_interval_minutes=60,
claimed_at=timezone.now() - timedelta(minutes=CLAIM_STALE_MINUTES + 1), claimed_at=timezone.now() - timedelta(minutes=CLAIM_STALE_MINUTES + 1),
) )
@@ -127,3 +122,22 @@ def test_fetch_feed_task_skips_inactive_feed():
@pytest.mark.django_db @pytest.mark.django_db
def test_fetch_feed_task_skips_missing_feed(): def test_fetch_feed_task_skips_missing_feed():
assert fetch_feed(999999) is None assert fetch_feed(999999) is None
@pytest.mark.django_db
def test_fetch_feed_task_records_error_and_clears_claim_on_unexpected_exception():
feed = Feed.objects.create(
feed_url="https://example.com/feed.xml",
claimed_at=timezone.now(),
fetch_error_count=0,
)
with patch("apps.feeds.tasks.fetch_and_ingest", side_effect=RuntimeError("boom")):
with pytest.raises(RuntimeError, match="boom"):
fetch_feed(feed.id)
feed.refresh_from_db()
assert feed.last_fetch_status == Feed.STATUS_ERROR
assert feed.fetch_error_count == 1
assert feed.claimed_at is None
assert feed.last_fetched_at is not None
+25
View File
@@ -106,9 +106,34 @@ EVECTION_SIGNUP_ENABLED = env.bool("EVECTION_SIGNUP_ENABLED", default=False)
EVECTION_MIN_FETCH_INTERVAL_MINUTES = env.int("EVECTION_MIN_FETCH_INTERVAL_MINUTES", default=15) EVECTION_MIN_FETCH_INTERVAL_MINUTES = env.int("EVECTION_MIN_FETCH_INTERVAL_MINUTES", default=15)
EVECTION_PWA_CACHE_VERSION = env.str("EVECTION_PWA_CACHE_VERSION", default="v1") EVECTION_PWA_CACHE_VERSION = env.str("EVECTION_PWA_CACHE_VERSION", default="v1")
# Redis-backed rather than Django's default per-process LocMemCache: login
# rate limiting (apps.accounts.views) must share counters across every
# gunicorn worker, not just whichever one happened to handle a given
# request. Redis is already a hard install requirement for Celery, so
# this adds no new infrastructure — just a distinct key prefix so cache
# keys can't collide with anything else sharing that Redis instance.
CACHES = {
"default": {
"BACKEND": "django.core.cache.backends.redis.RedisCache",
"LOCATION": env.str("REDIS_URL", default="redis://localhost:6379/0"),
"KEY_PREFIX": "evection-cache",
}
}
# Celery # Celery
CELERY_BROKER_URL = env.str("REDIS_URL", default="redis://localhost:6379/0") CELERY_BROKER_URL = env.str("REDIS_URL", default="redis://localhost:6379/0")
CELERY_RESULT_BACKEND = None CELERY_RESULT_BACKEND = None
CELERY_TASK_SERIALIZER = "json" CELERY_TASK_SERIALIZER = "json"
CELERY_ACCEPT_CONTENT = ["json"] CELERY_ACCEPT_CONTENT = ["json"]
CELERY_TIMEZONE = TIME_ZONE CELERY_TIMEZONE = TIME_ZONE
# Without these, a worker process killed mid-task (OOM, deploy restart,
# crash) silently loses that task: the default is to ack a message the
# moment a worker *starts* it, not when it finishes. acks_late + reject-
# on-worker-lost means the message is only consumed on completion (success
# or a handled failure) and is redelivered to another worker otherwise.
# prefetch_multiplier=1 keeps a worker from grabbing a batch of tasks it
# could similarly lose all at once.
CELERY_TASK_ACKS_LATE = True
CELERY_TASK_REJECT_ON_WORKER_LOST = True
CELERY_WORKER_PREFETCH_MULTIPLIER = 1
+7
View File
@@ -5,3 +5,10 @@ CELERY_TASK_ALWAYS_EAGER = True
CELERY_TASK_EAGER_PROPAGATES = True CELERY_TASK_EAGER_PROPAGATES = True
PASSWORD_HASHERS = ["django.contrib.auth.hashers.MD5PasswordHasher"] PASSWORD_HASHERS = ["django.contrib.auth.hashers.MD5PasswordHasher"]
# base.py's CACHES needs a real Redis instance — tests must not depend on
# one being available, and a plain per-process cache is also what keeps
# login-rate-limit tests isolated from each other (LocMemCache's default
# location is per-process but shared across tests in that process; see
# apps.accounts.test_rate_limit's cache.clear() fixture).
CACHES = {"default": {"BACKEND": "django.core.cache.backends.locmem.LocMemCache"}}
+5 -1
View File
@@ -66,7 +66,7 @@ and 4 below first — prerequisites and what `.env` controls.
- Python 3.11 or 3.12, with its `venv` module (Debian/Ubuntu split this into a - Python 3.11 or 3.12, with its `venv` module (Debian/Ubuntu split this into a
separate `python3.X-venv` package — see below) separate `python3.X-venv` package — see below)
- Redis (Celery's broker) - Redis (Celery's broker, and the app's cache — e.g. login rate limiting)
- **Postgres only**: a PostgreSQL server (12+) and its client headers if you're not - **Postgres only**: a PostgreSQL server (12+) and its client headers if you're not
using the bundled `psycopg[binary]` wheel (most platforms are covered by the wheel, using the bundled `psycopg[binary]` wheel (most platforms are covered by the wheel,
so this is rarely needed) so this is rarely needed)
@@ -143,6 +143,10 @@ At minimum, set the following in `.env`:
- `EVECTION_GUNICORN_BIND` — the `address:port` gunicorn listens on; see - `EVECTION_GUNICORN_BIND` — the `address:port` gunicorn listens on; see
"Reverse proxy" below for how to pick this "Reverse proxy" below for how to pick this
How often every feed is polled is a site-wide setting (Django admin → Site
settings), not a `.env` value — `EVECTION_MIN_FETCH_INTERVAL_MINUTES` only sets
a floor under it, so it can't be configured lower than that without editing `.env`.
Evection reads `config.settings.prod` when run via the commands below (see each Evection reads `config.settings.prod` when run via the commands below (see each
systemd unit's `DJANGO_SETTINGS_MODULE`). That module turns on HTTPS-only cookies, systemd unit's `DJANGO_SETTINGS_MODULE`). That module turns on HTTPS-only cookies,
HSTS, and a Content-Security-Policy header on top of the base settings, and HSTS, and a Content-Security-Policy header on top of the base settings, and