Compare commits

...
Author SHA1 Message Date
ksmithandClaude Sonnet 5.5 0975a7634b Mark the browser tab with a red dot when new articles arrive (#31)
The in-page "new articles" banner is invisible from a background tab.
Every authenticated page now polls a small JSON endpoint
(reader:new_since, every 30s) and redraws the tab's favicon with a red
dot when unread, non-hidden articles from the user's feeds have
appeared since the page loaded. A reload (which is what clicking the
banner does) resets the baseline and the icon.

The endpoint is deliberately not scoped to the stream's current
filter/tag/feed, so the tab lights up for anything new whichever page
is open. Scope is the dot only: no desktop notifications, per the
request. Verified in a dev server enforcing the production CSP: the
icon swapped to the dotted version after a new article arrived and
reset on reload.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-11-01 10:18:29 -07:00
ksmithandClaude Sonnet 5.5 fd494a9c08 Make the readiness check catch fetching that is scheduled but failing (#27)
/health/ready/ proved beat and a worker were alive but not that
fetching worked: every feed could be failing, or the queue could be
hours behind, and it would still report ok.

apps.core.health adds three checks: Celery queue depth (real Redis list
length, over 1000 fails), a successful fetch within three poll intervals
(30 min minimum, with a grace period for a brand-new install), and the
share of active feeds failing (over half, once there are at least four).
The payload also reports queue depth, active/failing feed counts, and
the age of the last successful fetch. Verified against real Redis with
queued Celery tasks, and the suite on PostgreSQL. docs/install.md lists
each check.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-11-01 09:30:00 -07:00
ksmithandClaude Sonnet 5.5 f8cd1eb33e Stop retention from bringing deleted articles back (#29)
The nightly purge deleted articles, but a feed that still listed an
entry would re-insert it on its next poll as a brand-new, unread
article. For feeds with long archives, old items could reappear after
every retention run.

The purge now leaves a (feed, guid) tombstone per deleted article, and
ingestion skips tombstoned guids. With retention on, entries whose own
publish date is already outside the window are not ingested at all, so
a long archive isn't fetched only to be purged and tombstoned the same
night. Undated entries rely on the tombstone.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-11-01 09:30:00 -07:00
ksmithandClaude Sonnet 5.5 73dbee9400 Document running the test suite against PostgreSQL
The Postgres search backend's tests are skipped on SQLite, so they only
run when pointed at a real Postgres. Documents the throwaway-container
recipe used to verify the stored/GIN-indexed search vector migration.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-11-01 09:23:02 -07:00
ksmithandClaude Sonnet 5.5 6c6227af57 Treat an article with no publish date as published when we fetched it
Feeds that give no publish date left the row with no time at all and
grouped the article under "Undated". The stream's sort order already
fell back to fetched_at; the list row, its date-group header, and the
open article's header now use the same fallback. The article's own
published_at stays empty — nothing is invented in the database, so a
later refetch can't see a spurious change.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-11-01 09:14:09 -07:00
ksmithandClaude Sonnet 5.5 c339d456e4 Show the date as well as the time on each stream row
Rows only showed a time of day, leaving the date to the group header
above. Each row now shows "Oct 31, 5:30 PM" in the viewer's local
timezone, with the year added only for articles from a previous year.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-11-01 09:07:36 -07:00
ksmithandClaude Sonnet 5.5 7afda7d0ba Fix inline article expand and back-to-stream scope in production
Clicking an article title opened the detail page instead of expanding
it inline, and "Back to stream" then reset to every feed. Cause: the
title (and the new-articles banner) used inline onclick= handlers,
which production's CSP (script-src 'self') blocks — dev sends no CSP,
so it worked there. The title link just fell through to its href.

Behavior moves to delegated listeners in static/js/stream.js (modified
clicks still open in a new tab). A test now fails if any template gains
an inline on*= handler. The stream also passes its filter/tag/feed
scope through the article link so the detail page's back link returns
to it. Verified in a dev server enforcing the same CSP: title click
expands and collapses in place, URL unchanged.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-11-01 09:02:51 -07:00
23 changed files with 699 additions and 40 deletions

No files matched your search

+12
View File
@@ -54,6 +54,18 @@ ruff check .
black --check .
```
### Testing against PostgreSQL
The suite runs on SQLite by default; the Postgres-only tests (search backend)
are skipped there. To run everything against a real Postgres:
```bash
docker run -d --name evection-pg-test -e POSTGRES_PASSWORD=testpw -e POSTGRES_DB=evection \
-p 127.0.0.1:55432:5432 postgres:16
DATABASE_URL=postgres://postgres:testpw@127.0.0.1:55432/evection pytest
docker rm -f evection-pg-test
```
## License
[GNU AGPL-3.0-or-later](LICENSE) — if you run a modified version of Evection as
@@ -0,0 +1,27 @@
# Generated by Django 5.2.18 on 2026-11-01 16:28
import django.db.models.deletion
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
('articles', '0004_starred_saved_as_system_tags'),
('feeds', '0006_backfill_feed_next_fetch_at'),
]
operations = [
migrations.CreateModel(
name='PurgedArticle',
fields=[
('id', models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
('guid', models.CharField(max_length=1000)),
('purged_at', models.DateTimeField(auto_now_add=True)),
('feed', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, related_name='purged_articles', to='feeds.feed')),
],
options={
'unique_together': {('feed', 'guid')},
},
),
]
+17
View File
@@ -32,6 +32,23 @@ class Article(models.Model):
return self.title or self.guid
class PurgedArticle(models.Model):
"""Tombstone left by the retention purge (apps.articles.tasks.
purge_old_articles) for every article it deletes: a feed that still
lists an entry after we purged it would otherwise be re-inserted as a
brand-new, unread article on its next poll. Ingestion skips any guid
that has a tombstone here. Only the (feed, guid) pair is kept — a
small fraction of the article it replaces.
"""
feed = models.ForeignKey(Feed, on_delete=models.CASCADE, related_name="purged_articles")
guid = models.CharField(max_length=1000)
purged_at = models.DateTimeField(auto_now_add=True)
class Meta:
unique_together = [("feed", "guid")]
class UserArticleState(models.Model):
"""Per-user read/hide state for an Article.
+13 -3
View File
@@ -1,6 +1,7 @@
from datetime import timedelta
from celery import shared_task
from django.db import transaction
from django.utils import timezone
from apps.core.models import SiteSettings
@@ -8,7 +9,7 @@ from apps.feeds.models import Subscription
from apps.rules.engine import evaluate_rules_for_article
from apps.rules.models import FilterRule
from .models import Article, ArticleTag, SubscriptionTag
from .models import Article, ArticleTag, PurgedArticle, SubscriptionTag
@shared_task
@@ -75,6 +76,10 @@ def purge_old_articles():
ArticleTag rows go with their article via ON DELETE CASCADE, and the
SQLite FTS triggers drop the search entries. 0 days means keep
everything forever (the default).
Each deleted article leaves a PurgedArticle tombstone first, so a feed
that still lists the entry can't bring it back as a new, unread
article on its next poll (see apps.feeds.services.fetcher).
"""
retention_days = SiteSettings.load().retention_days
if retention_days <= 0:
@@ -87,9 +92,14 @@ def purge_old_articles():
chunk = list(
Article.objects.filter(fetched_at__lt=cutoff)
.exclude(pk__in=keep)
.values_list("pk", flat=True)[:PURGE_CHUNK_SIZE]
.values_list("pk", "feed_id", "guid")[:PURGE_CHUNK_SIZE]
)
if not chunk:
return deleted_total
Article.objects.filter(pk__in=chunk).delete()
with transaction.atomic():
PurgedArticle.objects.bulk_create(
[PurgedArticle(feed_id=feed_id, guid=guid) for _pk, feed_id, guid in chunk],
ignore_conflicts=True,
)
Article.objects.filter(pk__in=[pk for pk, _feed_id, _guid in chunk]).delete()
deleted_total += len(chunk)
+17
View File
@@ -72,3 +72,20 @@ def test_purge_works_through_more_than_one_chunk(feed, monkeypatch):
assert purge_old_articles() == 5
assert Article.objects.count() == 0
@pytest.mark.django_db
def test_purge_leaves_a_tombstone_per_deleted_article(feed):
from .models import PurgedArticle
SiteSettings.objects.create(retention_days=30)
_article(feed, "gone-1", age_days=60)
_article(feed, "gone-2", age_days=60)
_article(feed, "kept", age_days=1)
purge_old_articles()
assert set(PurgedArticle.objects.filter(feed=feed).values_list("guid", flat=True)) == {
"gone-1",
"gone-2",
}
+83
View File
@@ -0,0 +1,83 @@
from datetime import timedelta
import redis
from django.conf import settings
from django.db.models import Max, Min
from django.utils import timezone
from apps.feeds.models import Feed
from .models import SiteSettings
# More than this many tasks waiting means workers aren't keeping up (or
# aren't running) — fetches are piling up faster than they complete.
MAX_QUEUE_DEPTH = 1000
# Past this share of active feeds failing, it's far more likely something
# on our side (DNS, outbound network, a bad deploy) than every publisher
# breaking at once. Needs a handful of feeds first so one broken feed out
# of two doesn't read as an outage.
MAX_FAILING_RATIO = 0.5
MIN_FEEDS_FOR_RATIO = 4
MIN_SUCCESS_WINDOW_MINUTES = 30
def queue_depth() -> int | None:
"""Tasks waiting in Celery's default queue, or None if the broker
can't be reached. Celery's Redis transport keeps each queue as a Redis
list named after the queue."""
try:
client = redis.Redis.from_url(
settings.CELERY_BROKER_URL, socket_connect_timeout=2, socket_timeout=2
)
return int(client.llen("celery"))
except Exception:
return None
def _success_window() -> timedelta:
floor = getattr(settings, "EVECTION_MIN_FETCH_INTERVAL_MINUTES", 15)
base = max(SiteSettings.load().fetch_interval_minutes, floor)
return timedelta(minutes=max(3 * base, MIN_SUCCESS_WINDOW_MINUTES))
def fetch_pipeline_status(now=None) -> tuple[dict[str, bool], dict]:
"""Checks that fetching is actually *succeeding*, not just scheduled
(the poll heartbeat only proves beat and a worker are alive): returns
({check: passed}, detail) covering queue depth, whether any feed has
been fetched successfully recently, and the share of feeds failing.
"""
now = now or timezone.now()
detail: dict = {}
depth = queue_depth()
detail["queue_depth"] = depth
checks = {"queue_depth": depth is not None and depth <= MAX_QUEUE_DEPTH}
active = Feed.objects.filter(is_active=True)
active_count = active.count()
failing = active.filter(fetch_error_count__gt=0).count()
detail["active_feeds"] = active_count
detail["failing_feeds"] = failing
window = _success_window()
last_success = active.filter(
last_fetch_status__in=[Feed.STATUS_OK, Feed.STATUS_NOT_MODIFIED]
).aggregate(latest=Max("last_fetched_at"))["latest"]
if last_success is not None:
detail["last_successful_fetch_age_seconds"] = int((now - last_success).total_seconds())
recent_success = now - last_success <= window
elif active_count == 0:
recent_success = True
else:
# Nothing has ever fetched successfully — only a problem once
# the oldest feed has had a full window to be fetched.
oldest = active.aggregate(oldest=Min("created_at"))["oldest"]
recent_success = now - oldest <= window
checks["recent_fetch_success"] = recent_success
checks["feed_failures"] = not (
active_count >= MIN_FEEDS_FOR_RATIO and failing / active_count > MAX_FAILING_RATIO
)
return checks, detail
+137 -4
View File
@@ -22,8 +22,17 @@ def test_health_check_works_before_first_run_setup(client):
assert response.status_code == 200
@pytest.fixture
def broker_queue(monkeypatch):
"""Stands in for the Redis broker's queue length (tests have no Redis):
set `broker_queue["depth"]` to a number, or None for "unreachable"."""
state = {"depth": 0}
monkeypatch.setattr("apps.core.health.queue_depth", lambda: state["depth"])
return state
@pytest.mark.django_db
def test_health_ready_is_degraded_until_a_poll_tick_has_happened(client):
def test_health_ready_is_degraded_until_a_poll_tick_has_happened(client, broker_queue):
response = client.get(reverse("core:health_ready"))
assert response.status_code == 503
body = response.json()
@@ -34,22 +43,26 @@ def test_health_ready_is_degraded_until_a_poll_tick_has_happened(client):
@pytest.mark.django_db
def test_health_ready_is_ok_after_poll_due_feeds_runs(client):
def test_health_ready_is_ok_after_poll_due_feeds_runs(client, broker_queue):
from apps.feeds.tasks import poll_due_feeds
poll_due_feeds()
response = client.get(reverse("core:health_ready"))
assert response.status_code == 200
assert response.json()["checks"] == {
checks = response.json()["checks"]
assert checks == {
"database": True,
"redis": True,
"poll_heartbeat": True,
"queue_depth": True,
"recent_fetch_success": True,
"feed_failures": True,
}
@pytest.mark.django_db
def test_health_ready_flags_a_stale_poll_heartbeat(client):
def test_health_ready_flags_a_stale_poll_heartbeat(client, broker_queue):
from datetime import timedelta
from django.core.cache import cache
@@ -119,3 +132,123 @@ def test_retry_feed_resets_backoff_and_requeues(client, django_user_model):
assert response.status_code == 302
mock_reset.assert_called_once_with(feed.id)
mock_fetch.assert_called_once_with(feed.id)
def _ready(client):
from django.core.cache import cache
from django.utils import timezone
from apps.feeds.tasks import POLL_HEARTBEAT_CACHE_KEY
# Record the heartbeat directly: running poll_due_feeds here would
# execute fetches eagerly (CELERY_TASK_ALWAYS_EAGER) and rewrite the
# feed states each test sets up.
cache.set(POLL_HEARTBEAT_CACHE_KEY, timezone.now())
response = client.get(reverse("core:health_ready"))
return response.status_code, response.json()
@pytest.mark.django_db
def test_health_ready_flags_a_backed_up_queue(client, broker_queue):
from .health import MAX_QUEUE_DEPTH
broker_queue["depth"] = MAX_QUEUE_DEPTH + 1
status, body = _ready(client)
assert status == 503
assert body["checks"]["queue_depth"] is False
assert body["queue_depth"] == MAX_QUEUE_DEPTH + 1
@pytest.mark.django_db
def test_health_ready_flags_an_unreachable_broker_queue(client, broker_queue):
broker_queue["depth"] = None
status, body = _ready(client)
assert status == 503
assert body["checks"]["queue_depth"] is False
@pytest.mark.django_db
def test_health_ready_flags_no_recent_successful_fetch(client, broker_queue):
from datetime import timedelta
from django.utils import timezone
feed = Feed.objects.create(
feed_url="https://a.example.com/feed.xml",
last_fetch_status=Feed.STATUS_OK,
last_fetched_at=timezone.now() - timedelta(days=2),
)
Feed.objects.filter(pk=feed.pk).update(created_at=timezone.now() - timedelta(days=30))
status, body = _ready(client)
assert status == 503
assert body["checks"]["recent_fetch_success"] is False
assert body["last_successful_fetch_age_seconds"] > 86400
@pytest.mark.django_db
def test_health_ready_is_ok_with_a_recent_successful_fetch(client, broker_queue):
from django.utils import timezone
Feed.objects.create(
feed_url="https://a.example.com/feed.xml",
last_fetch_status=Feed.STATUS_NOT_MODIFIED,
last_fetched_at=timezone.now(),
)
status, body = _ready(client)
assert status == 200
assert body["active_feeds"] == 1
assert body["failing_feeds"] == 0
@pytest.mark.django_db
def test_health_ready_gives_a_brand_new_feed_time_before_it_counts_as_stale(client, broker_queue):
Feed.objects.create(feed_url="https://new.example.com/feed.xml")
status, body = _ready(client)
assert status == 200
assert body["checks"]["recent_fetch_success"] is True
@pytest.mark.django_db
def test_health_ready_flags_most_feeds_failing_but_not_a_single_broken_one(client, broker_queue):
from django.utils import timezone
now = timezone.now()
healthy = Feed.objects.create(
feed_url="https://ok.example.com/feed.xml",
last_fetch_status=Feed.STATUS_OK,
last_fetched_at=now,
)
broken = [
Feed.objects.create(
feed_url=f"https://bad{i}.example.com/feed.xml",
last_fetch_status=Feed.STATUS_ERROR,
fetch_error_count=3,
last_fetched_at=now,
)
for i in range(2)
]
status, body = _ready(client)
assert status == 200, "2 of 3 failing is below the minimum feed count for the ratio"
Feed.objects.create(
feed_url="https://bad9.example.com/feed.xml",
last_fetch_status=Feed.STATUS_ERROR,
fetch_error_count=1,
last_fetched_at=now,
)
status, body = _ready(client)
assert status == 503
assert body["checks"]["feed_failures"] is False
assert body["failing_feeds"] == 3
assert healthy.pk and broken
+10 -1
View File
@@ -12,6 +12,8 @@ from apps.articles.models import Article
from apps.feeds.models import Feed, Subscription
from apps.feeds.tasks import POLL_HEARTBEAT_CACHE_KEY, fetch_feed, reset_feed_backoff
from .health import fetch_pipeline_status
is_admin = user_passes_test(lambda u: u.is_authenticated and u.is_staff)
@@ -37,7 +39,9 @@ def health_ready(request):
"""Readiness: liveness plus everything background fetching depends on
(C8) — the database, the Redis cache/broker, and a recent
poll_due_feeds tick, which proves beat is scheduling AND a worker is
consuming. /health/ above stays DB-only on purpose: scripts/upgrade.sh
consuming. Beyond that tick, apps.core.health checks fetching is
actually succeeding: queue depth, a recent successful fetch, and the
share of feeds failing. /health/ above stays DB-only on purpose: scripts/upgrade.sh
gates a deploy on it right after restart, before beat has had a
chance to tick. Point an uptime monitor at this one instead."""
checks = {"database": False, "redis": False, "poll_heartbeat": False}
@@ -65,6 +69,11 @@ def health_ready(request):
detail["poll_heartbeat_age_seconds"] = int(age)
checks["poll_heartbeat"] = age <= POLL_HEARTBEAT_MAX_AGE_SECONDS
if checks["database"]:
pipeline_checks, pipeline_detail = fetch_pipeline_status()
checks.update(pipeline_checks)
detail.update(pipeline_detail)
ok = all(checks.values())
return JsonResponse(
{"status": "ok" if ok else "degraded", "checks": checks, **detail},
+21 -2
View File
@@ -11,7 +11,7 @@ import hashlib
import logging
from calendar import timegm
from dataclasses import dataclass
from datetime import datetime
from datetime import datetime, timedelta
from datetime import timezone as dt_timezone
import feedparser
@@ -21,7 +21,8 @@ from django.conf import settings
from django.db import IntegrityError, transaction
from django.utils import timezone
from apps.articles.models import Article
from apps.articles.models import Article, PurgedArticle
from apps.core.models import SiteSettings
from apps.feeds.models import Feed
from .ssrf_guard import SSRFBlocked, safe_get
@@ -263,6 +264,24 @@ def fetch_and_ingest(feed: Feed) -> FetchResult:
"content_hash": _content_hash(title, link, author, summary, content_html, published_at),
}
# Entries the retention purge already deleted (tombstoned) and, when
# retention is on, entries so old by their own publish date that the
# next purge would delete them again, are not (re-)ingested — see
# apps.articles.tasks.purge_old_articles.
retention_days = SiteSettings.load().retention_days
if retention_days > 0:
too_old_before = timezone.now() - timedelta(days=retention_days)
for guid in [
g
for g, f in parsed_entries.items()
if f["published_at"] is not None and f["published_at"] < too_old_before
]:
del parsed_entries[guid]
for guid in PurgedArticle.objects.filter(feed=feed, guid__in=parsed_entries).values_list(
"guid", flat=True
):
del parsed_entries[guid]
existing = {a.guid: a for a in Article.objects.filter(feed=feed, guid__in=parsed_entries)}
to_create = [
+61
View File
@@ -338,3 +338,64 @@ def test_ingest_falls_back_to_row_by_row_when_a_concurrent_insert_wins(feed):
assert len(result.new_article_ids) == 3
assert Article.objects.filter(feed=feed).count() == 3
@pytest.mark.django_db
@responses.activate
def test_purged_article_is_not_resurrected_by_a_feed_that_still_lists_it(feed):
"""#29: a feed still listing an entry the retention purge deleted must
not bring it back as a brand-new, unread article."""
from apps.articles.tasks import purge_old_articles
from apps.core.models import SiteSettings
SiteSettings.objects.create(retention_days=30)
responses.add(responses.GET, FEED_URL, body=_feed_xml(3), status=200)
fetch_and_ingest(feed)
Article.objects.filter(feed=feed, guid="guid-0").update(
fetched_at=timezone.now() - dt.timedelta(days=90)
)
assert purge_old_articles() == 1
assert not Article.objects.filter(feed=feed, guid="guid-0").exists()
result = fetch_and_ingest(feed)
assert result.new_article_ids == []
assert not Article.objects.filter(feed=feed, guid="guid-0").exists()
assert Article.objects.filter(feed=feed).count() == 2
@pytest.mark.django_db
@responses.activate
def test_entries_older_than_the_retention_window_are_not_ingested(feed):
"""With retention on, a feed's long archive isn't ingested only to be
purged (and tombstoned) the same night."""
from apps.core.models import SiteSettings
SiteSettings.objects.create(retention_days=30)
xml = (
'<?xml version="1.0"?><rss version="2.0"><channel><title>T</title>'
"<item><title>Ancient</title><guid>old</guid>"
"<pubDate>Mon, 01 Jan 2024 00:00:00 GMT</pubDate></item>"
"<item><title>Undated</title><guid>undated</guid></item>"
"</channel></rss>"
)
responses.add(responses.GET, FEED_URL, body=xml, status=200)
result = fetch_and_ingest(feed)
assert len(result.new_article_ids) == 1
assert Article.objects.filter(feed=feed).get().guid == "undated"
@pytest.mark.django_db
@responses.activate
def test_old_entries_are_ingested_normally_when_retention_is_off(feed):
xml = (
'<?xml version="1.0"?><rss version="2.0"><channel><title>T</title>'
"<item><title>Ancient</title><guid>old</guid>"
"<pubDate>Mon, 01 Jan 2024 00:00:00 GMT</pubDate></item>"
"</channel></rss>"
)
responses.add(responses.GET, FEED_URL, body=xml, status=200)
assert len(fetch_and_ingest(feed).new_article_ids) == 1
+119 -3
View File
@@ -204,7 +204,7 @@ def test_article_rows_carry_their_utc_instant_for_client_side_date_grouping(clie
dated = Article.objects.create(
feed=feed, guid="dated", title="Dated", published_at=timezone.now()
)
undated = Article.objects.create(feed=feed, guid="undated", title="Undated")
undated = Article.objects.create(feed=feed, guid="undated", title="No publish date")
client.login(username="alice", password="pw-alice-123")
response = client.get(reverse("reader:stream"), {"filter": "all"})
@@ -214,9 +214,15 @@ def test_article_rows_carry_their_utc_instant_for_client_side_date_grouping(clie
assert f'id="article-{dated.pk}"' in content
assert f'data-published-utc="{expected_iso}"' in content
undated_row = re.search(rf'<li id="article-{undated.pk}".*?>', content, flags=re.DOTALL)
# A feed that gave no publish date falls back to when it was fetched
# (the same fallback the stream's sort order uses), so the row still
# carries a time and groups under a real date, not "Undated".
undated_row = re.search(rf'<li id="article-{undated.pk}".*?</li>', content, flags=re.DOTALL)
assert undated_row is not None
assert 'data-published-utc=""' in undated_row.group(0)
row_html = undated_row.group(0)
assert f'data-published-utc="{undated.fetched_at.isoformat()}"' in row_html
assert 'class="article-time"' in row_html
assert "Undated" not in content
@pytest.mark.django_db
@@ -438,3 +444,113 @@ def test_stream_pagination_links_keep_the_feed_scope(client, alice):
_, content = _stream_article_ids(client, {"filter": "all", "feed": feed.pk})
assert _link_query(content, "Older")["feed"] == str(feed.pk)
def test_templates_use_no_inline_event_handlers():
"""Production's CSP (script-src 'self') blocks inline on*= handlers —
they work in dev, where no CSP is sent, and silently do nothing in
production. Behavior belongs in static/js."""
from pathlib import Path
from django.conf import settings
pattern = re.compile(r"""<[^>]+\s(on[a-z]+)\s*=""", re.IGNORECASE)
offenders = []
for template in Path(settings.BASE_DIR, "templates").rglob("*.html"):
for match in pattern.finditer(template.read_text()):
offenders.append(f"{template.name}: {match.group(1)}")
assert offenders == []
@pytest.mark.django_db
def test_article_detail_back_link_keeps_the_stream_scope(client, alice):
feed = Feed.objects.create(feed_url="https://example.com/feed.xml")
Subscription.objects.create(user=alice, feed=feed)
article = Article.objects.create(feed=feed, guid="g1", title="Hello")
client.login(username="alice", password="pw-alice-123")
stream = client.get(
reverse("reader:stream"), {"filter": "all", "feed": feed.pk}
).content.decode()
link = re.search(r'href="(/articles/\d+/\?[^"]+)"', stream)
assert link is not None
detail = client.get(link.group(1).replace("&amp;", "&")).content.decode()
assert f"filter=all&amp;feed={feed.pk}" in detail
assert "Back to stream" in detail
assert article.title in detail
@pytest.mark.django_db
def test_article_with_no_publish_date_shows_when_it_was_fetched(client, alice):
"""A feed that gives no date is treated as published when we fetched
it — in the list row, its group header, and the open article's header
— without storing an invented value on the article itself."""
feed = Feed.objects.create(feed_url="https://example.com/feed.xml")
Subscription.objects.create(user=alice, feed=feed)
article = Article.objects.create(feed=feed, guid="g1", title="No publish date")
assert article.published_at is None
client.login(username="alice", password="pw-alice-123")
iso = article.fetched_at.isoformat()
stream = client.get(reverse("reader:stream"), {"filter": "all"}).content.decode()
assert f'datetime="{iso}"' in stream
assert "Received today" in stream
detail = client.get(reverse("reader:article_detail", args=[article.pk])).content.decode()
assert f'datetime="{iso}"' in detail
article.refresh_from_db()
assert article.published_at is None
@pytest.mark.django_db
def test_new_since_requires_login(client):
response = client.get(reverse("reader:new_since"))
assert response.status_code == 302
@pytest.mark.django_db
def test_new_since_without_a_cursor_returns_the_baseline_id(client, alice):
feed = Feed.objects.create(feed_url="https://example.com/feed.xml")
Subscription.objects.create(user=alice, feed=feed)
article = Article.objects.create(feed=feed, guid="g1", title="Hello")
client.login(username="alice", password="pw-alice-123")
body = client.get(reverse("reader:new_since")).json()
assert body == {"latest_id": article.pk, "new_count": 0}
@pytest.mark.django_db
def test_new_since_counts_only_new_unread_visible_articles_from_my_feeds(client, alice, bob):
mine = Feed.objects.create(feed_url="https://mine.example.com/feed.xml")
theirs = Feed.objects.create(feed_url="https://theirs.example.com/feed.xml")
Subscription.objects.create(user=alice, feed=mine)
Subscription.objects.create(user=bob, feed=theirs)
cursor = Article.objects.create(feed=mine, guid="cursor", title="Cursor").pk
new_unread = Article.objects.create(feed=mine, guid="n1", title="New unread")
already_read = Article.objects.create(feed=mine, guid="n2", title="New but read")
hidden = Article.objects.create(feed=mine, guid="n3", title="New but hidden")
Article.objects.create(feed=theirs, guid="n4", title="Someone else's feed")
UserArticleState.objects.create(user=alice, article=already_read, is_read=True)
UserArticleState.objects.create(user=alice, article=hidden, is_hidden=True)
client.login(username="alice", password="pw-alice-123")
body = client.get(reverse("reader:new_since"), {"since": cursor}).json()
assert body["new_count"] == 1
assert new_unread.pk > cursor
@pytest.mark.django_db
def test_authenticated_pages_load_the_tab_badge_script_with_its_endpoint(client, alice):
client.login(username="alice", password="pw-alice-123")
content = client.get(reverse("reader:stream")).content.decode()
assert "js/tab-badge" in content
assert f'data-url="{reverse("reader:new_since")}"' in content
client.logout()
assert "tab-badge" not in client.get(reverse("accounts:login")).content.decode()
+1
View File
@@ -7,6 +7,7 @@ app_name = "reader"
urlpatterns = [
path("", views.stream, name="stream"),
path("check-new/", views.check_new_articles, name="check_new"),
path("new-since/", views.new_since, name="new_since"),
path("mark-all-read/", views.mark_all_read, name="mark_all_read"),
path("articles/<int:article_id>/", views.article_detail, name="article_detail"),
path(
+35 -3
View File
@@ -5,7 +5,7 @@ from urllib.parse import urlencode
from django.contrib.auth.decorators import login_required
from django.db.models import Max, Q
from django.db.models.functions import Coalesce
from django.http import Http404
from django.http import Http404, JsonResponse
from django.shortcuts import get_object_or_404, redirect, render
from django.urls import reverse
from django.utils import timezone
@@ -113,7 +113,8 @@ def _group_by_date(articles):
today = timezone.localdate()
def day_of(article):
return timezone.localtime(article.published_at).date() if article.published_at else None
when = article.published_at or article.fetched_at
return timezone.localtime(when).date() if when else None
return [
{"label": _date_label(day, today), "articles": list(items)}
@@ -238,6 +239,7 @@ def stream(request):
# regardless of how a feed backdates an entry's published date.
"latest_article_id": queryset.aggregate(Max("id"))["id__max"] or 0,
"new_check_qs": urlencode(scope_params),
"scope_qs": urlencode(scope_params),
}
return render(request, "reader/stream.html", context)
@@ -256,6 +258,29 @@ def check_new_articles(request):
return render(request, "reader/_new_articles_banner.html", {"new_count": new_count})
@login_required
def new_since(request):
"""Feeds the browser-tab dot (static/js/tab-badge.js) on every page:
without ?since it returns the current highest article id, which the
page keeps as its baseline; with it, how many unread, non-hidden
articles from the user's subscriptions have appeared after that id.
Deliberately not scoped to the stream's current filter/tag/feed — the
tab should light up for anything new, whichever page is open."""
latest_id = Article.objects.aggregate(Max("id"))["id__max"] or 0
try:
since_id = int(request.GET["since"])
except (KeyError, ValueError):
return JsonResponse({"latest_id": latest_id, "new_count": 0})
new_count = (
_base_queryset(request.user)
.filter(id__gt=since_id)
.exclude(user_states__user=request.user, user_states__is_read=True)
.count()
)
return JsonResponse({"latest_id": latest_id, "new_count": new_count})
@login_required
@require_POST
def mark_all_read(request):
@@ -311,7 +336,14 @@ def _load_article_and_mark_read(user, article_id):
@login_required
def article_detail(request, article_id):
article = _load_article_and_mark_read(request.user, article_id)
return render(request, "reader/article_detail.html", {"article": article})
# The stream passes its filter/tag/feed scope through the link so
# "Back to stream" returns to it instead of resetting to every feed.
back_params = {k: request.GET[k] for k in ("filter", "tag", "feed") if request.GET.get(k)}
return render(
request,
"reader/article_detail.html",
{"article": article, "back_qs": urlencode(back_params)},
)
@login_required
+22 -6
View File
@@ -314,17 +314,33 @@ misconfigured feed or an outage at the source.
`/health/` is an unauthenticated liveness check (DB connectivity only) —
`scripts/upgrade.sh` uses it to confirm a deploy actually came back up.
Point an external uptime monitor at `/health/ready/` instead: it also checks
Redis and that the background poller has ticked in the last five minutes
(proving Celery beat is scheduling *and* a worker is consuming), and returns
503 with a per-check breakdown when any of them fails — so "the site loads
but nothing is being fetched" actually alerts.
Point an external uptime monitor at `/health/ready/` instead. It returns 503
with a per-check breakdown when any of these fail:
- `database` and `redis` — reachable.
- `poll_heartbeat` — the background poller ticked in the last five minutes
(proves Celery beat is scheduling *and* a worker is consuming).
- `queue_depth` — no more than 1000 tasks waiting in the Celery queue
(workers aren't keeping up, or aren't running).
- `recent_fetch_success` — at least one feed has been fetched successfully
within three poll intervals (minimum 30 minutes), so "scheduled but every
fetch is failing" is caught. A brand-new install gets that long before it
counts as stale.
- `feed_failures` — no more than half of your active feeds are failing, once
you have at least four (a handful of dead feeds is normal; most of them
failing at once points at your server, DNS, or outbound network).
The payload also reports `queue_depth`, `active_feeds`, `failing_feeds`, and
the age of the last successful fetch, so you can graph them.
**Retention.** By default every fetched article is kept forever. In the Django
admin, "Site settings" has a *Retention days* field: set it to delete articles
fetched longer ago than that, nightly (03:30 UTC, via Celery beat). Articles
anyone has starred or saved for later are always kept; read state and tags for
deleted articles go with them. The dashboard shows the article count and the
deleted articles go with them. A small record (feed and entry id) is kept for
each deleted article so a feed that still lists it can't bring it back as new
and unread, and entries whose own publish date is already older than the window
aren't fetched in the first place. The dashboard shows the article count and the
age of the oldest one, so you can see what retention is doing.
**SQLite.** SQLite installs run in WAL mode with a 20-second busy timeout (set
+1
View File
@@ -559,6 +559,7 @@ form p {
font-size: 0.8rem;
color: var(--text-dim);
flex: 0 0 auto;
white-space: nowrap;
}
.article-actions {
+5 -1
View File
@@ -4,7 +4,7 @@
// instant plus a UTC-formatted fallback as their initial text, and this
// reformats them using the browser's own timezone, client-side, so the
// server never needs to know (or guess) where the viewer actually is.
var SHORT_OPTIONS = { hour: "numeric", minute: "2-digit" };
var SHORT_OPTIONS = { month: "short", day: "numeric", hour: "numeric", minute: "2-digit" };
var LONG_OPTIONS = {
year: "numeric",
month: "short",
@@ -21,6 +21,10 @@
var date = new Date(iso);
if (isNaN(date.getTime())) return;
var options = el.dataset.localFormat === "short" ? SHORT_OPTIONS : LONG_OPTIONS;
if (options === SHORT_OPTIONS && date.getFullYear() !== new Date().getFullYear()) {
// Date-only rows from a previous year would otherwise be ambiguous.
options = Object.assign({ year: "numeric" }, SHORT_OPTIONS);
}
try {
el.textContent = date.toLocaleString(undefined, options);
} catch (e) {
+23 -5
View File
@@ -1,8 +1,11 @@
function toggleArticle(event, id) {
event.preventDefault();
var link = event.currentTarget;
// Delegated listeners, not inline onclick="" attributes: production's
// Content Security Policy (config/settings/prod.py, script-src 'self')
// blocks inline event handlers, which silently turned the title click into
// a plain link to the detail page and the new-articles banner into a no-op.
function toggleArticle(link) {
var row = link.closest("li.article-row");
var body = document.getElementById("article-body-" + id);
var body = row.querySelector(".article-inline-body");
var wasOpen = row.classList.contains("is-expanded");
document.querySelectorAll("li.article-row.is-expanded").forEach(function (openRow) {
@@ -25,5 +28,20 @@ function toggleArticle(event, id) {
);
}
}
return false;
}
document.addEventListener("click", function (event) {
var link = event.target.closest("a.article-title[data-inline-url]");
if (link) {
// Leave modified clicks alone so "open in new tab" still works.
if (event.button !== 0 || event.ctrlKey || event.metaKey || event.shiftKey || event.altKey) {
return;
}
event.preventDefault();
toggleArticle(link);
return;
}
if (event.target.closest(".new-articles-banner")) {
location.reload();
}
});
+85
View File
@@ -0,0 +1,85 @@
// Marks the browser tab with a red dot on its favicon when new unread
// articles have arrived since this page loaded — visible from a background
// tab, unlike the in-page "new articles" banner. Runs on every
// authenticated page; a full page load resets the baseline (and the icon),
// which is exactly when the user has seen the new articles.
(function () {
var POLL_MS = 30000;
var url = document.currentScript && document.currentScript.dataset.url;
var icon = document.querySelector('link[rel="icon"]');
if (!url || !icon) return;
var originalHref = icon.href;
var baseline = null;
var badged = null; // the dotted icon's data: URL, drawn once on first need
var showing = false;
function drawDot() {
return new Promise(function (resolve) {
var image = new Image();
image.onload = function () {
var size = 64;
var canvas = document.createElement("canvas");
canvas.width = canvas.height = size;
var ctx = canvas.getContext("2d");
ctx.drawImage(image, 0, 0, size, size);
ctx.beginPath();
ctx.arc(size * 0.74, size * 0.26, size * 0.23, 0, 2 * Math.PI);
ctx.fillStyle = "#e53935";
ctx.fill();
ctx.lineWidth = size * 0.07;
ctx.strokeStyle = "#ffffff";
ctx.stroke();
resolve(canvas.toDataURL("image/png"));
};
image.onerror = function () {
resolve(null);
};
image.src = originalHref;
});
}
// Replace the <link> rather than editing href: some browsers ignore an
// in-place change to an existing favicon element.
function setIcon(href) {
var fresh = icon.cloneNode();
fresh.href = href;
icon.parentNode.replaceChild(fresh, icon);
icon = fresh;
}
function showDot(on) {
if (on === showing) return Promise.resolve();
showing = on;
if (!on) {
setIcon(originalHref);
return Promise.resolve();
}
var ready = badged ? Promise.resolve(badged) : drawDot();
return ready.then(function (href) {
badged = href;
if (href && showing) setIcon(href);
});
}
function poll() {
var query = baseline === null ? "" : "?since=" + baseline;
fetch(url + query, { credentials: "same-origin", headers: { Accept: "application/json" } })
.then(function (response) {
// A redirect means the session ended (we'd be reading the login
// page) — stop polling instead of hammering it.
if (!response.ok || response.redirected) throw new Error("stop");
return response.json();
})
.then(function (data) {
if (baseline === null) baseline = data.latest_id;
else showDot(data.new_count > 0);
setTimeout(poll, POLL_MS);
})
.catch(function (error) {
if (error.message !== "stop") setTimeout(poll, POLL_MS);
});
}
poll();
})();
+1
View File
@@ -58,6 +58,7 @@
{% if user.is_authenticated %}
<script src="{% static 'js/theme.js' %}" defer></script>
<script src="{% static 'js/sidebar-state.js' %}" defer></script>
<script src="{% static 'js/tab-badge.js' %}" data-url="{% url 'reader:new_since' %}" defer></script>
{% endif %}
<script src="{% static 'js/register-sw.js' %}" defer></script>
</body>
+2 -3
View File
@@ -3,9 +3,8 @@
<p class="article-meta">
{% if article.feed.favicon_url %}<img src="{{ article.feed.favicon_url }}" alt="" class="feed-favicon">{% endif %}
{{ article.feed.title }}
{% if article.published_at %}
&middot; <time datetime="{{ article.published_at|date:"c" }}" data-local-format="long">{{ article.published_at|date:"M j, Y H:i" }}</time>
{% endif %}
{# fetched_at stands in when the feed gave no publish date #}
&middot; <time datetime="{{ article.published_at|default:article.fetched_at|date:"c" }}" data-local-format="long">{{ article.published_at|default:article.fetched_at|date:"M j, Y H:i" }}</time>
{% if article.author %} &middot; {{ article.author }}{% endif %}
{% if article.link %} &middot; <a href="{{ article.link }}" target="_blank" rel="noopener noreferrer">Original &#8599;</a>{% endif %}
</p>
+5 -7
View File
@@ -1,17 +1,15 @@
<li id="article-{{ article.pk }}" class="article-row{% if article.is_read %} is-read{% endif %}"
data-published-utc="{% if article.published_at %}{{ article.published_at|date:"c" }}{% endif %}">
data-published-utc="{{ article.published_at|default:article.fetched_at|date:"c" }}">
<div class="article-row-main">
{% include "reader/_article_actions.html" %}
{% if article.feed.favicon_url %}
<img src="{{ article.feed.favicon_url }}" alt="" class="feed-favicon">
{% endif %}
<span class="article-feed-name">{{ article.feed.title|default:article.feed.feed_url }}</span>
<a href="{% url 'reader:article_detail' article.pk %}" class="article-title"
data-inline-url="{% url 'reader:article_inline' article.pk %}"
onclick="return toggleArticle(event, {{ article.pk }})">{{ article.title|default:"(untitled)" }}</a>
{% if article.published_at %}
<time class="article-time" datetime="{{ article.published_at|date:"c" }}" data-local-format="short">{{ article.published_at|date:"g:i A" }}</time>
{% endif %}
<a href="{% url 'reader:article_detail' article.pk %}{% if scope_qs %}?{{ scope_qs }}{% endif %}" class="article-title"
data-inline-url="{% url 'reader:article_inline' article.pk %}">{{ article.title|default:"(untitled)" }}</a>
{# fetched_at stands in when a feed gives no publish date — same fallback the stream's sort order uses #}
<time class="article-time" datetime="{{ article.published_at|default:article.fetched_at|date:"c" }}" data-local-format="short">{{ article.published_at|default:article.fetched_at|date:"M j, g:i A" }}</time>
</div>
<div id="article-body-{{ article.pk }}" class="article-inline-body" hidden></div>
</li>
+1 -1
View File
@@ -1,5 +1,5 @@
{% if new_count %}
<button type="button" class="new-articles-banner" onclick="location.reload()">
<button type="button" class="new-articles-banner">
{{ new_count }} new article{{ new_count|pluralize }} — click to show
</button>
{% endif %}
+1 -1
View File
@@ -1,7 +1,7 @@
{% extends "base.html" %}
{% block title %}{{ article.title|default:"Article" }} — Evection{% endblock %}
{% block content %}
<p><a href="{% url 'reader:stream' %}">&larr; Back to stream</a></p>
<p><a href="{% url 'reader:stream' %}{% if back_qs %}?{{ back_qs }}{% endif %}">&larr; Back to stream</a></p>
<article>
{% include "reader/_article_actions.html" %}
{% include "reader/_article_content.html" %}