Compare commits

...
Author SHA1 Message Date
ksmithandClaude Sonnet 5.5 0975a7634b Mark the browser tab with a red dot when new articles arrive (#31)
The in-page "new articles" banner is invisible from a background tab.
Every authenticated page now polls a small JSON endpoint
(reader:new_since, every 30s) and redraws the tab's favicon with a red
dot when unread, non-hidden articles from the user's feeds have
appeared since the page loaded. A reload (which is what clicking the
banner does) resets the baseline and the icon.

The endpoint is deliberately not scoped to the stream's current
filter/tag/feed, so the tab lights up for anything new whichever page
is open. Scope is the dot only: no desktop notifications, per the
request. Verified in a dev server enforcing the production CSP: the
icon swapped to the dotted version after a new article arrived and
reset on reload.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-11-01 10:18:29 -07:00
ksmithandClaude Sonnet 5.5 fd494a9c08 Make the readiness check catch fetching that is scheduled but failing (#27)
/health/ready/ proved beat and a worker were alive but not that
fetching worked: every feed could be failing, or the queue could be
hours behind, and it would still report ok.

apps.core.health adds three checks: Celery queue depth (real Redis list
length, over 1000 fails), a successful fetch within three poll intervals
(30 min minimum, with a grace period for a brand-new install), and the
share of active feeds failing (over half, once there are at least four).
The payload also reports queue depth, active/failing feed counts, and
the age of the last successful fetch. Verified against real Redis with
queued Celery tasks, and the suite on PostgreSQL. docs/install.md lists
each check.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-11-01 09:30:00 -07:00
ksmithandClaude Sonnet 5.5 f8cd1eb33e Stop retention from bringing deleted articles back (#29)
The nightly purge deleted articles, but a feed that still listed an
entry would re-insert it on its next poll as a brand-new, unread
article. For feeds with long archives, old items could reappear after
every retention run.

The purge now leaves a (feed, guid) tombstone per deleted article, and
ingestion skips tombstoned guids. With retention on, entries whose own
publish date is already outside the window are not ingested at all, so
a long archive isn't fetched only to be purged and tombstoned the same
night. Undated entries rely on the tombstone.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-11-01 09:30:00 -07:00
ksmithandClaude Sonnet 5.5 73dbee9400 Document running the test suite against PostgreSQL
The Postgres search backend's tests are skipped on SQLite, so they only
run when pointed at a real Postgres. Documents the throwaway-container
recipe used to verify the stored/GIN-indexed search vector migration.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-11-01 09:23:02 -07:00
ksmithandClaude Sonnet 5.5 6c6227af57 Treat an article with no publish date as published when we fetched it
Feeds that give no publish date left the row with no time at all and
grouped the article under "Undated". The stream's sort order already
fell back to fetched_at; the list row, its date-group header, and the
open article's header now use the same fallback. The article's own
published_at stays empty — nothing is invented in the database, so a
later refetch can't see a spurious change.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-11-01 09:14:09 -07:00
ksmithandClaude Sonnet 5.5 c339d456e4 Show the date as well as the time on each stream row
Rows only showed a time of day, leaving the date to the group header
above. Each row now shows "Oct 31, 5:30 PM" in the viewer's local
timezone, with the year added only for articles from a previous year.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-11-01 09:07:36 -07:00
ksmithandClaude Sonnet 5.5 7afda7d0ba Fix inline article expand and back-to-stream scope in production
Clicking an article title opened the detail page instead of expanding
it inline, and "Back to stream" then reset to every feed. Cause: the
title (and the new-articles banner) used inline onclick= handlers,
which production's CSP (script-src 'self') blocks — dev sends no CSP,
so it worked there. The title link just fell through to its href.

Behavior moves to delegated listeners in static/js/stream.js (modified
clicks still open in a new tab). A test now fails if any template gains
an inline on*= handler. The stream also passes its filter/tag/feed
scope through the article link so the detail page's back link returns
to it. Verified in a dev server enforcing the same CSP: title click
expands and collapses in place, URL unchanged.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-11-01 09:02:51 -07:00
ksmithandClaude Sonnet 5.5 37e658c49a Add configurable article retention (C7)
Articles and full HTML bodies accumulated forever, growing the
database, search index, and backups without bound. Site settings gets
a "Retention days" field (0, the default, keeps everything); a nightly
beat task deletes articles fetched longer ago than that in bounded
chunks, always keeping anything a user starred or saved for later.
Read state and tags cascade with the article and the FTS triggers drop
search entries. The dashboard shows the article count and oldest
article's age, and docs/install.md covers retention, the readiness
check, and SQLite's WAL tuning.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-11-01 08:51:20 -07:00
ksmithandClaude Sonnet 5.5 e5f1277014 Batch feed ingestion, add a readiness check, fail open without Redis (C2, C8)
Ingestion did a get_or_create per entry. It now prefetches existing
articles for the poll in one query, bulk-inserts new ones and
bulk-updates changed ones (falling back to row-by-row only if a
concurrent fetch wins a race), and dispatches rule evaluation in
bounded chunks so a huge first fetch is several small tasks.

/health/ stays a DB-only liveness check (upgrade.sh gates on it right
after restart). New /health/ready/ also checks Redis and that
poll_due_feeds ticked within five minutes — proving beat and a worker
are both alive — and returns 503 with per-check detail otherwise.

The sidebar unread-count cache and login throttle now degrade instead
of erroring when Redis is unreachable: every authenticated page and
every login would otherwise 500 during a cache outage. Found while
verifying in a sandbox with no Redis.

Also includes the dashboard's oldest-article metric (core/views.py).

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-11-01 08:51:20 -07:00
ksmithandClaude Sonnet 5.5 8d2900424a Use keyset pagination for the stream (C4)
The stream did a COUNT(*) and OFFSET slice on every request, ordered by
a non-unique (published_at, fetched_at) — deep pages got slower and
articles sharing a timestamp could repeat or vanish across pages.

It now orders by (Coalesce(published_at, fetched_at), id) and pages with
before/after cursors, fetching one extra row to know whether another
page exists instead of counting. Pagination links also keep the feed
scope, which the old links silently dropped.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-11-01 08:51:20 -07:00
ksmithandClaude Sonnet 5.5 70b0d4f277 Index Postgres search and harden SQLite search/concurrency (C5, C6)
Postgres recomputed a tsvector over every article's full text on every
search. A migration now adds a stored, GIN-indexed search_vector column
kept current by a trigger (so no ingest/update path can forget it), and
the backend reads that column. Untested here: no Postgres in the dev
sandbox, so the SQL is reviewed but not executed — worth one run of the
(skipped-on-SQLite) Postgres test module against a real instance.

SQLite search built one bound parameter per subscribed feed, so enough
feeds would exceed the variable limit, and the OperationalError was
swallowed into "no results". It now scopes through a subquery on
Subscription and logs failures. SQLite connections also get WAL mode,
a 20s busy timeout, and IMMEDIATE transactions so gunicorn, Celery, and
the FTS triggers stop tripping over each other.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-11-01 08:51:20 -07:00
43 changed files with 1453 additions and 105 deletions

No files matched your search

+12
View File
@@ -54,6 +54,18 @@ ruff check .
black --check .
```
### Testing against PostgreSQL
The suite runs on SQLite by default; the Postgres-only tests (search backend)
are skipped there. To run everything against a real Postgres:
```bash
docker run -d --name evection-pg-test -e POSTGRES_PASSWORD=testpw -e POSTGRES_DB=evection \
-p 127.0.0.1:55432:5432 postgres:16
DATABASE_URL=postgres://postgres:testpw@127.0.0.1:55432/evection pytest
docker rm -f evection-pg-test
```
## License
[GNU AGPL-3.0-or-later](LICENSE) — if you run a modified version of Evection as
+1
View File
@@ -45,4 +45,5 @@ def _setup_exempt_paths():
reverse("pwa:service_worker"),
reverse("pwa:offline"),
reverse("core:health"),
reverse("core:health_ready"),
}
+18
View File
@@ -72,3 +72,21 @@ def test_too_many_failed_attempts_against_one_username_blocks_that_username(clie
response = client.post(reverse("accounts:login"), {"username": "bob", "password": "pw-bob-123"})
assert response.status_code == 302
assert response.wsgi_request.user == bob
@pytest.mark.django_db
def test_login_still_works_when_the_throttle_cache_is_down(client, monkeypatch):
from unittest.mock import Mock
from django.contrib.auth import get_user_model
get_user_model().objects.create_user("alice", password="pw-alice-123")
broken = Mock(side_effect=ConnectionError("redis down"))
monkeypatch.setattr("apps.accounts.views.cache.get", broken)
monkeypatch.setattr("apps.accounts.views.cache.incr", broken)
monkeypatch.setattr("apps.accounts.views.cache.set", broken)
bad = client.post(reverse("accounts:login"), {"username": "alice", "password": "wrong"})
assert bad.status_code == 200
good = client.post(reverse("accounts:login"), {"username": "alice", "password": "pw-alice-123"})
assert good.status_code == 302
+14
View File
@@ -1,3 +1,5 @@
import logging
from django.contrib import messages
from django.contrib.auth import login
from django.contrib.auth.decorators import user_passes_test
@@ -10,6 +12,8 @@ from .forms import CreateUserForm, SetupForm
from .middleware import SetupWizardMiddleware
from .models import SetupLock, User
logger = logging.getLogger(__name__)
is_admin = user_passes_test(lambda u: u.is_authenticated and u.is_staff)
# Two independent throttles, either of which can block an attempt: by
@@ -41,14 +45,24 @@ def _attempts_key(kind: str, value: str) -> str:
def _too_many_attempts(key: str, limit: int) -> bool:
# Fails open if the cache (Redis) is unreachable: logging in must keep
# working through a cache outage, at the cost of no throttling until
# it's back — an outage shouldn't lock every user out of the app.
try:
return cache.get(key, 0) >= limit
except Exception:
logger.warning("Login throttle cache unavailable; not rate limiting", exc_info=True)
return False
def _record_failed_attempt(key: str) -> None:
try:
try:
cache.incr(key)
except ValueError:
cache.set(key, 1, LOGIN_RATE_LIMIT_WINDOW_SECONDS)
except Exception:
logger.warning("Login throttle cache unavailable; attempt not recorded", exc_info=True)
class LoginView(DjangoLoginView):
@@ -0,0 +1,27 @@
# Generated by Django 5.2.18 on 2026-11-01 16:28
import django.db.models.deletion
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
('articles', '0004_starred_saved_as_system_tags'),
('feeds', '0006_backfill_feed_next_fetch_at'),
]
operations = [
migrations.CreateModel(
name='PurgedArticle',
fields=[
('id', models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
('guid', models.CharField(max_length=1000)),
('purged_at', models.DateTimeField(auto_now_add=True)),
('feed', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, related_name='purged_articles', to='feeds.feed')),
],
options={
'unique_together': {('feed', 'guid')},
},
),
]
+17
View File
@@ -32,6 +32,23 @@ class Article(models.Model):
return self.title or self.guid
class PurgedArticle(models.Model):
"""Tombstone left by the retention purge (apps.articles.tasks.
purge_old_articles) for every article it deletes: a feed that still
lists an entry after we purged it would otherwise be re-inserted as a
brand-new, unread article on its next poll. Ingestion skips any guid
that has a tombstone here. Only the (feed, guid) pair is kept — a
small fraction of the article it replaces.
"""
feed = models.ForeignKey(Feed, on_delete=models.CASCADE, related_name="purged_articles")
guid = models.CharField(max_length=1000)
purged_at = models.DateTimeField(auto_now_add=True)
class Meta:
unique_together = [("feed", "guid")]
class UserArticleState(models.Model):
"""Per-user read/hide state for an Article.
+14
View File
@@ -1,3 +1,5 @@
import logging
from django.core.cache import cache
from django.db.models import Count
@@ -5,6 +7,8 @@ from apps.core.cache_utils import SIDEBAR_UNREAD_CACHE_TTL_SECONDS, sidebar_unre
from .models import SAVED_TAG_NAME, STARRED_TAG_NAME, Article, ArticleTag, UserArticleState
logger = logging.getLogger(__name__)
def with_user_state(articles, user):
"""Annotates an iterable of Article instances with is_read/is_starred/
@@ -56,7 +60,14 @@ def unread_counts_by_feed(user):
the current user's own read state, so this is rarely visibly stale.
"""
cache_key = sidebar_unread_cache_key(user.pk)
# The cache is an optimization here, never a dependency: every
# authenticated page render goes through this, so a Redis outage must
# degrade to recomputing the count, not to a 500 on every page.
try:
counts = cache.get(cache_key)
except Exception:
logger.warning("Sidebar cache unavailable; computing unread counts directly")
counts = None
if counts is not None:
return counts
@@ -68,5 +79,8 @@ def unread_counts_by_feed(user):
.annotate(unread=Count("id"))
)
counts = {row["feed_id"]: row["unread"] for row in rows}
try:
cache.set(cache_key, counts, SIDEBAR_UNREAD_CACHE_TTL_SECONDS)
except Exception:
logger.warning("Sidebar cache unavailable; skipping cache write")
return counts
+48 -2
View File
@@ -1,10 +1,15 @@
from celery import shared_task
from datetime import timedelta
from celery import shared_task
from django.db import transaction
from django.utils import timezone
from apps.core.models import SiteSettings
from apps.feeds.models import Subscription
from apps.rules.engine import evaluate_rules_for_article
from apps.rules.models import FilterRule
from .models import Article, SubscriptionTag
from .models import Article, ArticleTag, PurgedArticle, SubscriptionTag
@shared_task
@@ -57,3 +62,44 @@ def apply_filter_rules_for_feed(feed_id: int, article_ids: list[int]):
evaluated += 1
return evaluated
PURGE_CHUNK_SIZE = 500
@shared_task
def purge_old_articles():
"""Deletes articles older than SiteSettings.retention_days (C7), in
chunks so one run never holds a huge delete in a single transaction.
Anything a user starred or saved for later (a system ArticleTag) is
kept — those are explicit "keep this" signals. UserArticleState and
ArticleTag rows go with their article via ON DELETE CASCADE, and the
SQLite FTS triggers drop the search entries. 0 days means keep
everything forever (the default).
Each deleted article leaves a PurgedArticle tombstone first, so a feed
that still lists the entry can't bring it back as a new, unread
article on its next poll (see apps.feeds.services.fetcher).
"""
retention_days = SiteSettings.load().retention_days
if retention_days <= 0:
return 0
cutoff = timezone.now() - timedelta(days=retention_days)
keep = ArticleTag.objects.filter(tag__is_system=True).values("article_id")
deleted_total = 0
while True:
chunk = list(
Article.objects.filter(fetched_at__lt=cutoff)
.exclude(pk__in=keep)
.values_list("pk", "feed_id", "guid")[:PURGE_CHUNK_SIZE]
)
if not chunk:
return deleted_total
with transaction.atomic():
PurgedArticle.objects.bulk_create(
[PurgedArticle(feed_id=feed_id, guid=guid) for _pk, feed_id, guid in chunk],
ignore_conflicts=True,
)
Article.objects.filter(pk__in=[pk for pk, _feed_id, _guid in chunk]).delete()
deleted_total += len(chunk)
+16
View File
@@ -58,3 +58,19 @@ def test_unread_counts_by_feed_cache_is_per_user(alice, feed):
assert unread_counts_by_feed(alice) == {feed.pk: 1}
assert unread_counts_by_feed(bob) == {feed.pk: 1}
@pytest.mark.django_db
def test_unread_counts_still_work_when_the_cache_is_down(alice, feed, monkeypatch):
"""A Redis outage must degrade the sidebar to recomputing counts, not
500 every authenticated page."""
from unittest.mock import Mock
Subscription.objects.create(user=alice, feed=feed)
Article.objects.create(feed=feed, guid="a-1")
broken = Mock(side_effect=ConnectionError("redis down"))
monkeypatch.setattr("apps.articles.queries.cache.get", broken)
monkeypatch.setattr("apps.articles.queries.cache.set", broken)
assert unread_counts_by_feed(alice) == {feed.pk: 1}
invalidate_sidebar_unread_cache(alice.pk)
+91
View File
@@ -0,0 +1,91 @@
from datetime import timedelta
import pytest
from django.contrib.auth import get_user_model
from django.utils import timezone
from apps.core.models import SiteSettings
from apps.feeds.models import Feed
from .models import STARRED_TAG_NAME, Article, ArticleTag, UserArticleState, get_system_tag
from .tasks import purge_old_articles
User = get_user_model()
def _article(feed, guid, age_days):
article = Article.objects.create(feed=feed, guid=guid, title=guid)
Article.objects.filter(pk=article.pk).update(
fetched_at=timezone.now() - timedelta(days=age_days)
)
return article
@pytest.fixture
def feed(db):
return Feed.objects.create(feed_url="https://example.com/feed.xml")
@pytest.mark.django_db
def test_purge_is_a_noop_with_the_default_keep_forever_setting(feed):
_article(feed, "ancient", age_days=5000)
assert purge_old_articles() == 0
assert Article.objects.count() == 1
@pytest.mark.django_db
def test_purge_deletes_only_articles_older_than_the_retention_window(feed):
SiteSettings.objects.create(retention_days=30)
old = _article(feed, "old", age_days=45)
recent = _article(feed, "recent", age_days=5)
assert purge_old_articles() == 1
assert not Article.objects.filter(pk=old.pk).exists()
assert Article.objects.filter(pk=recent.pk).exists()
@pytest.mark.django_db
def test_purge_keeps_starred_articles_and_cascades_user_state(feed):
SiteSettings.objects.create(retention_days=30)
user = User.objects.create_user("alice", password="pw-alice-123")
starred = _article(feed, "starred-old", age_days=90)
ArticleTag.objects.create(
user=user, article=starred, tag=get_system_tag(user, STARRED_TAG_NAME)
)
doomed = _article(feed, "plain-old", age_days=90)
state = UserArticleState.objects.create(user=user, article=doomed, is_read=True)
purge_old_articles()
assert Article.objects.filter(pk=starred.pk).exists()
assert not Article.objects.filter(pk=doomed.pk).exists()
assert not UserArticleState.objects.filter(pk=state.pk).exists()
@pytest.mark.django_db
def test_purge_works_through_more_than_one_chunk(feed, monkeypatch):
monkeypatch.setattr("apps.articles.tasks.PURGE_CHUNK_SIZE", 2)
SiteSettings.objects.create(retention_days=30)
for i in range(5):
_article(feed, f"old-{i}", age_days=60)
assert purge_old_articles() == 5
assert Article.objects.count() == 0
@pytest.mark.django_db
def test_purge_leaves_a_tombstone_per_deleted_article(feed):
from .models import PurgedArticle
SiteSettings.objects.create(retention_days=30)
_article(feed, "gone-1", age_days=60)
_article(feed, "gone-2", age_days=60)
_article(feed, "kept", age_days=1)
purge_old_articles()
assert set(PurgedArticle.objects.filter(feed=feed).values_list("guid", flat=True)) == {
"gone-1",
"gone-2",
}
+1 -1
View File
@@ -14,7 +14,7 @@ class SiteSettingsAdmin(admin.ModelAdmin):
install. No changelist/add/delete views exist at all here.
"""
fields = ["fetch_interval_minutes"]
fields = ["fetch_interval_minutes", "retention_days"]
def has_add_permission(self, request):
return False
+7
View File
@@ -1,5 +1,9 @@
import logging
from django.core.cache import cache
logger = logging.getLogger(__name__)
# Short enough that a missed invalidation point self-heals within one
# page load's worth of staleness; long enough to actually absorb repeat
# authenticated-page renders (apps.core.context_processors.sidebar runs
@@ -19,4 +23,7 @@ def invalidate_sidebar_unread_cache(user_id: int) -> None:
out the TTL above. Rule-driven state changes during background feed
ingestion aren't invalidated this way — those self-heal within the
TTL, same as a missed call site here would."""
try:
cache.delete(sidebar_unread_cache_key(user_id))
except Exception:
logger.warning("Sidebar cache unavailable; nothing to invalidate")
+83
View File
@@ -0,0 +1,83 @@
from datetime import timedelta
import redis
from django.conf import settings
from django.db.models import Max, Min
from django.utils import timezone
from apps.feeds.models import Feed
from .models import SiteSettings
# More than this many tasks waiting means workers aren't keeping up (or
# aren't running) — fetches are piling up faster than they complete.
MAX_QUEUE_DEPTH = 1000
# Past this share of active feeds failing, it's far more likely something
# on our side (DNS, outbound network, a bad deploy) than every publisher
# breaking at once. Needs a handful of feeds first so one broken feed out
# of two doesn't read as an outage.
MAX_FAILING_RATIO = 0.5
MIN_FEEDS_FOR_RATIO = 4
MIN_SUCCESS_WINDOW_MINUTES = 30
def queue_depth() -> int | None:
"""Tasks waiting in Celery's default queue, or None if the broker
can't be reached. Celery's Redis transport keeps each queue as a Redis
list named after the queue."""
try:
client = redis.Redis.from_url(
settings.CELERY_BROKER_URL, socket_connect_timeout=2, socket_timeout=2
)
return int(client.llen("celery"))
except Exception:
return None
def _success_window() -> timedelta:
floor = getattr(settings, "EVECTION_MIN_FETCH_INTERVAL_MINUTES", 15)
base = max(SiteSettings.load().fetch_interval_minutes, floor)
return timedelta(minutes=max(3 * base, MIN_SUCCESS_WINDOW_MINUTES))
def fetch_pipeline_status(now=None) -> tuple[dict[str, bool], dict]:
"""Checks that fetching is actually *succeeding*, not just scheduled
(the poll heartbeat only proves beat and a worker are alive): returns
({check: passed}, detail) covering queue depth, whether any feed has
been fetched successfully recently, and the share of feeds failing.
"""
now = now or timezone.now()
detail: dict = {}
depth = queue_depth()
detail["queue_depth"] = depth
checks = {"queue_depth": depth is not None and depth <= MAX_QUEUE_DEPTH}
active = Feed.objects.filter(is_active=True)
active_count = active.count()
failing = active.filter(fetch_error_count__gt=0).count()
detail["active_feeds"] = active_count
detail["failing_feeds"] = failing
window = _success_window()
last_success = active.filter(
last_fetch_status__in=[Feed.STATUS_OK, Feed.STATUS_NOT_MODIFIED]
).aggregate(latest=Max("last_fetched_at"))["latest"]
if last_success is not None:
detail["last_successful_fetch_age_seconds"] = int((now - last_success).total_seconds())
recent_success = now - last_success <= window
elif active_count == 0:
recent_success = True
else:
# Nothing has ever fetched successfully — only a problem once
# the oldest feed has had a full window to be fetched.
oldest = active.aggregate(oldest=Min("created_at"))["oldest"]
recent_success = now - oldest <= window
checks["recent_fetch_success"] = recent_success
checks["feed_failures"] = not (
active_count >= MIN_FEEDS_FOR_RATIO and failing / active_count > MAX_FAILING_RATIO
)
return checks, detail
@@ -0,0 +1,18 @@
# Generated by Django 5.2.18 on 2026-11-01 15:48
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
('core', '0001_initial'),
]
operations = [
migrations.AddField(
model_name='sitesettings',
name='retention_days',
field=models.PositiveIntegerField(default=0, help_text="Articles older than this many days (by when they were fetched) are deleted daily, along with everyone's read state and tags for them — except articles anyone has starred or saved for later. 0 keeps everything forever."),
),
]
+10
View File
@@ -18,6 +18,16 @@ class SiteSettings(models.Model):
),
)
retention_days = models.PositiveIntegerField(
default=0,
help_text=(
"Articles older than this many days (by when they were fetched) are "
"deleted daily, along with everyone's read state and tags for them — "
"except articles anyone has starred or saved for later. 0 keeps "
"everything forever."
),
)
class Meta:
verbose_name = "Site settings"
verbose_name_plural = "Site settings"
+175
View File
@@ -22,6 +22,61 @@ def test_health_check_works_before_first_run_setup(client):
assert response.status_code == 200
@pytest.fixture
def broker_queue(monkeypatch):
"""Stands in for the Redis broker's queue length (tests have no Redis):
set `broker_queue["depth"]` to a number, or None for "unreachable"."""
state = {"depth": 0}
monkeypatch.setattr("apps.core.health.queue_depth", lambda: state["depth"])
return state
@pytest.mark.django_db
def test_health_ready_is_degraded_until_a_poll_tick_has_happened(client, broker_queue):
response = client.get(reverse("core:health_ready"))
assert response.status_code == 503
body = response.json()
assert body["status"] == "degraded"
assert body["checks"]["database"] is True
assert body["checks"]["redis"] is True
assert body["checks"]["poll_heartbeat"] is False
@pytest.mark.django_db
def test_health_ready_is_ok_after_poll_due_feeds_runs(client, broker_queue):
from apps.feeds.tasks import poll_due_feeds
poll_due_feeds()
response = client.get(reverse("core:health_ready"))
assert response.status_code == 200
checks = response.json()["checks"]
assert checks == {
"database": True,
"redis": True,
"poll_heartbeat": True,
"queue_depth": True,
"recent_fetch_success": True,
"feed_failures": True,
}
@pytest.mark.django_db
def test_health_ready_flags_a_stale_poll_heartbeat(client, broker_queue):
from datetime import timedelta
from django.core.cache import cache
from django.utils import timezone
from apps.feeds.tasks import POLL_HEARTBEAT_CACHE_KEY
cache.set(POLL_HEARTBEAT_CACHE_KEY, timezone.now() - timedelta(minutes=30))
response = client.get(reverse("core:health_ready"))
assert response.status_code == 503
assert response.json()["checks"]["poll_heartbeat"] is False
@pytest.mark.django_db
def test_dashboard_requires_staff(client, django_user_model):
django_user_model.objects.create_user("regular", password="pw-regular-123")
@@ -77,3 +132,123 @@ def test_retry_feed_resets_backoff_and_requeues(client, django_user_model):
assert response.status_code == 302
mock_reset.assert_called_once_with(feed.id)
mock_fetch.assert_called_once_with(feed.id)
def _ready(client):
from django.core.cache import cache
from django.utils import timezone
from apps.feeds.tasks import POLL_HEARTBEAT_CACHE_KEY
# Record the heartbeat directly: running poll_due_feeds here would
# execute fetches eagerly (CELERY_TASK_ALWAYS_EAGER) and rewrite the
# feed states each test sets up.
cache.set(POLL_HEARTBEAT_CACHE_KEY, timezone.now())
response = client.get(reverse("core:health_ready"))
return response.status_code, response.json()
@pytest.mark.django_db
def test_health_ready_flags_a_backed_up_queue(client, broker_queue):
from .health import MAX_QUEUE_DEPTH
broker_queue["depth"] = MAX_QUEUE_DEPTH + 1
status, body = _ready(client)
assert status == 503
assert body["checks"]["queue_depth"] is False
assert body["queue_depth"] == MAX_QUEUE_DEPTH + 1
@pytest.mark.django_db
def test_health_ready_flags_an_unreachable_broker_queue(client, broker_queue):
broker_queue["depth"] = None
status, body = _ready(client)
assert status == 503
assert body["checks"]["queue_depth"] is False
@pytest.mark.django_db
def test_health_ready_flags_no_recent_successful_fetch(client, broker_queue):
from datetime import timedelta
from django.utils import timezone
feed = Feed.objects.create(
feed_url="https://a.example.com/feed.xml",
last_fetch_status=Feed.STATUS_OK,
last_fetched_at=timezone.now() - timedelta(days=2),
)
Feed.objects.filter(pk=feed.pk).update(created_at=timezone.now() - timedelta(days=30))
status, body = _ready(client)
assert status == 503
assert body["checks"]["recent_fetch_success"] is False
assert body["last_successful_fetch_age_seconds"] > 86400
@pytest.mark.django_db
def test_health_ready_is_ok_with_a_recent_successful_fetch(client, broker_queue):
from django.utils import timezone
Feed.objects.create(
feed_url="https://a.example.com/feed.xml",
last_fetch_status=Feed.STATUS_NOT_MODIFIED,
last_fetched_at=timezone.now(),
)
status, body = _ready(client)
assert status == 200
assert body["active_feeds"] == 1
assert body["failing_feeds"] == 0
@pytest.mark.django_db
def test_health_ready_gives_a_brand_new_feed_time_before_it_counts_as_stale(client, broker_queue):
Feed.objects.create(feed_url="https://new.example.com/feed.xml")
status, body = _ready(client)
assert status == 200
assert body["checks"]["recent_fetch_success"] is True
@pytest.mark.django_db
def test_health_ready_flags_most_feeds_failing_but_not_a_single_broken_one(client, broker_queue):
from django.utils import timezone
now = timezone.now()
healthy = Feed.objects.create(
feed_url="https://ok.example.com/feed.xml",
last_fetch_status=Feed.STATUS_OK,
last_fetched_at=now,
)
broken = [
Feed.objects.create(
feed_url=f"https://bad{i}.example.com/feed.xml",
last_fetch_status=Feed.STATUS_ERROR,
fetch_error_count=3,
last_fetched_at=now,
)
for i in range(2)
]
status, body = _ready(client)
assert status == 200, "2 of 3 failing is below the minimum feed count for the ratio"
Feed.objects.create(
feed_url="https://bad9.example.com/feed.xml",
last_fetch_status=Feed.STATUS_ERROR,
fetch_error_count=1,
last_fetched_at=now,
)
status, body = _ready(client)
assert status == 503
assert body["checks"]["feed_failures"] is False
assert body["failing_feeds"] == 3
assert healthy.pk and broken
+1
View File
@@ -6,6 +6,7 @@ app_name = "core"
urlpatterns = [
path("health/", views.health, name="health"),
path("health/ready/", views.health_ready, name="health_ready"),
path("dashboard/", views.dashboard, name="dashboard"),
path("dashboard/feeds/<int:feed_id>/retry/", views.retry_feed, name="retry_feed"),
]
+59 -3
View File
@@ -1,20 +1,29 @@
from django.contrib import messages
from django.contrib.auth.decorators import user_passes_test
from django.core.cache import cache
from django.db import connection
from django.db.models import Count, Q
from django.db.models import Count, Min, Q
from django.http import JsonResponse
from django.shortcuts import get_object_or_404, redirect, render
from django.utils import timezone
from django.views.decorators.http import require_POST
from apps.articles.models import Article
from apps.feeds.models import Feed, Subscription
from apps.feeds.tasks import fetch_feed, reset_feed_backoff
from apps.feeds.tasks import POLL_HEARTBEAT_CACHE_KEY, fetch_feed, reset_feed_backoff
from .health import fetch_pipeline_status
is_admin = user_passes_test(lambda u: u.is_authenticated and u.is_staff)
# Beat ticks poll_due_feeds every minute; this many minutes without a tick
# means beat or the worker (or the broker between them) isn't working.
POLL_HEARTBEAT_MAX_AGE_SECONDS = 5 * 60
def health(request):
"""Unauthenticated liveness/readiness check — reachable even before
"""Unauthenticated liveness check — reachable even before
first-run setup (see SetupWizardMiddleware). Used by scripts/upgrade.sh
to confirm a freshly restarted deploy actually came up, and is a
reasonable target for any external uptime monitor too."""
@@ -26,6 +35,52 @@ def health(request):
return JsonResponse({"status": "ok"})
def health_ready(request):
"""Readiness: liveness plus everything background fetching depends on
(C8) — the database, the Redis cache/broker, and a recent
poll_due_feeds tick, which proves beat is scheduling AND a worker is
consuming. Beyond that tick, apps.core.health checks fetching is
actually succeeding: queue depth, a recent successful fetch, and the
share of feeds failing. /health/ above stays DB-only on purpose: scripts/upgrade.sh
gates a deploy on it right after restart, before beat has had a
chance to tick. Point an uptime monitor at this one instead."""
checks = {"database": False, "redis": False, "poll_heartbeat": False}
detail = {}
try:
with connection.cursor() as cursor:
cursor.execute("SELECT 1")
checks["database"] = True
except Exception:
pass
try:
cache.set("health-ready-probe", "1", timeout=10)
checks["redis"] = cache.get("health-ready-probe") == "1"
except Exception:
pass
try:
last_tick = cache.get(POLL_HEARTBEAT_CACHE_KEY)
except Exception:
last_tick = None
if last_tick is not None:
age = (timezone.now() - last_tick).total_seconds()
detail["poll_heartbeat_age_seconds"] = int(age)
checks["poll_heartbeat"] = age <= POLL_HEARTBEAT_MAX_AGE_SECONDS
if checks["database"]:
pipeline_checks, pipeline_detail = fetch_pipeline_status()
checks.update(pipeline_checks)
detail.update(pipeline_detail)
ok = all(checks.values())
return JsonResponse(
{"status": "ok" if ok else "degraded", "checks": checks, **detail},
status=200 if ok else 503,
)
@is_admin
def dashboard(request):
feeds = Feed.objects.annotate(subscriber_count=Count("subscriptions"))
@@ -39,6 +94,7 @@ def dashboard(request):
"error_feed_count": erroring_feeds.count(),
"subscription_count": Subscription.objects.count(),
"article_count": Article.objects.count(),
"oldest_article_at": Article.objects.aggregate(oldest=Min("fetched_at"))["oldest"],
}
return render(
+90 -33
View File
@@ -11,16 +11,18 @@ import hashlib
import logging
from calendar import timegm
from dataclasses import dataclass
from datetime import datetime
from datetime import datetime, timedelta
from datetime import timezone as dt_timezone
import feedparser
import nh3
import requests
from django.conf import settings
from django.db import IntegrityError, transaction
from django.utils import timezone
from apps.articles.models import Article
from apps.articles.models import Article, PurgedArticle
from apps.core.models import SiteSettings
from apps.feeds.models import Feed
from .ssrf_guard import SSRFBlocked, safe_get
@@ -131,6 +133,26 @@ def _entry_content_html(entry) -> str:
return getattr(entry, "summary", "") or ""
UPDATABLE_ARTICLE_FIELDS = (
"title",
"link",
"author",
"summary",
"content_html",
"published_at",
"content_hash",
)
def _ensure_pks(feed: Feed, articles: list[Article]) -> list[int]:
"""bulk_create only populates primary keys on backends that can
RETURN them (Postgres, SQLite 3.35+); look the rest up by guid."""
if all(a.pk for a in articles):
return [a.pk for a in articles]
guids = [a.guid for a in articles]
return list(Article.objects.filter(feed=feed, guid__in=guids).values_list("pk", flat=True))
def effective_interval_minutes(feed: Feed, configured_interval_minutes: int) -> int:
"""The actual polling interval to use for `feed`, applying exponential
backoff on top of the site-wide `configured_interval_minutes` (see
@@ -217,52 +239,87 @@ def fetch_and_ingest(feed: Feed) -> FetchResult:
)
entries = entries[:MAX_ENTRIES_PER_POLL]
# One prefetch + bulk writes for the whole poll (C2), instead of a
# get_or_create (SELECT, then INSERT) per entry. Entries are keyed by
# guid so a feed that repeats one within a single response is only
# processed once.
parsed_entries: dict[str, dict] = {}
for entry in entries:
guid = _entry_guid(entry)
if guid in parsed_entries:
continue
title = (getattr(entry, "title", "") or "")[:1000]
link = (getattr(entry, "link", "") or "")[:2000]
author = (getattr(entry, "author", "") or "")[:500]
summary = (getattr(entry, "summary", "") or "")[:5000]
content_html = sanitize_html(_entry_content_html(entry))
published_at = _entry_published_at(entry)
content_hash = _content_hash(title, link, author, summary, content_html, published_at)
article, created = Article.objects.get_or_create(
feed=feed,
guid=guid,
defaults={
"link": link,
parsed_entries[guid] = {
"title": title,
"link": link,
"author": author,
"content_html": content_html,
"summary": summary,
"content_hash": content_hash,
"content_html": content_html,
"published_at": published_at,
},
)
"content_hash": _content_hash(title, link, author, summary, content_html, published_at),
}
if created:
new_ids.append(article.pk)
elif article.content_hash != content_hash:
article.title = title
article.link = link
article.author = author
article.summary = summary
article.content_html = content_html
article.published_at = published_at
article.content_hash = content_hash
article.save(
update_fields=[
"title",
"link",
"author",
"summary",
"content_html",
"published_at",
"content_hash",
# Entries the retention purge already deleted (tombstoned) and, when
# retention is on, entries so old by their own publish date that the
# next purge would delete them again, are not (re-)ingested — see
# apps.articles.tasks.purge_old_articles.
retention_days = SiteSettings.load().retention_days
if retention_days > 0:
too_old_before = timezone.now() - timedelta(days=retention_days)
for guid in [
g
for g, f in parsed_entries.items()
if f["published_at"] is not None and f["published_at"] < too_old_before
]:
del parsed_entries[guid]
for guid in PurgedArticle.objects.filter(feed=feed, guid__in=parsed_entries).values_list(
"guid", flat=True
):
del parsed_entries[guid]
existing = {a.guid: a for a in Article.objects.filter(feed=feed, guid__in=parsed_entries)}
to_create = [
Article(feed=feed, guid=guid, **fields)
for guid, fields in parsed_entries.items()
if guid not in existing
]
to_update = []
for guid, article in existing.items():
fields = parsed_entries[guid]
if article.content_hash != fields["content_hash"]:
for name, value in fields.items():
setattr(article, name, value)
to_update.append(article)
if to_update:
Article.objects.bulk_update(to_update, list(UPDATABLE_ARTICLE_FIELDS), batch_size=100)
updated_ids.extend(a.pk for a in to_update)
if to_create:
try:
with transaction.atomic():
Article.objects.bulk_create(to_create)
except IntegrityError:
# Another worker inserted some of these guids between the
# prefetch above and now (two fetches of the same feed can
# overlap) — fall back to row-by-row so nothing is lost or
# duplicated, at the cost of the old per-entry round trips
# for this rare case only.
for article in to_create:
fields = {name: getattr(article, name) for name in UPDATABLE_ARTICLE_FIELDS}
obj, created = Article.objects.get_or_create(
feed=feed, guid=article.guid, defaults=fields
)
updated_ids.append(article.pk)
if created:
new_ids.append(obj.pk)
else:
new_ids.extend(_ensure_pks(feed, to_create))
return FetchResult(
status=Feed.STATUS_OK, new_article_ids=new_ids, updated_article_ids=updated_ids
+15 -1
View File
@@ -4,6 +4,7 @@ from io import BytesIO
from celery import shared_task
from django.contrib.auth import get_user_model
from django.core.cache import cache
from django.db.models import Q
from django.utils import timezone
@@ -27,6 +28,12 @@ CLAIM_STALE_MINUTES = 10
# across several beat ticks instead of enqueuing everything — and every
# bounded Feed query below stays index-only work, not a full scan (C1).
POLL_BATCH_SIZE = 500
RULE_DISPATCH_CHUNK_SIZE = 50
# Written on every poll_due_feeds tick; apps.core.views.health_ready
# reads it to tell whether beat is scheduling and a worker is consuming.
POLL_HEARTBEAT_CACHE_KEY = "poll-due-feeds-heartbeat"
POLL_HEARTBEAT_TTL_SECONDS = 3600
@shared_task
@@ -39,6 +46,7 @@ def poll_due_feeds():
here, not via one Celery Beat entry per feed.
"""
now = timezone.now()
cache.set(POLL_HEARTBEAT_CACHE_KEY, now, timeout=POLL_HEARTBEAT_TTL_SECONDS)
claim_stale_before = now - timedelta(minutes=CLAIM_STALE_MINUTES)
due_feed_ids = list(
Feed.objects.filter(is_active=True)
@@ -125,7 +133,13 @@ def fetch_feed(feed_id: int):
len(result.new_article_ids),
len(result.updated_article_ids),
)
apply_filter_rules_for_feed.delay(feed_id, changed_article_ids)
# Chunked so one huge poll (a first fetch of a long archive feed)
# becomes several bounded rule-evaluation tasks, each well inside
# the worker's time limits, instead of one unbounded one (C2).
for start in range(0, len(changed_article_ids), RULE_DISPATCH_CHUNK_SIZE):
apply_filter_rules_for_feed.delay(
feed_id, changed_article_ids[start : start + RULE_DISPATCH_CHUNK_SIZE]
)
return {
"status": result.status,
+135
View File
@@ -264,3 +264,138 @@ def test_fetch_only_processes_the_first_max_entries_per_poll(feed):
assert len(result.new_article_ids) == MAX_ENTRIES_PER_POLL
assert Article.objects.filter(feed=feed).count() == MAX_ENTRIES_PER_POLL
assert not Article.objects.filter(feed=feed, guid=f"guid-{MAX_ENTRIES_PER_POLL}").exists()
def _feed_xml(count, title_suffix=""):
items = "".join(
f"<item><title>Item {i}{title_suffix}</title><guid>guid-{i}</guid></item>"
for i in range(count)
)
return (
f'<?xml version="1.0"?><rss version="2.0"><channel><title>T</title>{items}</channel></rss>'
)
@pytest.mark.django_db
@responses.activate
def test_ingest_uses_a_constant_number_of_article_queries(feed):
"""C2: one prefetch + bulk writes per poll, not a get_or_create per
entry — query count must not grow with the number of entries."""
from django.db import connection
from django.test.utils import CaptureQueriesContext
responses.add(responses.GET, FEED_URL, body=_feed_xml(40), status=200)
with CaptureQueriesContext(connection) as ctx:
result = fetch_and_ingest(feed)
assert len(result.new_article_ids) == 40
assert len(ctx.captured_queries) < 10
assert Article.objects.filter(feed=feed).count() == 40
@pytest.mark.django_db
@responses.activate
def test_ingest_dedups_a_guid_repeated_within_one_response(feed):
xml = (
'<?xml version="1.0"?><rss version="2.0"><channel><title>T</title>'
"<item><title>A</title><guid>same</guid></item>"
"<item><title>B</title><guid>same</guid></item>"
"</channel></rss>"
)
responses.add(responses.GET, FEED_URL, body=xml, status=200)
result = fetch_and_ingest(feed)
assert len(result.new_article_ids) == 1
assert Article.objects.filter(feed=feed).count() == 1
@pytest.mark.django_db
@responses.activate
def test_ingest_reports_updates_separately_from_new_articles(feed):
responses.add(responses.GET, FEED_URL, body=_feed_xml(3), status=200)
fetch_and_ingest(feed)
responses.replace(responses.GET, FEED_URL, body=_feed_xml(4, " edited"), status=200)
result = fetch_and_ingest(feed)
assert len(result.new_article_ids) == 1
assert len(result.updated_article_ids) == 3
assert Article.objects.get(feed=feed, guid="guid-0").title == "Item 0 edited"
@pytest.mark.django_db
@responses.activate
def test_ingest_falls_back_to_row_by_row_when_a_concurrent_insert_wins(feed):
from unittest.mock import patch
from django.db import IntegrityError
responses.add(responses.GET, FEED_URL, body=_feed_xml(3), status=200)
with patch.object(Article.objects, "bulk_create", side_effect=IntegrityError("race")):
result = fetch_and_ingest(feed)
assert len(result.new_article_ids) == 3
assert Article.objects.filter(feed=feed).count() == 3
@pytest.mark.django_db
@responses.activate
def test_purged_article_is_not_resurrected_by_a_feed_that_still_lists_it(feed):
"""#29: a feed still listing an entry the retention purge deleted must
not bring it back as a brand-new, unread article."""
from apps.articles.tasks import purge_old_articles
from apps.core.models import SiteSettings
SiteSettings.objects.create(retention_days=30)
responses.add(responses.GET, FEED_URL, body=_feed_xml(3), status=200)
fetch_and_ingest(feed)
Article.objects.filter(feed=feed, guid="guid-0").update(
fetched_at=timezone.now() - dt.timedelta(days=90)
)
assert purge_old_articles() == 1
assert not Article.objects.filter(feed=feed, guid="guid-0").exists()
result = fetch_and_ingest(feed)
assert result.new_article_ids == []
assert not Article.objects.filter(feed=feed, guid="guid-0").exists()
assert Article.objects.filter(feed=feed).count() == 2
@pytest.mark.django_db
@responses.activate
def test_entries_older_than_the_retention_window_are_not_ingested(feed):
"""With retention on, a feed's long archive isn't ingested only to be
purged (and tombstoned) the same night."""
from apps.core.models import SiteSettings
SiteSettings.objects.create(retention_days=30)
xml = (
'<?xml version="1.0"?><rss version="2.0"><channel><title>T</title>'
"<item><title>Ancient</title><guid>old</guid>"
"<pubDate>Mon, 01 Jan 2024 00:00:00 GMT</pubDate></item>"
"<item><title>Undated</title><guid>undated</guid></item>"
"</channel></rss>"
)
responses.add(responses.GET, FEED_URL, body=xml, status=200)
result = fetch_and_ingest(feed)
assert len(result.new_article_ids) == 1
assert Article.objects.filter(feed=feed).get().guid == "undated"
@pytest.mark.django_db
@responses.activate
def test_old_entries_are_ingested_normally_when_retention_is_off(feed):
xml = (
'<?xml version="1.0"?><rss version="2.0"><channel><title>T</title>'
"<item><title>Ancient</title><guid>old</guid>"
"<pubDate>Mon, 01 Jan 2024 00:00:00 GMT</pubDate></item>"
"</channel></rss>"
)
responses.add(responses.GET, FEED_URL, body=xml, status=200)
assert len(fetch_and_ingest(feed).new_article_ids) == 1
+20
View File
@@ -208,3 +208,23 @@ def test_fetch_feed_schedules_the_next_fetch_using_the_site_wide_interval():
assert feed.next_fetch_at is not None
expected = before + timedelta(minutes=120)
assert abs((feed.next_fetch_at - expected).total_seconds()) < 5
@pytest.mark.django_db
@responses.activate
def test_fetch_feed_dispatches_rule_evaluation_in_bounded_chunks(monkeypatch):
monkeypatch.setattr("apps.feeds.tasks.RULE_DISPATCH_CHUNK_SIZE", 2)
feed = Feed.objects.create(feed_url="https://example.com/feed.xml")
items = "".join(f"<item><title>T{i}</title><guid>g{i}</guid></item>" for i in range(5))
responses.add(
responses.GET,
feed.feed_url,
body=f'<?xml version="1.0"?><rss version="2.0"><channel>{items}</channel></rss>',
status=200,
)
with patch("apps.feeds.tasks.apply_filter_rules_for_feed.delay") as mock_delay:
fetch_feed(feed.id)
chunk_sizes = [len(call.args[1]) for call in mock_delay.call_args_list]
assert chunk_sizes == [2, 2, 1]
+214 -3
View File
@@ -204,7 +204,7 @@ def test_article_rows_carry_their_utc_instant_for_client_side_date_grouping(clie
dated = Article.objects.create(
feed=feed, guid="dated", title="Dated", published_at=timezone.now()
)
undated = Article.objects.create(feed=feed, guid="undated", title="Undated")
undated = Article.objects.create(feed=feed, guid="undated", title="No publish date")
client.login(username="alice", password="pw-alice-123")
response = client.get(reverse("reader:stream"), {"filter": "all"})
@@ -214,9 +214,15 @@ def test_article_rows_carry_their_utc_instant_for_client_side_date_grouping(clie
assert f'id="article-{dated.pk}"' in content
assert f'data-published-utc="{expected_iso}"' in content
undated_row = re.search(rf'<li id="article-{undated.pk}".*?>', content, flags=re.DOTALL)
# A feed that gave no publish date falls back to when it was fetched
# (the same fallback the stream's sort order uses), so the row still
# carries a time and groups under a real date, not "Undated".
undated_row = re.search(rf'<li id="article-{undated.pk}".*?</li>', content, flags=re.DOTALL)
assert undated_row is not None
assert 'data-published-utc=""' in undated_row.group(0)
row_html = undated_row.group(0)
assert f'data-published-utc="{undated.fetched_at.isoformat()}"' in row_html
assert 'class="article-time"' in row_html
assert "Undated" not in content
@pytest.mark.django_db
@@ -343,3 +349,208 @@ def test_sidebar_unread_count_updates_immediately_after_mark_all_read(client, al
response = client.get(reverse("reader:stream"))
assert "unread-count" not in response.content.decode()
def _stream_article_ids(client, query):
response = client.get(reverse("reader:stream"), query)
content = response.content.decode()
ids = [int(m) for m in re.findall(r'<li id="article-(\d+)"', content)]
return ids, content
def _link_query(content, label):
match = re.search(rf'<a href="\?([^"]+)">{label}', content)
if not match:
return None
from urllib.parse import parse_qsl
return dict(parse_qsl(match.group(1).replace("&amp;", "&")))
@pytest.mark.django_db
def test_stream_keyset_pagination_pages_through_tied_timestamps_without_gaps(client, alice):
"""Keyset pagination (C4) on (sort_at, id): articles sharing an exact
timestamp must neither repeat nor go missing across page boundaries,
which offset paging over a non-unique order can't guarantee."""
from django.utils import timezone
from apps.reader.views import PAGE_SIZE
feed = Feed.objects.create(feed_url="https://example.com/feed.xml")
Subscription.objects.create(user=alice, feed=feed)
same_moment = timezone.now()
total = PAGE_SIZE * 2 + 5
created = [
Article.objects.create(feed=feed, guid=f"g{i}", title=f"T{i}", published_at=same_moment)
for i in range(total)
]
client.login(username="alice", password="pw-alice-123")
seen = []
query = {"filter": "all"}
pages = 0
while query:
ids, content = _stream_article_ids(client, query)
assert len(ids) <= PAGE_SIZE
seen.extend(ids)
query = _link_query(content, "Older")
pages += 1
assert pages < 10
assert pages == 3
assert len(seen) == total
assert sorted(seen) == sorted(a.pk for a in created)
@pytest.mark.django_db
def test_stream_newer_link_returns_to_the_previous_page(client, alice):
from django.utils import timezone
from apps.reader.views import PAGE_SIZE
feed = Feed.objects.create(feed_url="https://example.com/feed.xml")
Subscription.objects.create(user=alice, feed=feed)
now = timezone.now()
for i in range(PAGE_SIZE + 5):
Article.objects.create(
feed=feed,
guid=f"g{i}",
title=f"T{i}",
published_at=now - timezone.timedelta(minutes=i),
)
client.login(username="alice", password="pw-alice-123")
first_ids, first_content = _stream_article_ids(client, {"filter": "all"})
assert _link_query(first_content, "Newer") is None
second_ids, second_content = _stream_article_ids(client, _link_query(first_content, "Older"))
assert len(second_ids) == 5
assert _link_query(second_content, "Older") is None
back_ids, _ = _stream_article_ids(client, _link_query(second_content, "Newer"))
assert back_ids == first_ids
@pytest.mark.django_db
def test_stream_pagination_links_keep_the_feed_scope(client, alice):
from apps.reader.views import PAGE_SIZE
feed = Feed.objects.create(feed_url="https://example.com/feed.xml")
Subscription.objects.create(user=alice, feed=feed)
for i in range(PAGE_SIZE + 1):
Article.objects.create(feed=feed, guid=f"g{i}", title=f"T{i}")
client.login(username="alice", password="pw-alice-123")
_, content = _stream_article_ids(client, {"filter": "all", "feed": feed.pk})
assert _link_query(content, "Older")["feed"] == str(feed.pk)
def test_templates_use_no_inline_event_handlers():
"""Production's CSP (script-src 'self') blocks inline on*= handlers —
they work in dev, where no CSP is sent, and silently do nothing in
production. Behavior belongs in static/js."""
from pathlib import Path
from django.conf import settings
pattern = re.compile(r"""<[^>]+\s(on[a-z]+)\s*=""", re.IGNORECASE)
offenders = []
for template in Path(settings.BASE_DIR, "templates").rglob("*.html"):
for match in pattern.finditer(template.read_text()):
offenders.append(f"{template.name}: {match.group(1)}")
assert offenders == []
@pytest.mark.django_db
def test_article_detail_back_link_keeps_the_stream_scope(client, alice):
feed = Feed.objects.create(feed_url="https://example.com/feed.xml")
Subscription.objects.create(user=alice, feed=feed)
article = Article.objects.create(feed=feed, guid="g1", title="Hello")
client.login(username="alice", password="pw-alice-123")
stream = client.get(
reverse("reader:stream"), {"filter": "all", "feed": feed.pk}
).content.decode()
link = re.search(r'href="(/articles/\d+/\?[^"]+)"', stream)
assert link is not None
detail = client.get(link.group(1).replace("&amp;", "&")).content.decode()
assert f"filter=all&amp;feed={feed.pk}" in detail
assert "Back to stream" in detail
assert article.title in detail
@pytest.mark.django_db
def test_article_with_no_publish_date_shows_when_it_was_fetched(client, alice):
"""A feed that gives no date is treated as published when we fetched
it — in the list row, its group header, and the open article's header
— without storing an invented value on the article itself."""
feed = Feed.objects.create(feed_url="https://example.com/feed.xml")
Subscription.objects.create(user=alice, feed=feed)
article = Article.objects.create(feed=feed, guid="g1", title="No publish date")
assert article.published_at is None
client.login(username="alice", password="pw-alice-123")
iso = article.fetched_at.isoformat()
stream = client.get(reverse("reader:stream"), {"filter": "all"}).content.decode()
assert f'datetime="{iso}"' in stream
assert "Received today" in stream
detail = client.get(reverse("reader:article_detail", args=[article.pk])).content.decode()
assert f'datetime="{iso}"' in detail
article.refresh_from_db()
assert article.published_at is None
@pytest.mark.django_db
def test_new_since_requires_login(client):
response = client.get(reverse("reader:new_since"))
assert response.status_code == 302
@pytest.mark.django_db
def test_new_since_without_a_cursor_returns_the_baseline_id(client, alice):
feed = Feed.objects.create(feed_url="https://example.com/feed.xml")
Subscription.objects.create(user=alice, feed=feed)
article = Article.objects.create(feed=feed, guid="g1", title="Hello")
client.login(username="alice", password="pw-alice-123")
body = client.get(reverse("reader:new_since")).json()
assert body == {"latest_id": article.pk, "new_count": 0}
@pytest.mark.django_db
def test_new_since_counts_only_new_unread_visible_articles_from_my_feeds(client, alice, bob):
mine = Feed.objects.create(feed_url="https://mine.example.com/feed.xml")
theirs = Feed.objects.create(feed_url="https://theirs.example.com/feed.xml")
Subscription.objects.create(user=alice, feed=mine)
Subscription.objects.create(user=bob, feed=theirs)
cursor = Article.objects.create(feed=mine, guid="cursor", title="Cursor").pk
new_unread = Article.objects.create(feed=mine, guid="n1", title="New unread")
already_read = Article.objects.create(feed=mine, guid="n2", title="New but read")
hidden = Article.objects.create(feed=mine, guid="n3", title="New but hidden")
Article.objects.create(feed=theirs, guid="n4", title="Someone else's feed")
UserArticleState.objects.create(user=alice, article=already_read, is_read=True)
UserArticleState.objects.create(user=alice, article=hidden, is_hidden=True)
client.login(username="alice", password="pw-alice-123")
body = client.get(reverse("reader:new_since"), {"since": cursor}).json()
assert body["new_count"] == 1
assert new_unread.pk > cursor
@pytest.mark.django_db
def test_authenticated_pages_load_the_tab_badge_script_with_its_endpoint(client, alice):
client.login(username="alice", password="pw-alice-123")
content = client.get(reverse("reader:stream")).content.decode()
assert "js/tab-badge" in content
assert f'data-url="{reverse("reader:new_since")}"' in content
client.logout()
assert "tab-badge" not in client.get(reverse("accounts:login")).content.decode()
+1
View File
@@ -7,6 +7,7 @@ app_name = "reader"
urlpatterns = [
path("", views.stream, name="stream"),
path("check-new/", views.check_new_articles, name="check_new"),
path("new-since/", views.new_since, name="new_since"),
path("mark-all-read/", views.mark_all_read, name="mark_all_read"),
path("articles/<int:article_id>/", views.article_detail, name="article_detail"),
path(
+92 -15
View File
@@ -1,9 +1,11 @@
from datetime import datetime
from itertools import groupby
from urllib.parse import urlencode
from django.contrib.auth.decorators import login_required
from django.db.models import Max, Q
from django.http import Http404
from django.db.models.functions import Coalesce
from django.http import Http404, JsonResponse
from django.shortcuts import get_object_or_404, redirect, render
from django.urls import reverse
from django.utils import timezone
@@ -111,7 +113,8 @@ def _group_by_date(articles):
today = timezone.localdate()
def day_of(article):
return timezone.localtime(article.published_at).date() if article.published_at else None
when = article.published_at or article.fetched_at
return timezone.localtime(when).date() if when else None
return [
{"label": _date_label(day, today), "articles": list(items)}
@@ -157,26 +160,68 @@ def _filtered_queryset(request, active_filter):
return _apply_scope(request, queryset)
def _encode_cursor(article) -> str:
return f"{article.sort_at.isoformat()},{article.pk}"
def _decode_cursor(raw):
"""(sort_at, id) from a cursor produced by _encode_cursor, or None for
a missing/garbled one (treated as "no cursor" — the first page)."""
if not raw:
return None
try:
timestamp, pk = raw.rsplit(",", 1)
return datetime.fromisoformat(timestamp), int(pk)
except ValueError:
return None
@login_required
def stream(request):
active_filter = _resolve_active_filter(request)
queryset, active_tag, active_feed = _filtered_queryset(request, active_filter)
try:
page = max(int(request.GET.get("page", 1)), 1)
except ValueError:
page = 1
start, end = (page - 1) * PAGE_SIZE, page * PAGE_SIZE
total_count = queryset.count()
articles = with_user_state(queryset[start:end], request.user)
scope_params = {"filter": active_filter}
if active_tag:
scope_params["tag"] = active_tag.pk
if active_feed:
scope_params["feed"] = active_feed.pk
# Keyset pagination (C4) over (sort_at, id) — sort_at falls back to
# fetched_at for entries a feed gave no published date, and id breaks
# ties so pages neither shift nor repeat articles that share a
# timestamp. No COUNT(*): fetching one row past the page size is
# enough to know whether another page exists in that direction.
ordered = queryset.annotate(sort_at=Coalesce("published_at", "fetched_at"))
after = _decode_cursor(request.GET.get("after"))
before = _decode_cursor(request.GET.get("before"))
if after:
sort_at, pk = after
rows = list(
ordered.filter(Q(sort_at__gt=sort_at) | Q(sort_at=sort_at, id__gt=pk)).order_by(
"sort_at", "id"
)[: PAGE_SIZE + 1]
)
has_newer_page = len(rows) > PAGE_SIZE
has_older_page = True
rows = rows[:PAGE_SIZE][::-1]
else:
if before:
sort_at, pk = before
ordered = ordered.filter(Q(sort_at__lt=sort_at) | Q(sort_at=sort_at, id__lt=pk))
rows = list(ordered.order_by("-sort_at", "-id")[: PAGE_SIZE + 1])
has_older_page = len(rows) > PAGE_SIZE
has_newer_page = bool(before)
rows = rows[:PAGE_SIZE]
articles = with_user_state(rows, request.user)
newer_qs = older_qs = ""
if articles:
newer_qs = urlencode({**scope_params, "after": _encode_cursor(articles[0])})
older_qs = urlencode({**scope_params, "before": _encode_cursor(articles[-1])})
context = {
"date_groups": _group_by_date(articles),
"active_filter": active_filter,
@@ -184,15 +229,17 @@ def stream(request):
"active_tag": active_tag,
"active_feed": active_feed,
"user_tags": Tag.objects.filter(user=request.user),
"page": page,
"has_next_page": end < total_count,
"has_previous_page": page > 1,
"has_newer_page": has_newer_page and bool(articles),
"has_older_page": has_older_page and bool(articles),
"newer_qs": newer_qs,
"older_qs": older_qs,
# Lets the "new articles" banner ask, on this exact same scope,
# whether anything with a higher id has shown up since this page
# was rendered — id (not published_at) because it's monotonic
# regardless of how a feed backdates an entry's published date.
"latest_article_id": queryset.aggregate(Max("id"))["id__max"] or 0,
"new_check_qs": urlencode(scope_params),
"scope_qs": urlencode(scope_params),
}
return render(request, "reader/stream.html", context)
@@ -211,6 +258,29 @@ def check_new_articles(request):
return render(request, "reader/_new_articles_banner.html", {"new_count": new_count})
@login_required
def new_since(request):
"""Feeds the browser-tab dot (static/js/tab-badge.js) on every page:
without ?since it returns the current highest article id, which the
page keeps as its baseline; with it, how many unread, non-hidden
articles from the user's subscriptions have appeared after that id.
Deliberately not scoped to the stream's current filter/tag/feed — the
tab should light up for anything new, whichever page is open."""
latest_id = Article.objects.aggregate(Max("id"))["id__max"] or 0
try:
since_id = int(request.GET["since"])
except (KeyError, ValueError):
return JsonResponse({"latest_id": latest_id, "new_count": 0})
new_count = (
_base_queryset(request.user)
.filter(id__gt=since_id)
.exclude(user_states__user=request.user, user_states__is_read=True)
.count()
)
return JsonResponse({"latest_id": latest_id, "new_count": new_count})
@login_required
@require_POST
def mark_all_read(request):
@@ -266,7 +336,14 @@ def _load_article_and_mark_read(user, article_id):
@login_required
def article_detail(request, article_id):
article = _load_article_and_mark_read(request.user, article_id)
return render(request, "reader/article_detail.html", {"article": article})
# The stream passes its filter/tag/feed scope through the link so
# "Back to stream" returns to it instead of resetting to every feed.
back_params = {k: request.GET[k] for k in ("filter", "tag", "feed") if request.GET.get(k)}
return render(
request,
"reader/article_detail.html",
{"article": article, "back_qs": urlencode(back_params)},
)
@login_required
+13 -12
View File
@@ -1,27 +1,28 @@
from django.contrib.postgres.search import SearchQuery, SearchRank, SearchVector
from apps.articles.models import Article
from .base import SearchBackend
# Matches the stored search_vector column, GIN index, and maintaining
# trigger created by apps/search/migrations/0002_postgres_search_vector.py
# — the query reads that column directly instead of recomputing a
# tsvector over every article's full text on every search (C5).
_VECTOR = "articles_article.search_vector"
_QUERY = "plainto_tsquery('english', %s)"
class PostgresSearchBackend(SearchBackend):
"""Computes the search vector on the fly at query time rather than
storing/indexing it on Article — keeps the schema identical to the
SQLite backend. Revisit with a stored+GIN-indexed column only if an
instance's archive grows large enough to need it."""
def search(self, query: str, user, limit: int = 50):
if not query.strip():
return Article.objects.none()
vector = SearchVector("title", weight="A") + SearchVector("content_html", weight="B")
search_query = SearchQuery(query)
return (
Article.objects.filter(feed__subscriptions__user=user)
.select_related("feed")
.annotate(search=vector, rank=SearchRank(vector, search_query))
.filter(search=search_query)
.extra(
select={"rank": f"ts_rank({_VECTOR}, {_QUERY})"},
select_params=[query],
where=[f"{_VECTOR} @@ {_QUERY}"],
params=[query],
)
.order_by("-rank", "-published_at")[:limit]
)
+15 -9
View File
@@ -1,12 +1,14 @@
import logging
import sqlite3
from django.db import connection
from apps.articles.models import Article
from apps.feeds.models import Subscription
from .base import SearchBackend
logger = logging.getLogger(__name__)
def _fts_phrase_query(raw_query: str) -> str:
"""Wraps the whole user input as a single FTS5 phrase so arbitrary
@@ -21,24 +23,28 @@ class SQLiteFTSBackend(SearchBackend):
if not query.strip():
return []
feed_ids = list(Subscription.objects.filter(user=user).values_list("feed_id", flat=True))
if not feed_ids:
return []
placeholders = ",".join(["%s"] * len(feed_ids))
sql = f"""
# Scoped through a subquery on Subscription rather than an IN
# list built from every subscribed feed id (C6): a user with
# thousands of feeds would otherwise exceed SQLite's bound-
# parameter limit and the whole search would fail.
sql = """
SELECT a.id
FROM articles_article a
JOIN articles_fts f ON f.rowid = a.id
WHERE articles_fts MATCH %s AND a.feed_id IN ({placeholders})
WHERE articles_fts MATCH %s
AND a.feed_id IN (SELECT feed_id FROM feeds_subscription WHERE user_id = %s)
ORDER BY rank
LIMIT %s
"""
try:
with connection.cursor() as cursor:
cursor.execute(sql, [_fts_phrase_query(query), *feed_ids, limit])
cursor.execute(sql, [_fts_phrase_query(query), user.pk, limit])
ordered_ids = [row[0] for row in cursor.fetchall()]
except sqlite3.OperationalError:
# A genuinely bad FTS query is rare (the input is quoted as a
# phrase), so this is far more likely a real operational
# failure — log it instead of silently showing "no results".
logger.exception("SQLite FTS search failed for user %s", user.pk)
return []
if not ordered_ids:
@@ -0,0 +1,61 @@
"""Stored, GIN-indexed tsvector for Postgres search (C5), kept current by a
trigger so no ingestion/update code path can forget to maintain it.
No-ops entirely on SQLite — its FTS5 table (0001_sqlite_fts) already plays
this role. The column is deliberately not a Django model field: a
SearchVectorField would break SQLite, and nothing but the search backend
ever reads it (apps/search/backends/postgres.py).
"""
from django.db import migrations
VECTOR_EXPR = (
"setweight(to_tsvector('english', coalesce({row}title, '')), 'A') || "
"setweight(to_tsvector('english', coalesce({row}content_html, '')), 'B')"
)
CREATE = f"""
ALTER TABLE articles_article ADD COLUMN IF NOT EXISTS search_vector tsvector;
UPDATE articles_article SET search_vector = {VECTOR_EXPR.format(row="")};
CREATE INDEX IF NOT EXISTS articles_article_search_vector_gin
ON articles_article USING GIN (search_vector);
CREATE OR REPLACE FUNCTION articles_article_search_vector_update() RETURNS trigger AS $$
BEGIN
NEW.search_vector := {VECTOR_EXPR.format(row="NEW.")};
RETURN NEW;
END
$$ LANGUAGE plpgsql;
DROP TRIGGER IF EXISTS articles_article_search_vector_trg ON articles_article;
CREATE TRIGGER articles_article_search_vector_trg
BEFORE INSERT OR UPDATE OF title, content_html ON articles_article
FOR EACH ROW EXECUTE FUNCTION articles_article_search_vector_update();
"""
DROP = """
DROP TRIGGER IF EXISTS articles_article_search_vector_trg ON articles_article;
DROP FUNCTION IF EXISTS articles_article_search_vector_update();
DROP INDEX IF EXISTS articles_article_search_vector_gin;
ALTER TABLE articles_article DROP COLUMN IF EXISTS search_vector;
"""
def create_vector(apps, schema_editor):
if schema_editor.connection.vendor != "postgresql":
return
schema_editor.execute(CREATE)
def drop_vector(apps, schema_editor):
if schema_editor.connection.vendor != "postgresql":
return
schema_editor.execute(DROP)
class Migration(migrations.Migration):
dependencies = [("search", "0001_sqlite_fts"), ("articles", "0001_initial")]
operations = [migrations.RunPython(create_vector, drop_vector)]
+4
View File
@@ -17,4 +17,8 @@ app.conf.beat_schedule = {
"task": "apps.feeds.tasks.poll_due_feeds",
"schedule": crontab(minute="*/1"),
},
"purge-old-articles": {
"task": "apps.articles.tasks.purge_old_articles",
"schedule": crontab(hour=3, minute=30),
},
}
+15
View File
@@ -77,6 +77,21 @@ ASGI_APPLICATION = "config.asgi.application"
# postgres://user:pass@host:5432/dbname
DATABASES = {"default": env.db_url("DATABASE_URL", default=f"sqlite:///{BASE_DIR / 'db.sqlite3'}")}
# gunicorn, Celery workers, and the FTS triggers all write to the same
# SQLite file (C6): without WAL, readers and a writer block each other,
# and without a busy timeout a contended write fails immediately with
# "database is locked" instead of waiting its turn. IMMEDIATE takes the
# write lock at transaction start so a read-then-write transaction can't
# deadlock upgrading it mid-way.
if DATABASES["default"]["ENGINE"] == "django.db.backends.sqlite3":
DATABASES["default"].setdefault("OPTIONS", {}).update(
{
"timeout": 20,
"transaction_mode": "IMMEDIATE",
"init_command": "PRAGMA journal_mode=WAL; PRAGMA synchronous=NORMAL;",
}
)
AUTH_USER_MODEL = "accounts.User"
AUTH_PASSWORD_VALIDATORS = [
+1 -1
View File
@@ -36,7 +36,7 @@ SECURE_PROXY_SSL_HEADER = ("HTTP_X_FORWARDED_PROTO", "https")
# over loopback (see ALLOWED_HOSTS above) it has no X-Forwarded-Proto
# header to prove it's "secure", and there's no TLS listener on gunicorn's
# own port to redirect to anyway.
SECURE_REDIRECT_EXEMPT = [r"^health/$"]
SECURE_REDIRECT_EXEMPT = [r"^health/$", r"^health/ready/$"]
# WhiteNoise serves /static/ directly from gunicorn instead of relying on
# the reverse proxy to alias the staticfiles directory on disk — the proxy
+36 -2
View File
@@ -313,8 +313,42 @@ surfaces any feeds that are failing to fetch, which is the fastest way to catch
misconfigured feed or an outage at the source.
`/health/` is an unauthenticated liveness check (DB connectivity only) —
`scripts/upgrade.sh` uses it to confirm a deploy actually came back up, and
it's a reasonable target for an external uptime monitor too.
`scripts/upgrade.sh` uses it to confirm a deploy actually came back up.
Point an external uptime monitor at `/health/ready/` instead. It returns 503
with a per-check breakdown when any of these fail:
- `database` and `redis` — reachable.
- `poll_heartbeat` — the background poller ticked in the last five minutes
(proves Celery beat is scheduling *and* a worker is consuming).
- `queue_depth` — no more than 1000 tasks waiting in the Celery queue
(workers aren't keeping up, or aren't running).
- `recent_fetch_success` — at least one feed has been fetched successfully
within three poll intervals (minimum 30 minutes), so "scheduled but every
fetch is failing" is caught. A brand-new install gets that long before it
counts as stale.
- `feed_failures` — no more than half of your active feeds are failing, once
you have at least four (a handful of dead feeds is normal; most of them
failing at once points at your server, DNS, or outbound network).
The payload also reports `queue_depth`, `active_feeds`, `failing_feeds`, and
the age of the last successful fetch, so you can graph them.
**Retention.** By default every fetched article is kept forever. In the Django
admin, "Site settings" has a *Retention days* field: set it to delete articles
fetched longer ago than that, nightly (03:30 UTC, via Celery beat). Articles
anyone has starred or saved for later are always kept; read state and tags for
deleted articles go with them. A small record (feed and entry id) is kept for
each deleted article so a feed that still lists it can't bring it back as new
and unread, and entries whose own publish date is already older than the window
aren't fetched in the first place. The dashboard shows the article count and the
age of the oldest one, so you can see what retention is doing.
**SQLite.** SQLite installs run in WAL mode with a 20-second busy timeout (set
automatically in `config/settings/base.py`) so gunicorn, the Celery worker, and
the full-text-search triggers can share the file without "database is locked"
errors. That's plenty for a personal or small-team instance; for many users or
a very large archive, use PostgreSQL (its search uses a stored, GIN-indexed
vector kept current by a trigger, added by migration automatically).
## Backups
+1
View File
@@ -559,6 +559,7 @@ form p {
font-size: 0.8rem;
color: var(--text-dim);
flex: 0 0 auto;
white-space: nowrap;
}
.article-actions {
+5 -1
View File
@@ -4,7 +4,7 @@
// instant plus a UTC-formatted fallback as their initial text, and this
// reformats them using the browser's own timezone, client-side, so the
// server never needs to know (or guess) where the viewer actually is.
var SHORT_OPTIONS = { hour: "numeric", minute: "2-digit" };
var SHORT_OPTIONS = { month: "short", day: "numeric", hour: "numeric", minute: "2-digit" };
var LONG_OPTIONS = {
year: "numeric",
month: "short",
@@ -21,6 +21,10 @@
var date = new Date(iso);
if (isNaN(date.getTime())) return;
var options = el.dataset.localFormat === "short" ? SHORT_OPTIONS : LONG_OPTIONS;
if (options === SHORT_OPTIONS && date.getFullYear() !== new Date().getFullYear()) {
// Date-only rows from a previous year would otherwise be ambiguous.
options = Object.assign({ year: "numeric" }, SHORT_OPTIONS);
}
try {
el.textContent = date.toLocaleString(undefined, options);
} catch (e) {
+23 -5
View File
@@ -1,8 +1,11 @@
function toggleArticle(event, id) {
event.preventDefault();
var link = event.currentTarget;
// Delegated listeners, not inline onclick="" attributes: production's
// Content Security Policy (config/settings/prod.py, script-src 'self')
// blocks inline event handlers, which silently turned the title click into
// a plain link to the detail page and the new-articles banner into a no-op.
function toggleArticle(link) {
var row = link.closest("li.article-row");
var body = document.getElementById("article-body-" + id);
var body = row.querySelector(".article-inline-body");
var wasOpen = row.classList.contains("is-expanded");
document.querySelectorAll("li.article-row.is-expanded").forEach(function (openRow) {
@@ -25,5 +28,20 @@ function toggleArticle(event, id) {
);
}
}
return false;
}
document.addEventListener("click", function (event) {
var link = event.target.closest("a.article-title[data-inline-url]");
if (link) {
// Leave modified clicks alone so "open in new tab" still works.
if (event.button !== 0 || event.ctrlKey || event.metaKey || event.shiftKey || event.altKey) {
return;
}
event.preventDefault();
toggleArticle(link);
return;
}
if (event.target.closest(".new-articles-banner")) {
location.reload();
}
});
+85
View File
@@ -0,0 +1,85 @@
// Marks the browser tab with a red dot on its favicon when new unread
// articles have arrived since this page loaded — visible from a background
// tab, unlike the in-page "new articles" banner. Runs on every
// authenticated page; a full page load resets the baseline (and the icon),
// which is exactly when the user has seen the new articles.
(function () {
var POLL_MS = 30000;
var url = document.currentScript && document.currentScript.dataset.url;
var icon = document.querySelector('link[rel="icon"]');
if (!url || !icon) return;
var originalHref = icon.href;
var baseline = null;
var badged = null; // the dotted icon's data: URL, drawn once on first need
var showing = false;
function drawDot() {
return new Promise(function (resolve) {
var image = new Image();
image.onload = function () {
var size = 64;
var canvas = document.createElement("canvas");
canvas.width = canvas.height = size;
var ctx = canvas.getContext("2d");
ctx.drawImage(image, 0, 0, size, size);
ctx.beginPath();
ctx.arc(size * 0.74, size * 0.26, size * 0.23, 0, 2 * Math.PI);
ctx.fillStyle = "#e53935";
ctx.fill();
ctx.lineWidth = size * 0.07;
ctx.strokeStyle = "#ffffff";
ctx.stroke();
resolve(canvas.toDataURL("image/png"));
};
image.onerror = function () {
resolve(null);
};
image.src = originalHref;
});
}
// Replace the <link> rather than editing href: some browsers ignore an
// in-place change to an existing favicon element.
function setIcon(href) {
var fresh = icon.cloneNode();
fresh.href = href;
icon.parentNode.replaceChild(fresh, icon);
icon = fresh;
}
function showDot(on) {
if (on === showing) return Promise.resolve();
showing = on;
if (!on) {
setIcon(originalHref);
return Promise.resolve();
}
var ready = badged ? Promise.resolve(badged) : drawDot();
return ready.then(function (href) {
badged = href;
if (href && showing) setIcon(href);
});
}
function poll() {
var query = baseline === null ? "" : "?since=" + baseline;
fetch(url + query, { credentials: "same-origin", headers: { Accept: "application/json" } })
.then(function (response) {
// A redirect means the session ended (we'd be reading the login
// page) — stop polling instead of hammering it.
if (!response.ok || response.redirected) throw new Error("stop");
return response.json();
})
.then(function (data) {
if (baseline === null) baseline = data.latest_id;
else showDot(data.new_count > 0);
setTimeout(poll, POLL_MS);
})
.catch(function (error) {
if (error.message !== "stop") setTimeout(poll, POLL_MS);
});
}
poll();
})();
+1
View File
@@ -58,6 +58,7 @@
{% if user.is_authenticated %}
<script src="{% static 'js/theme.js' %}" defer></script>
<script src="{% static 'js/sidebar-state.js' %}" defer></script>
<script src="{% static 'js/tab-badge.js' %}" data-url="{% url 'reader:new_since' %}" defer></script>
{% endif %}
<script src="{% static 'js/register-sw.js' %}" defer></script>
</body>
+1 -1
View File
@@ -6,7 +6,7 @@
<ul class="dashboard-stats">
<li>{{ stats.feed_count }} feed(s) — {{ stats.active_feed_count }} active</li>
<li>{{ stats.subscription_count }} subscription(s)</li>
<li>{{ stats.article_count }} article(s)</li>
<li>{{ stats.article_count }} article(s){% if stats.oldest_article_at %}, oldest fetched <time datetime="{{ stats.oldest_article_at|date:"c" }}" data-local-format="long">{{ stats.oldest_article_at|date:"M j, Y H:i" }}</time>{% endif %}</li>
<li>{{ stats.error_feed_count }} feed(s) currently erroring</li>
</ul>
+2 -3
View File
@@ -3,9 +3,8 @@
<p class="article-meta">
{% if article.feed.favicon_url %}<img src="{{ article.feed.favicon_url }}" alt="" class="feed-favicon">{% endif %}
{{ article.feed.title }}
{% if article.published_at %}
&middot; <time datetime="{{ article.published_at|date:"c" }}" data-local-format="long">{{ article.published_at|date:"M j, Y H:i" }}</time>
{% endif %}
{# fetched_at stands in when the feed gave no publish date #}
&middot; <time datetime="{{ article.published_at|default:article.fetched_at|date:"c" }}" data-local-format="long">{{ article.published_at|default:article.fetched_at|date:"M j, Y H:i" }}</time>
{% if article.author %} &middot; {{ article.author }}{% endif %}
{% if article.link %} &middot; <a href="{{ article.link }}" target="_blank" rel="noopener noreferrer">Original &#8599;</a>{% endif %}
</p>
+4 -4
View File
@@ -10,10 +10,10 @@
</ul>
<nav class="pagination">
{% if has_previous_page %}
<a href="?filter={{ active_filter }}{% if active_tag %}&tag={{ active_tag.pk }}{% endif %}&page={{ page|add:"-1" }}">&larr; Newer</a>
{% if has_newer_page %}
<a href="?{{ newer_qs }}">&larr; Newer</a>
{% endif %}
{% if has_next_page %}
<a href="?filter={{ active_filter }}{% if active_tag %}&tag={{ active_tag.pk }}{% endif %}&page={{ page|add:"1" }}">Older &rarr;</a>
{% if has_older_page %}
<a href="?{{ older_qs }}">Older &rarr;</a>
{% endif %}
</nav>
+5 -7
View File
@@ -1,17 +1,15 @@
<li id="article-{{ article.pk }}" class="article-row{% if article.is_read %} is-read{% endif %}"
data-published-utc="{% if article.published_at %}{{ article.published_at|date:"c" }}{% endif %}">
data-published-utc="{{ article.published_at|default:article.fetched_at|date:"c" }}">
<div class="article-row-main">
{% include "reader/_article_actions.html" %}
{% if article.feed.favicon_url %}
<img src="{{ article.feed.favicon_url }}" alt="" class="feed-favicon">
{% endif %}
<span class="article-feed-name">{{ article.feed.title|default:article.feed.feed_url }}</span>
<a href="{% url 'reader:article_detail' article.pk %}" class="article-title"
data-inline-url="{% url 'reader:article_inline' article.pk %}"
onclick="return toggleArticle(event, {{ article.pk }})">{{ article.title|default:"(untitled)" }}</a>
{% if article.published_at %}
<time class="article-time" datetime="{{ article.published_at|date:"c" }}" data-local-format="short">{{ article.published_at|date:"g:i A" }}</time>
{% endif %}
<a href="{% url 'reader:article_detail' article.pk %}{% if scope_qs %}?{{ scope_qs }}{% endif %}" class="article-title"
data-inline-url="{% url 'reader:article_inline' article.pk %}">{{ article.title|default:"(untitled)" }}</a>
{# fetched_at stands in when a feed gives no publish date — same fallback the stream's sort order uses #}
<time class="article-time" datetime="{{ article.published_at|default:article.fetched_at|date:"c" }}" data-local-format="short">{{ article.published_at|default:article.fetched_at|date:"M j, g:i A" }}</time>
</div>
<div id="article-body-{{ article.pk }}" class="article-inline-body" hidden></div>
</li>
+1 -1
View File
@@ -1,5 +1,5 @@
{% if new_count %}
<button type="button" class="new-articles-banner" onclick="location.reload()">
<button type="button" class="new-articles-banner">
{{ new_count }} new article{{ new_count|pluralize }} — click to show
</button>
{% endif %}
+1 -1
View File
@@ -1,7 +1,7 @@
{% extends "base.html" %}
{% block title %}{{ article.title|default:"Article" }} — Evection{% endblock %}
{% block content %}
<p><a href="{% url 'reader:stream' %}">&larr; Back to stream</a></p>
<p><a href="{% url 'reader:stream' %}{% if back_qs %}?{{ back_qs }}{% endif %}">&larr; Back to stream</a></p>
<article>
{% include "reader/_article_actions.html" %}
{% include "reader/_article_content.html" %}