Batch feed ingestion, add a readiness check, fail open without Redis (C2, C8)

Ingestion did a get_or_create per entry. It now prefetches existing
articles for the poll in one query, bulk-inserts new ones and
bulk-updates changed ones (falling back to row-by-row only if a
concurrent fetch wins a race), and dispatches rule evaluation in
bounded chunks so a huge first fetch is several small tasks.

/health/ stays a DB-only liveness check (upgrade.sh gates on it right
after restart). New /health/ready/ also checks Redis and that
poll_due_feeds ticked within five minutes — proving beat and a worker
are both alive — and returns 503 with per-check detail otherwise.

The sidebar unread-count cache and login throttle now degrade instead
of erroring when Redis is unreachable: every authenticated page and
every login would otherwise 500 during a cache outage. Found while
verifying in a sandbox with no Redis.

Also includes the dashboard's oldest-article metric (core/views.py).

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
ksmithandClaude Sonnet 5.5 committed 2026-11-01 08:51:20 -07:00
1 parent 8d2900424a
commit e5f1277014
14 files changed
+354 -48

No files matched your search

+15 -1
View File
@@ -4,6 +4,7 @@ from io import BytesIO
from celery import shared_task
from django.contrib.auth import get_user_model
from django.core.cache import cache
from django.db.models import Q
from django.utils import timezone
@@ -27,6 +28,12 @@ CLAIM_STALE_MINUTES = 10
# across several beat ticks instead of enqueuing everything — and every
# bounded Feed query below stays index-only work, not a full scan (C1).
POLL_BATCH_SIZE = 500
RULE_DISPATCH_CHUNK_SIZE = 50
# Written on every poll_due_feeds tick; apps.core.views.health_ready
# reads it to tell whether beat is scheduling and a worker is consuming.
POLL_HEARTBEAT_CACHE_KEY = "poll-due-feeds-heartbeat"
POLL_HEARTBEAT_TTL_SECONDS = 3600
@shared_task
@@ -39,6 +46,7 @@ def poll_due_feeds():
here, not via one Celery Beat entry per feed.
"""
now = timezone.now()
cache.set(POLL_HEARTBEAT_CACHE_KEY, now, timeout=POLL_HEARTBEAT_TTL_SECONDS)
claim_stale_before = now - timedelta(minutes=CLAIM_STALE_MINUTES)
due_feed_ids = list(
Feed.objects.filter(is_active=True)
@@ -125,7 +133,13 @@ def fetch_feed(feed_id: int):
len(result.new_article_ids),
len(result.updated_article_ids),
)
apply_filter_rules_for_feed.delay(feed_id, changed_article_ids)
# Chunked so one huge poll (a first fetch of a long archive feed)
# becomes several bounded rule-evaluation tasks, each well inside
# the worker's time limits, instead of one unbounded one (C2).
for start in range(0, len(changed_article_ids), RULE_DISPATCH_CHUNK_SIZE):
apply_filter_rules_for_feed.delay(
feed_id, changed_article_ids[start : start + RULE_DISPATCH_CHUNK_SIZE]
)
return {
"status": result.status,