Private
Public Access
Make the readiness check catch fetching that is scheduled but failing (#27)
/health/ready/ proved beat and a worker were alive but not that fetching worked: every feed could be failing, or the queue could be hours behind, and it would still report ok. apps.core.health adds three checks: Celery queue depth (real Redis list length, over 1000 fails), a successful fetch within three poll intervals (30 min minimum, with a grace period for a brand-new install), and the share of active feeds failing (over half, once there are at least four). The payload also reports queue depth, active/failing feed counts, and the age of the last successful fetch. Verified against real Redis with queued Celery tasks, and the suite on PostgreSQL. docs/install.md lists each check. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
f8cd1eb33e
commit
fd494a9c08
4 files changed
+252
-11
No files matched your search
+10
-1
@@ -12,6 +12,8 @@ from apps.articles.models import Article
|
||||
from apps.feeds.models import Feed, Subscription
|
||||
from apps.feeds.tasks import POLL_HEARTBEAT_CACHE_KEY, fetch_feed, reset_feed_backoff
|
||||
|
||||
from .health import fetch_pipeline_status
|
||||
|
||||
is_admin = user_passes_test(lambda u: u.is_authenticated and u.is_staff)
|
||||
|
||||
|
||||
@@ -37,7 +39,9 @@ def health_ready(request):
|
||||
"""Readiness: liveness plus everything background fetching depends on
|
||||
(C8) — the database, the Redis cache/broker, and a recent
|
||||
poll_due_feeds tick, which proves beat is scheduling AND a worker is
|
||||
consuming. /health/ above stays DB-only on purpose: scripts/upgrade.sh
|
||||
consuming. Beyond that tick, apps.core.health checks fetching is
|
||||
actually succeeding: queue depth, a recent successful fetch, and the
|
||||
share of feeds failing. /health/ above stays DB-only on purpose: scripts/upgrade.sh
|
||||
gates a deploy on it right after restart, before beat has had a
|
||||
chance to tick. Point an uptime monitor at this one instead."""
|
||||
checks = {"database": False, "redis": False, "poll_heartbeat": False}
|
||||
@@ -65,6 +69,11 @@ def health_ready(request):
|
||||
detail["poll_heartbeat_age_seconds"] = int(age)
|
||||
checks["poll_heartbeat"] = age <= POLL_HEARTBEAT_MAX_AGE_SECONDS
|
||||
|
||||
if checks["database"]:
|
||||
pipeline_checks, pipeline_detail = fetch_pipeline_status()
|
||||
checks.update(pipeline_checks)
|
||||
detail.update(pipeline_detail)
|
||||
|
||||
ok = all(checks.values())
|
||||
return JsonResponse(
|
||||
{"status": "ok" if ok else "degraded", "checks": checks, **detail},
|
||||
|
||||
Reference in new issue
Block a user