Private
Public Access
Isolate unexpected fetch failures and harden task durability (T2)
fetch_feed had no time limit and no handling for anything beyond the network/parse errors fetch_and_ingest already catches internally — an unexpected exception (a parser/sanitizer bug, a DB error) crashed the task with no error recorded on the Feed, the claim left held for the full CLAIM_STALE_MINUTES, and no visibility beyond a bare traceback. fetch_feed now catches that, records it the same way a handled failure would be (reusing fetcher.record_fetch_error, promoted from a private helper), clears the claim, and re-raises so Celery still marks the task FAILED. soft_time_limit/time_limit bound a single fetch that somehow outruns FETCH_TIMEOUT_SECONDS + ssrf_guard's response cap during parsing/ingestion. Also: a worker process killed mid-task (OOM, deploy restart, crash) previously lost that task silently — Celery's default is to ack a message the moment a worker *starts* it. CELERY_TASK_ACKS_LATE + CELERY_TASK_REJECT_ON_WORKER_LOST (config/settings/base.py) make the message redeliverable instead, and CELERY_WORKER_PREFETCH_MULTIPLIER=1 keeps a worker from grabbing a batch of tasks it could lose all at once. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
1 parent
0466ccc334
commit
64f25789f5
4 files changed
+104
-10
No files matched your search
@@ -106,9 +106,34 @@ EVECTION_SIGNUP_ENABLED = env.bool("EVECTION_SIGNUP_ENABLED", default=False)
|
||||
EVECTION_MIN_FETCH_INTERVAL_MINUTES = env.int("EVECTION_MIN_FETCH_INTERVAL_MINUTES", default=15)
|
||||
EVECTION_PWA_CACHE_VERSION = env.str("EVECTION_PWA_CACHE_VERSION", default="v1")
|
||||
|
||||
# Redis-backed rather than Django's default per-process LocMemCache: login
|
||||
# rate limiting (apps.accounts.views) must share counters across every
|
||||
# gunicorn worker, not just whichever one happened to handle a given
|
||||
# request. Redis is already a hard install requirement for Celery, so
|
||||
# this adds no new infrastructure — just a distinct key prefix so cache
|
||||
# keys can't collide with anything else sharing that Redis instance.
|
||||
CACHES = {
|
||||
"default": {
|
||||
"BACKEND": "django.core.cache.backends.redis.RedisCache",
|
||||
"LOCATION": env.str("REDIS_URL", default="redis://localhost:6379/0"),
|
||||
"KEY_PREFIX": "evection-cache",
|
||||
}
|
||||
}
|
||||
|
||||
# Celery
|
||||
CELERY_BROKER_URL = env.str("REDIS_URL", default="redis://localhost:6379/0")
|
||||
CELERY_RESULT_BACKEND = None
|
||||
CELERY_TASK_SERIALIZER = "json"
|
||||
CELERY_ACCEPT_CONTENT = ["json"]
|
||||
CELERY_TIMEZONE = TIME_ZONE
|
||||
|
||||
# Without these, a worker process killed mid-task (OOM, deploy restart,
|
||||
# crash) silently loses that task: the default is to ack a message the
|
||||
# moment a worker *starts* it, not when it finishes. acks_late + reject-
|
||||
# on-worker-lost means the message is only consumed on completion (success
|
||||
# or a handled failure) and is redelivered to another worker otherwise.
|
||||
# prefetch_multiplier=1 keeps a worker from grabbing a batch of tasks it
|
||||
# could similarly lose all at once.
|
||||
CELERY_TASK_ACKS_LATE = True
|
||||
CELERY_TASK_REJECT_ON_WORKER_LOST = True
|
||||
CELERY_WORKER_PREFETCH_MULTIPLIER = 1
|
||||
Reference in new issue
Block a user