Private
Public Access
Add an atomic per-feed claim to poll_due_feeds (T1)
poll_due_feeds scanned for due feeds and dispatched fetch_feed without marking them claimed, so overlapping beat ticks or a slow-running fetch could dispatch the same feed twice, racing on ETag/status/error- count writes. Feed.claimed_at is now set via an UPDATE ... WHERE claimed_at IS NULL OR claimed_at < stale-cutoff, so only one caller's UPDATE can ever match a given row; fetch_feed clears it when done. A claim older than CLAIM_STALE_MINUTES is treated as abandoned so a worker dying mid-fetch doesn't wedge the feed forever. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
1 parent
d7b433e580
commit
b7d9ee672c
4 files changed
+108
-4
No files matched your search
@@ -40,6 +40,15 @@ class Feed(models.Model):
|
||||
fetch_error_count = models.PositiveIntegerField(default=0)
|
||||
is_active = models.BooleanField(default=True)
|
||||
|
||||
# Set by poll_due_feeds' atomic claim (an UPDATE ... WHERE clause, not
|
||||
# select_for_update — Celery Beat and fetch_feed run in separate
|
||||
# transactions) and cleared once fetch_feed finishes, so overlapping
|
||||
# beat ticks or a fetch that runs long can't both dispatch the same
|
||||
# feed. A claim older than CLAIM_STALE_MINUTES (tasks.py) is treated as
|
||||
# abandoned and can be reclaimed, so a worker dying mid-fetch doesn't
|
||||
# wedge the feed forever.
|
||||
claimed_at = models.DateTimeField(null=True, blank=True)
|
||||
|
||||
created_at = models.DateTimeField(auto_now_add=True)
|
||||
|
||||
def __str__(self):
|
||||
|
||||
Reference in new issue
Block a user