Push due-feed scheduling into an indexed DB query (C1)

poll_due_feeds loaded every active feed's timing fields and recomputed
each one's error-backoff due-ness in Python on every beat tick (every
1-2 minutes) — scheduler CPU and DB reads grew linearly with total feed
count regardless of how many were actually due.

Feed.next_fetch_at (indexed together with is_active) is now set by
fetch_feed itself right after every fetch attempt, success or error,
using the same backoff formula that used to run per-feed in
poll_due_feeds. The scheduler's query is now a direct indexed range
scan (next_fetch_at <= now), ordered so the most overdue feeds claim
first, and bounded to POLL_BATCH_SIZE so a large backlog spreads across
several ticks instead of dispatching everything at once. A migration
backfills next_fetch_at for existing feeds using the same formula.
reset_feed_backoff also clears next_fetch_at so a manually-reset feed
is immediately due again, matching its prior behavior.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
ksmithandClaude Sonnet 5 committed 2026-10-31 16:09:18 -06:00
1 parent b84cfe6474
commit b33fe905d2
6 files changed
+186 -50

No files matched your search

-27
View File
@@ -1,13 +1,6 @@
from datetime import timedelta
from unittest.mock import patch
import pytest
from django.contrib.auth import get_user_model
from django.urls import reverse
from django.utils import timezone
from apps.feeds.models import Feed
from apps.feeds.tasks import poll_due_feeds
from .models import SiteSettings
@@ -30,26 +23,6 @@ def test_load_returns_the_existing_row_without_creating_a_second_one():
assert SiteSettings.objects.count() == 1
@pytest.mark.django_db
def test_poll_due_feeds_uses_the_site_wide_interval_for_every_feed():
SiteSettings.objects.create(fetch_interval_minutes=120)
due = Feed.objects.create(
feed_url="https://a.example.com/feed.xml",
last_fetched_at=timezone.now() - timedelta(minutes=150),
)
not_due = Feed.objects.create(
feed_url="https://b.example.com/feed.xml",
last_fetched_at=timezone.now() - timedelta(minutes=90),
)
with patch("apps.feeds.tasks.fetch_feed.delay") as mock_delay:
poll_due_feeds()
dispatched_ids = {call.args[0] for call in mock_delay.call_args_list}
assert dispatched_ids == {due.id}
assert not_due.id not in dispatched_ids
@pytest.mark.django_db
def test_admin_changelist_redirects_straight_to_the_change_form(client):
assert not SiteSettings.objects.exists()