Private
Public Access
Batch feed ingestion, add a readiness check, fail open without Redis (C2, C8)
Ingestion did a get_or_create per entry. It now prefetches existing articles for the poll in one query, bulk-inserts new ones and bulk-updates changed ones (falling back to row-by-row only if a concurrent fetch wins a race), and dispatches rule evaluation in bounded chunks so a huge first fetch is several small tasks. /health/ stays a DB-only liveness check (upgrade.sh gates on it right after restart). New /health/ready/ also checks Redis and that poll_due_feeds ticked within five minutes — proving beat and a worker are both alive — and returns 503 with per-check detail otherwise. The sidebar unread-count cache and login throttle now degrade instead of erroring when Redis is unreachable: every authenticated page and every login would otherwise 500 during a cache outage. Found while verifying in a sandbox with no Redis. Also includes the dashboard's oldest-article metric (core/views.py). Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
8d2900424a
commit
e5f1277014
14 files changed
+354
-48
No files matched your search
@@ -208,3 +208,23 @@ def test_fetch_feed_schedules_the_next_fetch_using_the_site_wide_interval():
|
||||
assert feed.next_fetch_at is not None
|
||||
expected = before + timedelta(minutes=120)
|
||||
assert abs((feed.next_fetch_at - expected).total_seconds()) < 5
|
||||
|
||||
|
||||
@pytest.mark.django_db
|
||||
@responses.activate
|
||||
def test_fetch_feed_dispatches_rule_evaluation_in_bounded_chunks(monkeypatch):
|
||||
monkeypatch.setattr("apps.feeds.tasks.RULE_DISPATCH_CHUNK_SIZE", 2)
|
||||
feed = Feed.objects.create(feed_url="https://example.com/feed.xml")
|
||||
items = "".join(f"<item><title>T{i}</title><guid>g{i}</guid></item>" for i in range(5))
|
||||
responses.add(
|
||||
responses.GET,
|
||||
feed.feed_url,
|
||||
body=f'<?xml version="1.0"?><rss version="2.0"><channel>{items}</channel></rss>',
|
||||
status=200,
|
||||
)
|
||||
|
||||
with patch("apps.feeds.tasks.apply_filter_rules_for_feed.delay") as mock_delay:
|
||||
fetch_feed(feed.id)
|
||||
|
||||
chunk_sizes = [len(call.args[1]) for call in mock_delay.call_args_list]
|
||||
assert chunk_sizes == [2, 2, 1]
|
||||
Reference in new issue
Block a user