Files
Evection/apps/search/migrations/0002_postgres_search_vector.py
T
ksmithandClaude Sonnet 5.5 d6e310b127 Make the Postgres search migration online-safe (R2-T3, #39)
One migration did a full-table UPDATE, a blocking GIN build and the
trigger install together — long locks and a deploy that could outlast
its health-check window on a large archive.

Now two stages. 0002 only adds the nullable column and the trigger
(metadata-only; new and updated rows get a vector from then on). 0003
is non-atomic: it backfills in 5,000-id batches, each its own short
transaction and only touching rows still NULL, then builds the index
with CREATE INDEX CONCURRENTLY (dropping a stale INVALID one from a
failed earlier run first). Re-running is safe, and it's a cheap no-op on
an install that applied the earlier single-step version.

Measured on a real PostgreSQL 16 with 150,000 existing articles while a
second connection kept inserting/updating: the migration took ~20s, the
slowest concurrent write was 37ms (nothing blocked), no NULL vectors
remained, the index was valid, and unapply/re-apply worked. Full suite
passes on Postgres.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-11-01 12:02:20 -07:00

64 lines
2.3 KiB
Python

"""Stored tsvector for Postgres search (C5), kept current by a trigger so
no ingestion/update code path can forget to maintain it.
Stage 1 of 2, and deliberately cheap: it only adds the (nullable) column
and installs the trigger, so new and updated rows get a vector from this
point on. Both are metadata-only/brief-lock operations even on a huge
table. The slow work — backfilling existing rows and building the GIN
index — is 0003, done in batches and CONCURRENTLY so a large existing
archive doesn't lock the table for the length of a deploy (R2-T3).
No-ops entirely on SQLite — its FTS5 table (0001_sqlite_fts) already plays
this role. The column is deliberately not a Django model field: a
SearchVectorField would break SQLite, and nothing but the search backend
ever reads it (apps/search/backends/postgres.py).
"""
from django.db import migrations
VECTOR_EXPR = (
"setweight(to_tsvector('english', coalesce({row}title, '')), 'A') || "
"setweight(to_tsvector('english', coalesce({row}content_html, '')), 'B')"
)
CREATE = f"""
ALTER TABLE articles_article ADD COLUMN IF NOT EXISTS search_vector tsvector;
CREATE OR REPLACE FUNCTION articles_article_search_vector_update() RETURNS trigger AS $$
BEGIN
NEW.search_vector := {VECTOR_EXPR.format(row="NEW.")};
RETURN NEW;
END
$$ LANGUAGE plpgsql;
DROP TRIGGER IF EXISTS articles_article_search_vector_trg ON articles_article;
CREATE TRIGGER articles_article_search_vector_trg
BEFORE INSERT OR UPDATE OF title, content_html ON articles_article
FOR EACH ROW EXECUTE FUNCTION articles_article_search_vector_update();
"""
DROP = """
DROP TRIGGER IF EXISTS articles_article_search_vector_trg ON articles_article;
DROP FUNCTION IF EXISTS articles_article_search_vector_update();
DROP INDEX IF EXISTS articles_article_search_vector_gin;
ALTER TABLE articles_article DROP COLUMN IF EXISTS search_vector;
"""
def create_vector(apps, schema_editor):
if schema_editor.connection.vendor != "postgresql":
return
schema_editor.execute(CREATE)
def drop_vector(apps, schema_editor):
if schema_editor.connection.vendor != "postgresql":
return
schema_editor.execute(DROP)
class Migration(migrations.Migration):
dependencies = [("search", "0001_sqlite_fts"), ("articles", "0001_initial")]
operations = [migrations.RunPython(create_vector, drop_vector)]