Skip to main content

SecuraWall.com

Email security and continuity

Explore

Deployment · Artificial Intelligence / AI

Embedding Model Migration with Dual Indexes and Rollback

Vectors from different embedding models should not be assumed to share a semantic space, even when dimensions match. Mixing old document vectors with new query vectors can make distances…

Technical review:

Architecture and operating model

Vectors from different embedding models should not be assumed to share a semantic space, even when dimensions match. Mixing old document vectors with new query vectors can make distances meaningless. A safe migration keeps `embedding_v1` and `embedding_v2` plus their indexes side by side, dual-writes new content, backfills history, compares a fixed evaluation set and shifts reads gradually. Keep the old path through the rollback window.

Record the model contract first: version, dimension, normalization, distance metric, preprocessing, chunking and language scope. A dimension change also changes schema and index operator choices. Store stable source IDs, content hashes and embedding versions so backfill is repeatable and stale records are detectable. Test chunking changes separately from model changes when possible.

During dual-write, new and updated records receive both representations while production reads remain on v1. Backfill in checkpointed batches according to quota and capacity. Reconcile failed, duplicate and wrong-version vectors even when coverage says 100 percent. After building the v2 index, compare recall@k, nDCG or task success plus P95 latency, size and cost. Segment results by language and document type.

Cut over with a feature flag or retrieval profile at controlled percentages. Shadow comparison of v1 and v2 results can expose regression without changing user output. If error, latency or quality thresholds fail, route reads back to v1 while dual-write keeps it fresh. Before removing v1, approve retention, re-generation cost, audit needs and a final rollback test.

  1. 1Versioned source records and dual-write
  2. 2v2 backfill and separate vector index
  3. 3Offline and shadow quality gates
  4. 4Gradual read cutover and v1 rollback
Do not mix new vectors into the old space; run both paths with measurable, reversible cutover.

Design parameters

Model contract
Version model, dimension, metric, normalization, chunking and preprocessing together.
Capacity
Budget two vector fields, two ANN indexes and temporary backfill space.
Backfill rate
Batch against API quota, GPU or CPU, source I/O and indexing delay.
Quality gate
Report language, document type, query class and failures beyond the global average.
Rollback window
Define how long v1 reads and writes remain and who approves removal.

Worked example

Ten million 1,536-dimensional float32 vectors use about 61.4 GB before index and metadata. A second representation and ANN index require substantially more. Measure a 100,000-record pilot and include nonlinear index growth.

At 200 records per second, ten million records need an ideal 50,000 seconds, about 13.9 hours. Rate limits, retries, source reads and index maintenance extend that time. Checkpoints and idempotent upserts avoid restarting.

Example commands: replace lab values and confirm permissions and software versions before use.

ALTER TABLE chunks ADD COLUMN embedding_v2 vector(1536);
ALTER TABLE chunks ADD COLUMN embedding_v2_model text;
CREATE INDEX CONCURRENTLY chunks_embedding_v2_hnsw
ON chunks USING hnsw (embedding_v2 vector_cosine_ops);
SELECT count(*) FILTER (WHERE embedding_v2 IS NULL) AS missing,
       count(*) AS total FROM chunks;

Troubleshooting

ObservationLikely cause / distinctionVerification
Coverage is 100 percent but results are missingRows may contain wrong model versions, stale hashes or invalid dimensions.Reconcile model_id, dimension, content_hash and update time against the source.
v2 quality is good but latency is poorDimension or ANN parameters may increase scans and memory.Compare P50/P95, cache hit, index size and query plans with v1.
v1 is stale after rollbackDual-write may have stopped or v1 errors went unmonitored.Write both fields through the rollback window and alert on v1 freshness.

Acceptance checks

  1. Version the model and preprocessing contract.
  2. Measure v2 field and index capacity in a pilot.
  3. Enable idempotent dual-write for new content.
  4. Complete checkpointed backfill and reconciliation.
  5. Pass quality, latency, cost and subgroup gates.
  6. Cut over gradually and prove v1 rollback.

Related concepts

Indexes and execution plans

An index can reduce scanned data for suitable queries, but adds write, maintenance and storage costs. Evaluate column order, selectivity, filters and sorting together. Slow queries are not always caused by missing indexes: estimates, locks, storage latency or excessive data transfer may dominate. Inspect representative execution plans and actual row counts. Measure read improvement alongside write overhead, and avoid uncontrolled production maintenance.

Application consistency

A bootable copy does not prove application-data consistency. Operating-system caches, database logs and write ordering across disks or services affect the result. A crash-consistent copy resembles recovery after an unexpected shutdown; an application-consistent copy follows supported application preparation and write coordination. Validate transaction integrity, relationships between records and application behavior after recovery, rather than only counting files. Confirm backup integration, application version and reported errors before assuming that consistency was achieved.

Version and support lifecycle

Installability does not prove production support. Review the compatibility chain across OS, application, drivers, extensions and management tools. Update plans should record version, support end, restart needs and rollback methods. A test environment that does not reflect production can produce misleading results. Validate service health and existing workflows after a change, not just version numbers. Remember that pinning a version can also prevent future security fixes.

Capacity and usable headroom

Raw capacity is not the capacity available to applications. RAID or erasure coding, filesystems, reserved space, metadata, snapshots and growth headroom are separate deductions. TB and TiB representations also change the displayed number. Write calculations with units, establish protected usable capacity, then subtract operating reserves. Track growth rate as well as current utilization. The projected exhaustion date should leave enough time to procure and deploy additional capacity.

Latency distribution

Latency is the time between starting an operation and receiving its result. An average can hide a small number of very slow operations; medians and p95/p99 percentiles answer different questions. Network RTT, storage waits, processor queues and application processing contribute to end-to-end time. State whether measurements come from the client or server. Check whether increases coincide with traffic growth, maintenance or capacity limits. Record normal and peak-hour baselines before selecting an alert threshold.

Telemetry and time correlation

Telemetry combines logs, metrics and events that explain system behavior. A log describes an event, a metric shows behavior over time, and a distributed trace follows a request across components. Clock differences can make one event appear to occur at several times. Use synchronized clocks, reliable source identifiers and consistent time-zone handling. Alarm design should consider duration and user impact alongside thresholds. Monitor gaps in collection separately: absence of logs must not be interpreted as absence of incidents.

Primary documentation

Prepared by the Doz Teknoloji technical team using the primary references below. Calculations and lab scenarios state their assumptions; validate the applicable product version before rollout.

Knowledge Center

Enterprise IT Product Sales, Licensing and Deployment
Enterprise IT Project & Solution Scenarios
View all related content
Text on WhatsApp
Copied!