Architecture and operating model
A RAG pipeline does not send whole documents directly to a model. It parses sources, chunks them with authorization and version metadata, creates embeddings, retrieves candidates with lexical and vector search, optionally reranks them and sends only selected context to the generation model. Source ID, section and version accompany the answer.
Chunk size is not an isolated quality setting. Preserve headings, tables and code blocks, and measure overlap against duplication cost and continuity. Apply user authorization in retrieval. Hiding documents after generation does not stop the model from seeing unauthorized content.
Indexing and query paths are separate failure domains. Source changes enter an event queue and update idempotently using content hashes and embedding versions. The query path records top-k candidates, survivors after filters, reranker scores and cited passages without writing unlimited sensitive text to telemetry.
- 1Parse sources and permissions
- 2Chunk and version embeddings
- 3Hybrid retrieval and reranking
- 4Grounded generation and citations
Design parameters
- Chunk budget
- Start with 350-700 tokens aligned to headings; keep tables and code on semantic boundaries.
- Candidate count
- Use a funnel such as top-20 retrieval to top-5 reranking and measure recall@k.
- Authorization filter
- Make tenant, group, document class and validity date searchable metadata before ANN.
- Freshness SLO
- Set a target such as p95 ten minutes from source change to searchable version.
Worked example
If the relevant passage appears in the first 20 results for 9,300 of 10,000 questions, recall@20 is 93%. If reranking retains it in the first five for 8,900 questions, recall@5 is 89%. An 84% answer score should then be decomposed into retrieval, context-use and generation errors.
At 18 chunks per document and 3 KB of text plus metadata per chunk, one million documents produce about 54 GB before embeddings and index overhead. Capacity planning cannot be reduced to vector dimension multiplied by row count.
Example commands: replace lab values and confirm permissions and software versions before use.
rag-index ingest --source sharepoint --chunk-tokens 512 --overlap 64 --embedding-version e3
rag-eval run --dataset support-v4 --metrics recall@20,mrr,groundedness,citation_precisionTroubleshooting
| Observation | Likely cause / distinction | Verification |
|---|---|---|
| Correct document, wrong answer | The model ignores context or receives conflicting chunks. | Evaluate retrieval separately from groundedness and citation correctness. |
| Some users get no results | Authorization metadata is missing or filtering occurs after ANN. | Trace candidate counts before and after tenant/user filters. |
| Fresh content is delayed | The indexing queue, embedding quota or idempotency key is stuck. | Compare source version, queue age, failed job and indexed hash. |
Acceptance checks
- Test parsing and chunk rules for every document type.
- Version embedding model and vector dimension in the index schema.
- Enforce authorization before retrieval.
- Measure recall@k, MRR, groundedness and citation precision.
- Verify deletion and permission-revocation propagation.
- Provide safe refusal and feedback for unsupported answers.
Related concepts
Indexes and execution plans
An index can reduce scanned data for suitable queries, but adds write, maintenance and storage costs. Evaluate column order, selectivity, filters and sorting together. Slow queries are not always caused by missing indexes: estimates, locks, storage latency or excessive data transfer may dominate. Inspect representative execution plans and actual row counts. Measure read improvement alongside write overhead, and avoid uncontrolled production maintenance.
Latency distribution
Latency is the time between starting an operation and receiving its result. An average can hide a small number of very slow operations; medians and p95/p99 percentiles answer different questions. Network RTT, storage waits, processor queues and application processing contribute to end-to-end time. State whether measurements come from the client or server. Check whether increases coincide with traffic growth, maintenance or capacity limits. Record normal and peak-hour baselines before selecting an alert threshold.
Bandwidth and useful throughput
Link capacity differs from useful application throughput. Protocol headers, encryption, retransmissions, small files and storage waits reduce net transfer speed. Keep bits and bytes distinct: 1 Gbit/s corresponds to a theoretical 125 MB/s, not an application performance guarantee. Estimate transfer time as data size divided by measured useful throughput. Observe the network, source reads and destination writes together to locate the bottleneck. Consider temporary slowdowns and competing workloads as well as average speed.
Capacity and usable headroom
Raw capacity is not the capacity available to applications. RAID or erasure coding, filesystems, reserved space, metadata, snapshots and growth headroom are separate deductions. TB and TiB representations also change the displayed number. Write calculations with units, establish protected usable capacity, then subtract operating reserves. Track growth rate as well as current utilization. The projected exhaustion date should leave enough time to procure and deploy additional capacity.
Application consistency
A bootable copy does not prove application-data consistency. Operating-system caches, database logs and write ordering across disks or services affect the result. A crash-consistent copy resembles recovery after an unexpected shutdown; an application-consistent copy follows supported application preparation and write coordination. Validate transaction integrity, relationships between records and application behavior after recovery, rather than only counting files. Confirm backup integration, application version and reported errors before assuming that consistency was achieved.
File permissions and service identities
File access combines users/groups, permission bits, ACLs and security policies. Running an application as root can hide permission problems; prefer a justified, restricted service identity in production. Directory traversal permission differs from file-read permission. Sharing permissions do not necessarily override filesystem restrictions. Identify the account actually running the application, directory chain and ACLs when troubleshooting. Test narrowly required access rather than opening permissions globally.
Primary documentation
Prepared by the Doz Teknoloji technical team using the primary references below. Calculations and lab scenarios state their assumptions; validate the applicable product version before rollout.