Rust + Python RAG Chunking Pipeline

Situation

RAG quality lives or dies on chunking. Naive splitters blow past embedding-model token limits, truncate silently, and degrade retrieval; the Python tokenize loop becomes the ingestion bottleneck at scale.

Task

Build ingestion that is token-exact (never exceeds the model limit) and fast enough for large corpora, without leaving Python's embedding/orchestration ecosystem.

Action

• Pushed tokenize and chunk into Rust via PyO3 using tiktoken-rs; a chunk is a window over token IDs, so the limit cannot be exceeded by construction.
• Python orchestrates embeddings (local sentence-transformers or OpenAI) and writes to Qdrant; a CLI covers init-db, ingest, search, and ask.
• A two-layer hallucination guard declines before the model is called: a deterministic similarity cutoff (default 0.50) plus a strict context-only prompt.

How it works

Click to zoom

Python orchestrates; Rust owns the hot path. ingest_documents() makes exactly four calls and the only Rust↔Python crossing is the chunker. On the query side, a deterministic similarity cutoff declines before the model is ever called.

Result

Token-exact RAG chunking, proven in committed benchmarks: every token-aware variant produced 0 over-limit chunks from 1–50 MB, against up to 13,100 violations for a character-count splitter. The Rayon-parallel Rust path is 40% faster than Python tiktoken at 50 MB (24.99 s vs 35.04 s; 2.00 vs 1.43 MB/s), with a measured crossover between 10 MB and 50 MB below which pure Python wins. Five chunker variants benchmarked across four corpus sizes; 25 tests (8 Rust, 17 Python) cover the chunker and the ingest path.

Learning

The honest engineering call was knowing when not to reach for Rust. Below a measured 10–50 MB crossover the per-call FFI and BPE-init cost makes Rust slower than pure Python everywhere, so the benchmark keeps that negative result in the table instead of hiding it. The win is real, but it's narrow, and pretending otherwise would have been the easy lie.