Quantizing Embeddings: int8, Binary, and Matryoshka

Why Quantize

A 1024-dim float32 embedding is 4 KB. Ten million of them is 40 GB. Quantization reduces this dramatically with modest recall impact:

• int8 quantization: 4x smaller (~1 KB per vector)
• Binary quantization: 32x smaller (~128 bytes per vector)
• Matryoshka: configurable, often 2-4x smaller

For large corpora, quantization is the difference between fitting in RAM and not.

The Three Approaches

flowchart TB
    Quant[Quantization] --> Q1[int8: scale to 8 bits per dim]
    Quant --> Q2[Binary: 1 bit per dim]
    Quant --> Q3[Matryoshka: truncate to fewer dims]

int8 Quantization

Each float32 dimension is mapped to an int8. A scale factor and zero point are stored per vector or per group.

• Recall impact: typically 1-3 percent drop
• Storage: 4x smaller
• Compute: SIMD-friendly; often faster

The 2026 default for cost-conscious deployments.

Binary Quantization

Each dimension reduced to 1 bit (sign of the value). Distance computed via Hamming distance.

• Recall impact: can be substantial (5-15 percent)
• Storage: 32x smaller
• Compute: very fast (XOR popcount)
• Re-rank: typically rerank top candidates with full-precision

Binary works best with rerank: candidate generation in binary; final scoring in full precision.

Matryoshka Embeddings

The embedding model is trained so that truncating to fewer dimensions still produces useful vectors. Truncate to 256 or 512 dims for storage savings; rehydrate to full dims for accurate scoring.

• Recall impact: small if the model is Matryoshka-trained
• Storage: configurable
• Best with models that explicitly support it: OpenAI text-embedding-3, Cohere embed-v4

Decision Matrix

flowchart TD
    Q1{Storage critical?} -->|Yes, extreme| Bin[Binary]
    Q1 -->|Yes, modest| int8[int8]
    Q1 -->|No| Q2{Model supports Matryoshka?}
    Q2 -->|Yes| Mat[Matryoshka truncation]
    Q2 -->|No| Full[Keep full precision]

For most cost-sensitive deployments in 2026, int8 is the sweet spot: substantial savings, small recall impact.

Combining Approaches

You can combine:

• Matryoshka truncation to 512 dims, then int8 quantization → 8x storage saved
• Binary for top-K candidate generation, full precision for top-N rerank → 30x candidate-stage savings, full quality at top

These compositions are how 2026 production systems hit 1B+ vector scale on reasonable hardware.

What Quantization Does Not Help

• Very small corpora — savings are dollars, not real
• Latency-bound, RAM-constrained workloads where recall is paramount
• Recall-critical use cases that cannot tolerate even 1 percent drop

Implementation in 2026

• pgvector: native int8 support added in recent versions
• Qdrant: int8 + binary + Matryoshka support
• Milvus: native quantization support
• Pinecone: int8 / binary modes available
• FAISS: extensive quantization options (PQ, OPQ, etc.)

For most cloud vector DBs, quantization is a configuration toggle, not custom code.

Recall vs Storage Curve

Empirical 2026 numbers (varies by domain):

Setting	Storage	Recall@10 vs full
float32	1x	100%
int8	0.25x	98-99%
Matryoshka 512	0.5x	99%
Matryoshka 256	0.25x	96-98%
Binary	0.03x	85-92%
Binary + rerank	0.03x	96-98%

The "binary + rerank" combination is especially compelling.

Common Gotchas

• Mixing quantized and unquantized vectors in the same query — distances are not comparable
• Re-quantizing on small data — quantization is fitted on the data; small samples produce poor mappings
• Forgetting to renormalize after quantization where it matters
• Comparing quantized vectors with different schemes

Sources

• "Matryoshka Representation Learning" — https://arxiv.org/abs/2205.13147
• Cohere embed quantization — https://docs.cohere.com
• pgvector quantization — https://github.com/pgvector/pgvector
• Qdrant quantization documentation — https://qdrant.tech/documentation
• "Binary embeddings" research — https://arxiv.org

Quantizing Embeddings: int8, Binary, and Matryoshka: production view

Quantizing Embeddings: int8, Binary, and Matryoshka sounds like a single decision, but in production it splits into eval design, prompt cost, and observability. The deeper you push toward live traffic, the more those three pull against each other — better evals catch silent failures, prompt cost limits how often you can re-run them, and weak observability hides which retries are actually saving conversations versus burning latency budget.

Broader technology framing

The protocol layer determines what's possible: WebRTC for browser-side widgets, SIP trunks (Twilio, Telnyx) for PSTN voice, WebSockets for the Realtime API streaming session. Each has its own jitter buffer, its own ICE/STUN dance, and its own failure modes when a customer's corporate firewall is hostile.

Front-end is Next.js 15 + React 19 for the marketing surface and the in-app dashboards, with server components used heavily for the SEO-critical pages. Backend splits across FastAPI for the AI worker, NestJS + Prisma for the customer-facing API, and a thin Go gateway that does auth, rate limiting, and routing — letting each service scale on its own characteristics.

Datastores: Postgres as the source of truth (per-vertical schemas like healthcare_voice, realestate_voice), ChromaDB for RAG over support docs, Redis for ephemeral session state. Postgres RLS enforces tenant isolation at the row level so a misconfigured query can't leak across customers.

FAQ

What's the right way to scope the proof-of-concept? CallSphere runs 37 production agents and 90+ function tools across 115+ database tables in 6 verticals, so most workflows you'd want already have a template. For a topic like "Quantizing Embeddings: int8, Binary, and Matryoshka", that means you're not starting from scratch — you're configuring an agent template that's already been hardened across thousands of conversations.

How do you handle compliance and data isolation? Day one is integration mapping (scheduler, CRM, messaging) and prompt tuning against your top 20 real call transcripts. Day two through five is shadow-mode running, where the agent transcribes and recommends but a human still answers, so you can compare side-by-side. Go-live is the moment your eval pass-rate clears your internal bar.

When does it make sense to switch from a managed model to a self-hosted one? The honest answer: it scales until your tool catalog gets stale. The agent is only as good as the integrations it can actually call, so the operational discipline is keeping schemas, webhooks, and fallback paths green. The platform handles the rest — observability, retries, multi-region routing — without your team owning the GPU layer.

Talk to us

Explore a live demo and compare current plans to find the right fit for your business.