RAG With Structured Data: Tables, JSON, and Knowledge Graphs Together

Why Pure-Vector RAG Misses

Vector RAG embeds chunks of text and retrieves by similarity. Structured data — tables, JSON records, knowledge-graph triples — does not embed well. A row "John Smith | Engineering | Senior | Boston" loses information when smashed into text. Specific filters, aggregations, and joins are also impossible with vector retrieval alone.

By 2026 the answer is hybrid: vector retrieval for unstructured text, SQL or graph queries for structured data, fused at query time.

The Architecture

flowchart LR
    Q[User query] --> Class[Classify: text / structured / both]
    Class -->|text| Vec[Vector RAG]
    Class -->|structured| SQL[SQL / graph query]
    Class -->|both| Both[Both]
    Vec --> Combine[Combine results]
    SQL --> Combine
    Combine --> Gen[Generate answer]

A router decides which path. For mixed queries, both paths run and results are fused.

Text-to-SQL

For structured data, an LLM converts the user's question to SQL:

Q: "How many customers in California signed up in March?"
SQL: SELECT COUNT(*) FROM customers WHERE state = 'CA' AND signup_date >= '2026-03-01' AND signup_date < '2026-04-01';

The 2026 production patterns:

• Schema-aware prompting (the LLM knows the table structure)
• Constrained SQL generation (tools like outlines, SQLAlchemy schema)
• Validation before execution
• Read-only roles for the LLM-generated query
• Result re-ranking and capping

Text-to-SQL works well in 2026 for moderately complex queries; very complex queries still benefit from a human writing the SQL.

Knowledge-Graph Queries

For relationship-heavy domains:

Q: "Who is the manager of the engineer on team X?"
Cypher: MATCH (t:Team {name: 'X'})<-[:MEMBER]-(e:Engineer)-[:REPORTS_TO]->(m:Manager) RETURN m.name;

LLMs can generate Cypher / SPARQL similarly to SQL. Same patterns apply: schema-aware prompts, validation, read-only access.

JSON / Document Stores

For semi-structured data (records with varying fields):

• LLM generates a JSON path expression
• Document DB executes the query
• Results returned as structured records

MongoDB's MQL, PostgreSQL JSONB queries, Elasticsearch DSL — all are LLM-translatable in 2026.

Hybrid Query Pattern

For queries that need both:

Q: "Summarize the recent customer complaints about our pricing changes."

Steps:

1. Structured query: find recent complaints (filter by date and topic)
2. Vector query: find unstructured text content for those complaints
3. LLM summarizes both

The router invokes both subsystems and the generation step composes.

Schema and Vocabulary

For text-to-SQL or text-to-Cypher to work:

• Provide the LLM with the schema
• Include sample values for ambiguous columns ("status" can be "active" / "pending" / "closed")
• Document foreign keys and relationships
• For knowledge graphs, document edge types

Without this, the LLM hallucinates table or property names.

Validation

LLM-generated SQL / queries must be validated before execution:

• Parse for syntactic correctness
• Reject queries that touch forbidden tables
• Cap result size
• Time out long queries

Treat the LLM as untrusted input even when it is your own integration.

Common Failures

flowchart TD
    Fail[Failures] --> F1[Hallucinated table / column]
    Fail --> F2[Invalid join]
    Fail --> F3[Missing filter that produces too-large result]
    Fail --> F4[Wrong aggregation]
    Fail --> F5[Schema confusion across tenants]

Each preventable with proper schema introspection and validation.

When Pure-Vector Is Enough

For corpora that are mostly unstructured (long documents, articles, manuals), pure-vector RAG often suffices. Structured-RAG patterns are for domains where the answer requires aggregation, joins, or relationships.

A Production Example

For an internal Q&A bot at a SaaS company:

• Customer KB articles in vector store
• Customer master in Postgres
• Account relationships in Neo4j

A question like "Which Acme contacts have asked about pricing recently?" routes to all three: graph for Acme's contacts, Postgres for filter, vector for "asked about pricing" content. Combined and synthesized.

Sources

• "Text-to-SQL benchmarks" Spider/BIRD — https://yale-lily.github.io/spider
• LangChain SQL agents — https://python.langchain.com/docs/use_cases/sql
• "Knowledge graph RAG" research — https://arxiv.org
• LlamaIndex structured data guides — https://docs.llamaindex.ai
• "Hybrid retrieval methods" 2025 review — https://arxiv.org

RAG With Structured Data: Tables, JSON, and Knowledge Graphs Together: production view

RAG With Structured Data: Tables, JSON, and Knowledge Graphs Together sounds like a single decision, but in production it splits into eval design, prompt cost, and observability. The deeper you push toward live traffic, the more those three pull against each other — better evals catch silent failures, prompt cost limits how often you can re-run them, and weak observability hides which retries are actually saving conversations versus burning latency budget.

Broader technology framing

The protocol layer determines what's possible: WebRTC for browser-side widgets, SIP trunks (Twilio, Telnyx) for PSTN voice, WebSockets for the Realtime API streaming session. Each has its own jitter buffer, its own ICE/STUN dance, and its own failure modes when a customer's corporate firewall is hostile.

Front-end is Next.js 15 + React 19 for the marketing surface and the in-app dashboards, with server components used heavily for the SEO-critical pages. Backend splits across FastAPI for the AI worker, NestJS + Prisma for the customer-facing API, and a thin Go gateway that does auth, rate limiting, and routing — letting each service scale on its own characteristics.

Datastores: Postgres as the source of truth (per-vertical schemas like healthcare_voice, realestate_voice), ChromaDB for RAG over support docs, Redis for ephemeral session state. Postgres RLS enforces tenant isolation at the row level so a misconfigured query can't leak across customers.

FAQ

What's the right way to scope the proof-of-concept? CallSphere runs 37 production agents and 90+ function tools across 115+ database tables in 6 verticals, so most workflows you'd want already have a template. For a topic like "RAG With Structured Data: Tables, JSON, and Knowledge Graphs Together", that means you're not starting from scratch — you're configuring an agent template that's already been hardened across thousands of conversations.

How do you handle compliance and data isolation? Day one is integration mapping (scheduler, CRM, messaging) and prompt tuning against your top 20 real call transcripts. Day two through five is shadow-mode running, where the agent transcribes and recommends but a human still answers, so you can compare side-by-side. Go-live is the moment your eval pass-rate clears your internal bar.

When does it make sense to switch from a managed model to a self-hosted one? The honest answer: it scales until your tool catalog gets stale. The agent is only as good as the integrations it can actually call, so the operational discipline is keeping schemas, webhooks, and fallback paths green. The platform handles the rest — observability, retries, multi-region routing — without your team owning the GPU layer.

Talk to us

Explore a live demo and compare current plans to find the right fit for your business.