Schema Registry for AI Events: Confluent vs Karapace, Avro vs Protobuf vs JSON Schema

TL;DR — Schema registry is the boring infrastructure that prevents 90% of event-driven outages. Confluent Schema Registry is the canonical implementation; Karapace is the API-compatible Apache 2.0 alternative. Pair it with Avro, Protobuf, or JSON Schema and your AI event producers and consumers evolve independently without a war room.

The pattern

Producer team adds a field. Consumer team didn't get the memo. Production breaks. The fix: a schema registry between producers and consumers that checks every new schema against compatibility rules (BACKWARD, FORWARD, FULL). Producers can't ship an incompatible schema; consumers know the schema by reference instead of guessing from bytes.

How it works (architecture)

flowchart LR
  Prod[Producer<br/>writes Avro/Proto/JSON-SR] -->|register| SR[(Schema Registry<br/>Confluent or Karapace)]
  SR -->|schema id 42| Prod
  Prod -->|magic byte + id 42 + payload| K[(Kafka)]
  K --> Cons[Consumer]
  Cons -->|GET schema 42| SR
  SR --> Cons
  SR -.compat check.-> Block[Reject incompatible schema]

Each event carries a 4-byte schema ID prefix. Consumers fetch and cache the schema by ID. Compatibility rules ensure new schemas don't break old consumers (BACKWARD) or new consumers don't break on old data (FORWARD).

CallSphere implementation

CallSphere uses Karapace (open-source, Apache 2.0) for our internal Kafka topics across Real Estate OneRoof, Healthcare, IT Services, Salon, After-hours, and Sales. We picked Avro because compactness matters at our event volume. The CloudEvents envelope (post #12) wraps Avro-encoded data referenced via dataschema. After-hours and the simpler Bull/Redis paths use JSON Schema for readability. CI rejects PRs that change a schema without a compat declaration. 37 agents · 90+ tools · 115+ DB tables · 6 verticals · pricing $149/$499/$1499 · 14-day trial · 22% affiliate. Browse /pricing or take a demo.

Build steps with code

1. Pick registry: Confluent (managed, paid for SLA) or Karapace (open source).
2. Pick format: Avro (compact, schema evolution shines), Protobuf (gRPC alignment), JSON Schema (human-readable).
3. Register schemas in CI, not at runtime.
4. Set compatibility level per subject: BACKWARD by default.
5. Producers serialize with schema-aware serdes.
6. Consumers deserialize lazily via schema ID lookup + cache.
7. Lifecycle: deprecate old fields with default values; never remove.

from confluent_kafka import Producer
from confluent_kafka.serialization import SerializationContext, MessageField
from confluent_kafka.schema_registry import SchemaRegistryClient
from confluent_kafka.schema_registry.avro import AvroSerializer

schema_str = """
{
  "type": "record",
  "name": "CallCompleted",
  "namespace": "ai.callsphere",
  "fields": [
    {"name": "callId", "type": "string"},
    {"name": "durationSec", "type": "int"},
    {"name": "outcome", "type": "string"},
    {"name": "verticalId", "type": ["null","string"], "default": null}
  ]
}
"""

sr = SchemaRegistryClient({"url": "http://karapace:8081"})
ser = AvroSerializer(sr, schema_str)
p = Producer({"bootstrap.servers": "kafka:9092"})

p.produce(
    topic="call.completed",
    key="abc",
    value=ser({"callId": "abc", "durationSec": 142, "outcome": "booked",
               "verticalId": "real-estate"},
              SerializationContext("call.completed", MessageField.VALUE)),
)
p.flush()

# CI compat check via REST
curl -X POST -H "Content-Type: application/json" \
  --data '{"schema": "...", "schemaType": "AVRO"}' \
  "http://karapace:8081/compatibility/subjects/call.completed-value/versions/latest"

Common pitfalls

• No registry — every consumer guesses; outage at the next field add.
• Registering at runtime — race between producer and consumer; CI is the only safe place.
• NONE compatibility — defeats the point.
• Avro defaults missing — backward compat fails on field add.
• Karapace + Confluent client mismatches — Karapace doesn't implement every normalization feature; test in CI.

FAQ

Confluent vs Karapace? Karapace is API-compatible and Apache 2.0; Confluent is managed and paid. Pick by ops appetite.

Avro vs Protobuf vs JSON Schema? Avro for compactness + evolution; Proto for gRPC alignment; JSON Schema for readability and JSON-native pipelines.

BACKWARD vs FORWARD? BACKWARD: new schema readable by old consumers. FORWARD: old schema readable by new consumers. FULL: both. Default to BACKWARD.

How does CallSphere expose schemas? Internal — but our outbound webhooks reference public CloudEvents dataschema URLs. See /pricing and /demo.

Does it work for non-Kafka? Karapace is Kafka-flavored, but you can use schemas anywhere via REST.

Sources

• Confluent Schema Registry Docs
• Karapace (Aiven-Open) on GitHub
• Schema Evolution and Compatibility (Confluent)
• Schema Evolution in Avro, Protobuf, and JSON Schema