Skip to main content

Virtual topics

A virtual topic is to a Kafka topic what a SQL view is to a table. It is a topic name that exists only on the gateway, backed by a physical topic and optionally filtered. Clients produce, consume, commit offsets and join groups on it like on any topic. Nothing is copied: partitions, offsets, retention and storage are the physical topic's.

Consumers use the virtual topics orders.eu, orders.us and orders, backed by the hidden physical topic orders-v2, next to the ordinary physical topic payments, which passes through unchanged

That makes the topic name an interface you control, independent of where the data lives:

  • Stable names: clients bind to orders, not to orders-v2. Repointing a virtual topic switches producers at once, but committed offsets belong to the old physical topic: consumer groups restart on the new one from auto.offset.reset, so plan the cutover.
  • Filtered views without copies: one physical topic, many consumer-specific views. No stream-processing job, no duplicate storage; filtered records never leave the gateway.
  • Per-view access: ACLs apply to the virtual name, so a team can read orders.eu but not orders-v2.
  • Hidden physical layout: a physical topic behind a virtual topic is left out of Metadata (unless exposePhysicalTopic), so clients do not depend on its name. Other physical topics stay visible and usable as before.

Definition​

FieldDefaultDescription
namerequiredName clients use. Unique on the gateway.
topicrequiredPhysical topic. Need not exist yet.
exposePhysicalTopicfalseAlso list the physical topic in Metadata.
filternoneConsume filter.
valueFormatnoneValue format, so cel filters see decoded values.

A name that is both virtual and physical resolves to the virtual topic.

Filters​

Applied during Fetch. type selects the filter:

TypeKeeps a record whenFields
headerEqualsheader header equals valueheader, value
headerContainsheader header contains valueheader, value
headerStartsWithheader header starts with valueheader, value
headerMatchesheader header fully matches regex value (anchored)header, value
celthe CEL expression is trueexpression

An invalid headerMatches regex is rejected when the topic is written.

CEL bindings:

BindingTypeNotes
keystring"" without a key.
valuestring or decoded valueRaw string, or with a valueFormat the decoded record (value.region).
headersmap(string, string)Test presence with has(headers.region).
timestampintEpoch milliseconds.

Examples as VirtualTopic operator resources:

VirtualTopic orders.us: header filter
apiVersion: kawa-platform.github.io/v1alpha1
kind: VirtualTopic
metadata:
name: orders.us
spec:
physicalTopic: orders-v2
filter:
type: headerEquals
header: region
value: us
VirtualTopic orders.eu: CEL filter
apiVersion: kawa-platform.github.io/v1alpha1
kind: VirtualTopic
metadata:
name: orders.eu
spec:
physicalTopic: orders-v2
filter:
type: cel
expression: headers.region == "eu" && value.severity != "nominal"

Filtering drops records from the fetched batches without renumbering: surviving records keep their physical offsets and dropped ones leave gaps, as on a compacted topic. Commits, seeks and rebalances work unchanged. CEL expressions are compiled once, when the filter is created.

Value formats​

TypeValue
jsonA JSON document.
avroSchema registry wire format. Needs the cluster's schemaRegistry.

onDecodeError: skip (default) drops the record, include delivers it, fail fails the fetch.

Managing​

Create virtual topics in Studio, or declare them as VirtualTopic resources with the operator. Changes are checked by governance and apply live. A virtual topic owned by the operator can only be changed through its resource.

VirtualTopic operator resource​

Requires the Kubernetes operator.

apiVersion: kawa-platform.github.io/v1alpha1
kind: VirtualTopic
metadata:
name: orders-eu
namespace: team-a
spec:
virtualTopic: orders.eu
physicalTopic: orders-v2
filter:
type: headerEquals
header: region
value: eu
FieldDescription
physicalTopicRequired. topic above.
virtualTopicname above. Defaults to metadata.name; set it for names Kubernetes rejects. Immutable.
exposePhysicalTopic, filter, valueFormatAs above.
deletionPolicyDelete (default) removes the virtual topic on deletion, Retain keeps it.

Ownership, adoption and status are described in Kubernetes operator.