Apache Kafka

Prev Next

Overview

Replicate Preview enables efficient data movement to and from Apache Kafka, allowing Kafka clusters to be configured as either a source or target for event-streaming workloads. Users can quickly define replication pipelines by selecting Kafka topics as source objects or by writing records to existing topics, with topic schemas inferred from sampled messages or supplied explicitly, so data can be replicated with minimal setup and operational overhead.

Apache Kafka is an open-source distributed event streaming platform used to publish, subscribe to, store, and process streams of events (also called records or messages) that are organized into topics and spread across partitions on one or more broker servers. For more information about Kafka's architecture and terminology, refer to the official Apache Kafka Introduction.

Note

  • Apache Kafka is currently supported as both source and target in Replicate Preview, limited to Snapshot (full load/full-refresh) replication mode; Change Data Capture (CDC) and streaming replication are not yet supported.

  • Ensure that the Kafka server is running when executing a replication from Replicate Preview.

Prerequisites

The following steps must be completed before using the Kafka datastore as a source or target:

  • Ensure the Kafka cluster is reachable over the network using comma-separated host:port bootstrap server addresses over a PLAINTEXT listener.

  • Ensure the topics you intend to replicate already exist on the cluster. Automatic topic creation is not supported, topics must be created and configured directly in Kafka beforehand.

  • Ensure the required users/principals are granted the appropriate Kafka Access Control Lists (ACLs) on the topics being replicated. Refer to the User Permissions section below.

  • Create a Kafka datastore connection and assign Metadata Scanning and Cloud Replication usage types. Refer to Create a Connection and Connecting to Apache Kafka for more information.

  • If you are using this connection as a source, run a scan to ingest metadata from this connection's objects into the SKP.

User Permissions

Apache Kafka controls access with a pluggable authorization framework that evaluates ACLs against the connecting principal, the operation being performed (for example, Read, Write, Describe), and the resource being accessed (for example, a Topic). Refer to the official Kafka Authorization and ACLs documentation for the full authorization model.

When using Kafka as a source, the connecting principal needs at least Read and Describe permissions on each topic being replicated, equivalent to Kafka's built-in consumer role:

bin/kafka-acls.sh --bootstrap-server <broker:port> --add --allow-principal User:<principal> --consumer --topic <topic-name> --group "*"

For targets, the connecting principal needs at least Write and Describe permissions on each topic being written to, equivalent to Kafka's built-in producer role. Because Replicate Preview only writes to pre-existing topics, the Create permission is not required:

bin/kafka-acls.sh --bootstrap-server <broker:port> --add --allow-principal User:<principal> --producer --topic <topic-name>

If no ACL matches a resource, Kafka denies access to that resource by default (only super users are allowed), unless the cluster administrator has enabled allow.everyone.if.no.acl.found=true. Refer to the Kafka Authorization and ACLs documentation, "Behavior Without ACLs" section, for more information.

Apache Kafka as a Source

In a Replicate Preview workspace, create a source datastore using the Apache Kafka datastore connection. Refer to the Creating Source Datastores section for more information.

Create the required replications either by using the Source Table Selection dialog box or by adding them manually. Each Kafka topic is represented as a source table; reads are bounded by the partition offsets captured when the replication run starts, so events published to a topic after a run begins are not included in that run.

Note

Kafka reads use a dedicated, unique consumer group for each run and never commit offsets back to the cluster, so replications do not interfere with other Kafka consumers of the same topics.

Apache Kafka as a Target

In a Replicate Preview workspace, create a target datastore using the Apache Kafka datastore connection. Refer to the Creating Target Datastores section for more information.

Note

Configure your target replication settings with the Append write option; Kafka targets write only to a pre-existing topic and do not support Truncate, Drop, or Upsert.