Your primary Kafka cluster handles millions of events per day without complaint. Then a cloud provider has a regional outage. The brokers are unreachable, the consumer lag metrics are gone, and your on-call engineer is staring at a 45-minute RTO commitment they have no clear path to meeting. The standard answer for years was MirrorMaker 2: deploy a separate replication cluster, manage connector configs, accept that consumer offsets won't translate cleanly, and build a runbook for manually redirecting clients. That answer has not aged well.
Confluent Cluster Linking is a fundamentally different approach. It moves replication inside the broker layer itself, eliminates the need for a separate replication process, and preserves consumer offsets exactly across cluster boundaries. This post covers how it works, what it is made of, and every significant use case it enables.
The big picture
Cluster Linking creates a direct, persistent connection between two Confluent clusters. One cluster acts as the source; the other hosts mirror topics that are byte-for-byte replicas of their counterparts on the source. Crucially, mirror topics are read-only on the destination. Producers cannot write to them. The only writer is the cluster link itself.
The mechanism lives entirely inside the brokers. There is no MirrorMaker process, no Kafka Connect cluster, no separate replication service to deploy or monitor. The destination cluster's brokers fetch data directly from the source cluster's brokers, using the same internal replication protocol that Kafka uses for leader-follower replication within a cluster.
The result is that a consumer group sitting on the destination cluster can read a mirror topic using the exact same offset it had on the source. There is no offset translation table, no remapping, no consumer group reset required on failover. This is the detail that makes Cluster Linking qualitatively different from every previous approach.
Core components
The cluster link object
A cluster link is a named configuration object created on the destination cluster. It specifies the source cluster's bootstrap servers, the authentication credentials needed to reach it, and a set of replication behaviors. Once created, it stays alive indefinitely and continuously fetches new data as it arrives on the source.
Creating a cluster link looks like this using the Confluent CLI:
confluent kafka link create my-dr-link \
--source-cluster-id lkc-abc123 \
--source-bootstrap-server pkc-xyz.us-east-1.aws.confluent.cloud:9092 \
--source-api-key SOURCE_KEY \
--source-api-secret SOURCE_SECRETThe link itself has no topic scope at creation time. Topics are linked separately, which means you can selectively mirror a subset of topics rather than everything on the source cluster.
Mirror topics
A mirror topic is the destination-side artifact. It is created on the destination cluster referencing the cluster link and the name of the source topic to replicate. The mirror topic automatically inherits the partition count, replication factor, and configuration of the source topic at creation time.
confluent kafka mirror create orders \
--link my-dr-linkFrom that moment, the destination cluster begins fetching data for orders from the source. The mirror topic's partition count matches the source exactly. Offsets are replicated, not recalculated. A message that arrives at offset 1,500,000 on the source will be at offset 1,500,000 on the mirror.
Mirror topics are enforced as read-only by the broker. Any attempt to produce directly to a mirror topic is rejected with an error. This is not advisory; it is a hard broker-level constraint.
The link fetch mechanism
Under the hood, a cluster link operates using a variant of Kafka's standard partition replication protocol. The destination cluster brokers act as a special type of follower replica for each mirrored partition. They open fetch sessions to the source brokers and pull batches of records continuously.
This means the replication path is efficient: it uses the same zero-copy fetch path that inter-broker replication uses, and it does not require deserializing records. The bytes are transferred as-is, which is why offsets can be preserved. There is no re-encoding, no re-partitioning, and no re-compression step.
Consumer offset replication
Consumer group offsets are stored in the internal __consumer_offsets topic on each cluster. Cluster Linking can replicate these offsets to the destination cluster as part of the link configuration. This is what enables zero-touch failover: when consumers reconnect to the destination cluster after a failover event, their committed offsets are already there and valid.
Offset replication is configured with the consumer.offset.sync.enable and consumer.offset.sync.ms settings on the link. You control the sync interval; the tradeoff is between offset staleness and the volume of internal metadata traffic.
Schema Registry and Cluster Linking
Most Confluent deployments use Schema Registry to enforce schema contracts on topics. When you replicate topics via Cluster Linking, the message bytes arrive intact, but consumers on the destination cluster still need schema definitions to deserialize them.
Confluent offers bidirectional Schema Registry replication as a separate capability. You configure it independently of the cluster link. The destination Schema Registry pulls schema definitions from the source, preserving schema IDs. Because the schema IDs embedded in message bytes are identical across source and destination, consumers on the destination cluster can deserialize replicated messages without any changes to their deserialization code.
If you do not replicate schemas, consumers on the destination will fail to deserialize records after failover unless they can still reach the source Schema Registry — which may be unavailable in a DR scenario. Schema replication is effectively mandatory for DR use cases.
Replication lag and monitoring
A cluster link has an observable replication lag: the difference between the latest offset on the source topic and the latest offset that has been replicated to the mirror. This is surfaced in Confluent metrics as kafka.cluster_link.replication_latency_ms and as an offset lag count per partition.
In a healthy, well-connected link the lag is typically under a few seconds. Lag spikes indicate network congestion, source cluster overload, or a link configuration problem. Monitoring this metric is essential for DR readiness: an RPO commitment of, say, 30 seconds is only meaningful if you can verify the link is actually within 30 seconds of lag at all times.
Confluent Cloud exposes link health and per-topic mirror status through both the Cloud Console and the Metrics API, so these can feed directly into standard alerting pipelines.
Use case: disaster recovery
DR is the most common motivation for deploying Cluster Linking. The topology is active/passive: producers write to the primary cluster, a cluster link continuously replicates to a secondary cluster in a separate region or availability zone, and consumers are redirected to the secondary only in the event of a primary failure.
Failover
When the primary is unavailable, the operator promotes mirror topics on the secondary to writable by running a single command per topic:
confluent kafka mirror promote orders --link my-dr-linkPromoting a mirror topic severs its link to the source and converts it into a regular, writable topic. Producers can now write to it. Because consumer offsets were already synchronized to the secondary, consumers can reconnect and resume from exactly where they left off, with no offset reset and no duplicate processing caused by offset remapping.
Failback
Once the primary is restored, you have two options. The simpler option is to reverse the cluster link: create a new link from the secondary (now acting as primary) back to the original primary, and mirror the topics in reverse to bring it back in sync. Once caught up, promote the original primary's topics and redirect traffic back.
The more complex option is to treat the DR cluster as the new permanent primary and re-establish a forward link to the original cluster as the new DR target. Both are supported. Which you choose depends on whether your infrastructure is symmetric and whether the runbook complexity of the two-link approach is acceptable to your team.
RTO and RPO characteristics
With Cluster Linking, RPO (recovery point objective) is bounded by the replication lag. In a well-tuned deployment over a reliable network this is typically measured in seconds, not minutes. You can observe and alert on this value continuously, which means RPO is no longer a theoretical estimate: it is a measured, live metric.
RTO (recovery time objective) is bounded by two things: how quickly the outage is detected, and how long the promotion command takes to run. The promotion itself is near-instant. The practical RTO in a production setup with automated failover monitoring is therefore a few minutes, dominated by detection time rather than by the mechanics of the failover itself.
Use case: geo-replication
Some applications need to serve consumers in multiple geographic regions without shipping every consumer to a single cluster. A news feed service might produce events globally but want analytics consumers in each region to consume from a local cluster for latency and data residency reasons.
Cluster Linking supports this with a hub-and-spoke or full-mesh topology. A central production cluster holds the authoritative data; regional clusters each have a cluster link to the center and mirror the relevant topics. Consumers in each region read from their local cluster. Replication lag introduces a slight data freshness delay, but consumers are not crossing regional network boundaries on every fetch, which can reduce per-message latency significantly and provides resilience if the central cluster becomes unreachable.
Because offset preservation is maintained, a consumer can also fail over between regional clusters without losing its position, which is useful in architectures where a region itself needs an active/passive DR pair.
Use case: cloud migration
Migrating a self-managed Kafka cluster to Confluent Cloud without downtime is a well-understood Cluster Linking use case. The approach is a live cutover:
- Create a cluster link from the on-premises cluster to the Confluent Cloud cluster.
- Mirror all topics. Wait for lag to reach near-zero.
- Stop producers on the on-premises cluster.
- Wait for lag to hit exactly zero (all messages replicated).
- Promote mirror topics on Confluent Cloud.
- Redirect producers and consumers to the cloud cluster.
Because consumer offsets are replicated, consumers resume from the correct position on Confluent Cloud. The migration window is determined by how long it takes lag to drain to zero after producers stop, which in practice is seconds. The total downtime window is controlled and planned rather than forced by the migration mechanics.
Confluent supports Cluster Linking from self-managed Kafka clusters running version 2.4 or later as source clusters, in addition to Confluent Platform and Confluent Cloud sources.
Use case: development and staging environments
Production data is the most realistic input for testing new consumer logic, schema changes, or throughput benchmarks. Cluster Linking can replicate a subset of production topics into a staging or development cluster, giving engineers a live data environment without direct access to production infrastructure.
The mirror topics on the dev cluster are read-only, which means a misconfigured test producer cannot accidentally contaminate the replicated data. Access to the cluster link itself can be controlled separately from access to the topics, so you can grant engineers broad read access to the staging cluster without granting them any credentials to the production cluster.
This pattern also works for load testing: replicate production traffic to a test cluster and replay it against a new consumer implementation at scale, using real message shapes and real offset distributions.
Use case: multi-cloud and hybrid topologies
Organizations increasingly run workloads across multiple cloud providers, either deliberately for resilience or as a result of acquisitions and organic platform growth. Cluster Linking is transport-agnostic at the connectivity level: as long as the destination cluster can reach the source cluster's bootstrap servers over TLS, a link can be established regardless of whether the clusters are on AWS, GCP, Azure, or a data center.
A common topology is an AWS primary with a GCP secondary. Traffic between them uses public endpoints with mutual TLS authentication. The replication lag will be slightly higher than intra-cloud replication due to inter-provider network latency, but for use cases where an RPO of a few tens of seconds is acceptable, this is entirely workable.
Confluent Cloud's private networking options (AWS PrivateLink, Azure Private Link, GCP Private Service Connect) can be used to keep replication traffic off the public internet for data sovereignty or compliance reasons, though this requires more network configuration effort.
Configuration reference: key settings
The behavior of a cluster link is tunable through a set of configuration properties set at link creation time or modified afterward. The most operationally significant ones:
consumer.offset.sync.enable — enables periodic consumer offset synchronization from source to destination. Default is false; set to true for any DR or migration use case.
consumer.offset.sync.ms — interval in milliseconds at which consumer offsets are synced. Lower values give fresher offsets on the destination at the cost of additional internal metadata traffic. Default is 30000 (30 seconds).
topic.config.sync.ms — interval at which topic configuration changes on the source (retention, compaction settings) are synced to mirror topics. Important for keeping DR mirrors consistent with the source's operational configuration.
auto.create.mirror.topics.enable — when set to true, any new topic created on the source that matches a configured filter is automatically mirrored on the destination. Useful for large clusters where manual mirror creation for every new topic would be error-prone.
auto.create.mirror.topics.filters — a JSON filter specification controlling which source topics are automatically mirrored. Supports include and exclude patterns based on topic name prefix or regex.
{
"topicFilters": [
{
"name": "orders.*",
"patternType": "PREFIXED",
"filterType": "INCLUDE"
},
{
"name": "orders.internal.*",
"patternType": "PREFIXED",
"filterType": "EXCLUDE"
}
]
}This configuration mirrors all topics starting with orders. except those starting with orders.internal.. The filter is evaluated at topic creation time on the source; existing topics are not retroactively affected unless mirrored explicitly.
Security model
A cluster link authenticates to the source cluster using credentials specified at link creation time. On Confluent Cloud these are an API key and secret. On Confluent Platform the link supports SASL/PLAIN, SASL/SCRAM, and mTLS. In all cases, traffic between source and destination brokers is encrypted with TLS.
The credentials used for the link only need read permissions on the source cluster. The principle of least privilege applies directly: the service account used for the link should have DescribeConfigs and Read on the mirrored topics, and nothing else.
On the destination cluster, administering links and promoting mirror topics are separate privileges from reading and producing to topics. This matters for operator access control: the engineer who can trigger a failover (promote a topic) does not necessarily need to be the same person who can produce to the resulting topic.
Operational lifecycle: a request walkthrough
Let's trace the full journey of a single message from producer through a cluster link to a consumer executing a failover.
The key observation in this walkthrough is step 8. The consumer does not ask "what offset do I need to translate my old offset to?" It asks "what is my committed offset for this consumer group on this topic?" and the answer is already there, identical to what it was on the primary, because offset sync wrote it there during normal operations.
Failure modes and limitations
Network partition between clusters
If connectivity between source and destination is interrupted, the cluster link pauses replication. It does not fail the link permanently; it retries on a backoff schedule. When connectivity resumes, replication catches up from the last replicated offset. The mirror topic is not corrupted, and no manual intervention is needed beyond restoring network connectivity.
During the partition, the destination mirror reflects the state of the source at the moment connectivity was lost. Consumers reading from the mirror continue to function, but they will not see new messages until the link resumes.
Source topic deletion
If a topic is deleted on the source cluster while a mirror exists on the destination, the mirror topic is not automatically deleted. The link for that topic enters a failed state and produces an error in the link health metrics. The mirror topic continues to exist as a read-only snapshot of the data at the time of deletion. This is actually useful in some archival scenarios, but it means your monitoring needs to detect and alert on link topic errors rather than assuming a healthy link means all topics are healthy.
Partition count changes
Kafka does not support reducing the partition count of an existing topic. Increasing the partition count of a source topic while a mirror link is active is supported: the new partitions are automatically added to the mirror. However, changing partition count affects key-based routing guarantees, so this is an application-level concern rather than a Cluster Linking concern specifically.
Mirror topic is not a replica for durability
A common misconception is that a cluster link makes the destination cluster a hot standby in the same sense as an in-cluster replica. It does not. The destination cluster holds a copy of the data, but if the source cluster loses data (through a misconfigured retention policy, accidental deletion, or corruption) before the link replicates it, that data is gone. The link replicates what is on the source; it is not a defense against data loss on the source itself. Separate backup strategies are still required.
Confluent-only constraint
Cluster Linking requires at least one Confluent endpoint. Source clusters must be Confluent Platform 6.0 or later, or Confluent Cloud. You cannot use Cluster Linking to replicate between two vanilla Apache Kafka clusters. If your source is community Kafka, MirrorMaker 2 remains the only option until a migration to Confluent Platform is completed.
Cluster Linking vs MirrorMaker 2
It is worth being explicit about why Cluster Linking replaces MirrorMaker 2 in most Confluent deployments rather than complementing it.
MirrorMaker 2 operates at the consumer/producer layer: it consumes records from the source and produces them to the destination as new records. This means offsets on the destination are entirely different from offsets on the source. MirrorMaker 2 maintains an offset translation mapping in a dedicated topic, but that mapping introduces operational complexity, translation lag, and a failure surface of its own. Consumer groups that need to resume on the destination after failover must use the offset translation API or accept starting from the beginning or end of the topic.
MirrorMaker 2 is also a separate process: a Kafka Connect cluster that you must deploy, size, monitor, and maintain independently. It has its own lag metric that is distinct from the underlying replication lag, and its connector configuration is verbose and error-prone at scale.
Cluster Linking eliminates all of this. The tradeoff is the Confluent dependency and the lack of flexibility for transforming or filtering messages during replication. If you need to apply transformations in-flight (masking fields, re-partitioning by a different key, filtering sensitive records), MirrorMaker 2 with SMTs (Single Message Transforms) or a purpose-built pipeline is still the correct answer. For pure replication with failover semantics, Cluster Linking is the right tool.
Summary
Cluster Linking moves replication inside the broker layer. That architectural decision has a cascade of consequences: no separate replication process, no offset translation, no deserialization overhead, and a replication lag that is measurable in seconds rather than minutes. Consumer groups survive failover intact because their committed offsets are already on the destination cluster before the failover is triggered.
The full range of use cases — DR, geo-replication, cloud migration, environment mirroring, multi-cloud topologies — all derive from the same two properties: byte-identical replication and offset preservation. Once you understand those two invariants, every use case follows naturally from them.
The operational model is deliberately minimal. A cluster link is a named object you create once and monitor continuously via replication lag. Topic mirroring is explicit or filter-driven. Failover is a single promote command. Failback is a reverse link. The complexity budget that MirrorMaker 2 consumed in connector management and offset mapping is simply gone.
Where Cluster Linking does not apply: non-Confluent source clusters, use cases requiring in-flight transformation, and scenarios where source-side data loss protection is needed in addition to replication. For everything else in a Confluent environment, it is the right replication primitive.
Related on this blog: Architecture series