Architecture: Private NAT and Overlapping IP Ranges in AWS

Codelooru Private NAT and Overlapping IP Ranges in AWS

Two teams, two projects, two VPCs. Neither team talked to the other about addressing, so both grabbed the obvious default: 10.0.0.0/16. Everything worked fine in isolation. Then a business requirement lands: an application in VPC A needs to call a service in VPC B. You reach for VPC peering, and AWS rejects it. You reach for a Transit Gateway, attach both VPCs, and watch one of your routes silently vanish from the route table.

The problem is not a bug. It is arithmetic. Both networks claim the same address space, and no router on earth can deliver a packet to 10.0.0.5 when that address exists in two different places at once.

This post builds the standard AWS solution to that problem from the ground up: secondary CIDRs, a private NAT gateway for source translation, an ELB for destination translation, and a Transit Gateway to carry the now-unambiguous traffic between them. Every component earns its place. By the end you will understand not just the wiring, but why each piece has to exist.


Why overlapping ranges break everything

Routing is a lookup. A router holds a table that maps destination ranges to next hops, and when a packet arrives it finds the most specific matching entry and forwards accordingly. The entire model assumes one thing: a destination address identifies exactly one place.

Overlapping CIDRs violate that assumption at the root. If 10.0.0.0/16 lives in both VPC A and VPC B, then a route table entry for 10.0.0.0/16 is ambiguous by construction. Which VPC? There is no answer, so the routing layer refuses to guess.

This is why VPC peering flatly rejects overlapping CIDRs at creation time. A peering connection is a static route injection between two VPCs; if the ranges collide, the local route always wins and the peer is unreachable, so AWS blocks the setup rather than let you build something broken.

A Transit Gateway is more permissive at attach time but fails in a subtler way. You can attach both overlapping VPCs to the same TGW. What you cannot do is propagate both of their overlapping routes into the same TGW route table. The TGW encounters two candidate routes for 10.0.0.0/16, picks one according to its route-evaluation rules, and drops the other. One VPC becomes reachable; the other becomes a black hole. That is worse than a clean failure, because it looks like it might work.

The following diagram shows the core conflict: the identical range on both sides, and the router in the middle with no way to choose.

VPC A 10.0.0.0/16 instance 10.0.0.5 VPC B 10.0.0.0/16 instance 10.0.0.5 Router ? which one? Same destination address, two locations. The lookup is ambiguous. Fig 1. The overlapping CIDR conflict at the routing layer.

The core idea: give traffic a routable identity

Every real solution to overlapping CIDRs comes down to one principle. If a router cannot distinguish two addresses because they are identical, then before the traffic reaches that router, rewrite it into addresses that are distinct.

That rewriting is Network Address Translation. NAT swaps the source or destination IP of a packet for a different one, keeps a translation record, and reverses the swap on the return path. It is the same mechanism your home router uses to let many devices share one public IP. Here we use it for a different purpose: not to conserve addresses, but to manufacture unambiguous ones.

The AWS pattern for this uses secondary CIDR blocks. You leave the overlapping 10.0.0.0/16 in place, and you add a second, deliberately non-overlapping range to each VPC. In the scenario from the reference diagram, VPC A gains 10.1.0.0/16 and VPC B gains 10.2.0.0/16. These two ranges do not collide with each other or with anything else, so they are routable: a Transit Gateway can carry them without ambiguity.

The plan, then, is to make cross-VPC traffic appear to come from and go to these routable secondary ranges, even though the actual workloads still live on the overlapping 10.0.0.0/16. Two translations make that happen, and they solve two different halves of the problem.

VPC A 10.0.0.0/16 (primary) overlapping — not routable 10.1.0.0/16 (secondary) unique — routable VPC B 10.0.0.0/16 (primary) overlapping — not routable 10.2.0.0/16 (secondary) unique — routable Fig 2. Overlapping primaries stay; unique secondaries carry the cross-VPC traffic.

Component 1: Secondary CIDRs and the routable subnets

A VPC can hold more than one CIDR block. The first is the primary; you can attach additional secondary blocks after creation. This is the foundation the whole design rests on, because it lets you introduce routable address space into a VPC without touching the overlapping primary range or migrating a single workload off it.

Inside VPC A's secondary 10.1.0.0/16, you carve out a small subnet, say 10.1.0.0/24. This subnet does not hold your application. It holds infrastructure: specifically, the network interface of the private NAT gateway. Its only job is to be a routable place from which translated traffic can originate. VPC B gets the mirror image inside 10.2.0.0/16.

The why here is the crucial part for exams. You are not renumbering your workloads. Renumbering a running VPC is painful and often impossible on a business timeline. Instead you bolt on a small, unique range whose sole function is to give traffic a routable identity for the journey between VPCs. The workloads never move.


Component 2: The private NAT gateway and its ENI (source translation)

Here is the first half of the translation problem, and the piece most people get stuck on. Consider an instance in VPC A at 10.0.0.5 that wants to reach a service in VPC B. When its packet leaves, the source address is 10.0.0.5. That address is not routable, because VPC B also has a 10.0.0.5. If the packet somehow arrived at VPC B, the reply would try to go to 10.0.0.5 and loop straight back inside VPC B. The return path would be broken.

So the source address must be rewritten into something routable before the packet ever leaves VPC A. That is the job of the private NAT gateway.

A private NAT gateway is a managed NAT device with no internet connectivity, living on a private IP inside a subnet. You place it in the routable subnet, 10.1.0.0/24. It is represented in the VPC by an elastic network interface, an ENI, which holds a private IP such as 10.1.0.125 drawn from that routable range.

This is why the ENI matters, and why the reference diagrams draw the NAT gateway as a network-interface icon. The ENI is the NAT gateway's presence on the network. It is the routable address that overlapping source traffic gets rewritten to. When the instance's packet passes through the private NAT gateway, its source changes from the non-routable 10.0.0.5 to the routable 10.1.0.125. The NAT gateway records the mapping so it can reverse it later, and now the packet has a source address that exists in exactly one place in the world.

EC2 instance src 10.0.0.5 overlapping Private NAT GW ENI in 10.1.0.0/24 10.1.0.125 source NAT toward TGW src 10.1.0.125 routable Source NAT rewrites 10.0.0.5 to the routable ENI address 10.1.0.125. Fig 3. The private NAT gateway solves the overlapping-source half of the problem.

Component 3: The ELB and destination translation

Source translation only fixes half the problem. The packet now has a routable source, but where is it going? The instance in VPC A wanted to reach a service that lives at some address inside VPC B's overlapping 10.0.0.0/16. If the packet's destination stays as an address in that overlapping range, it is unroutable for the same reason the source was.

So the destination must also be a routable address. This is where an Elastic Load Balancer comes in, and it performs the second translation: destination NAT.

You place an ELB in VPC B's routable subnet, 10.2.0.0/24, so the load balancer presents a routable IP such as 10.2.0.10. Behind it, as targets, sit the real workloads on VPC B's overlapping 10.0.0.0/16. The instance in VPC A is configured to connect to 10.2.0.10, a routable address that exists only in VPC B. When traffic arrives at the ELB, the load balancer forwards it to a target on the overlapping range, rewriting the destination in the process.

The why is symmetry. The private NAT gateway makes the overlapping source routable on the way out. The ELB makes the overlapping destination routable on the way in. Between them, every address that crosses the Transit Gateway is drawn from a unique secondary range, and the ambiguity is gone. Neither device alone is enough; you need both ends translated.


Component 4: The Transit Gateway and its route table

With both source and destination now living in routable ranges, the Transit Gateway finally has something it can work with. The TGW is a regional hub that connects VPCs through attachments and makes forwarding decisions from its own route table, separate from the VPC route tables.

The key discipline is that the TGW route table only ever sees the non-overlapping secondary ranges. It carries an entry sending 10.1.0.0/16 to VPC A's attachment and 10.2.0.0/16 to VPC B's attachment. The overlapping 10.0.0.0/16 never appears in the TGW route table at all, because it never travels across the TGW as either a source or a destination. That is the entire point of the two translations: by the time traffic reaches the hub, it is expressed purely in unique addresses.

This is what removes the black-hole failure from earlier. There is no longer a duplicate 10.0.0.0/16 route for the TGW to choose between and drop. Each secondary range maps to exactly one attachment, and every forwarding decision is deterministic.

TGW Route Table 10.1.0.0/16 → VPC A attach 10.2.0.0/16 → VPC B attach VPC A attachment 10.1.0.0/16 VPC B attachment 10.2.0.0/16 Fig 4. The TGW sees only unique ranges. No overlap, no dropped route, no black hole.

The VPC route tables that tie it together

The Transit Gateway handles the middle of the journey, but each VPC needs local route tables that steer traffic into and out of the NAT and ELB devices correctly. This is the part the reference diagram spends the most space on, and it is worth being precise.

In VPC A, the subnet holding the source instance needs a route for VPC B's routable range. Traffic destined for 10.2.0.0/16 is pointed at the private NAT gateway. That is what forces the outbound packet through source translation before it goes anywhere.

The subnet holding the private NAT gateway needs its own route. Traffic destined for 10.2.0.0/16 is pointed at the Transit Gateway attachment, so that once the packet has been translated, it heads for the hub. The instance subnet points at the NAT gateway; the NAT gateway subnet points at the TGW. The ordering is deliberate.

In VPC B, the routable subnet holding the ELB needs a route back toward VPC A's routable range, 10.1.0.0/16, pointed at the Transit Gateway attachment. This is the return path. Because the source was translated to 10.1.0.125, replies are addressed to 10.1.0.0/16, and VPC B knows to send those back across the TGW.

Notice what every one of these route entries has in common: they reference only the unique secondary ranges. No VPC route table for cross-VPC traffic ever points at the overlapping 10.0.0.0/16, because that range is local and ambiguous. The overlapping range is used only for the final hop, inside the destination VPC, where the ELB delivers to its targets.


Request lifecycle: one packet, end to end

Now walk a single request through the whole system. An instance in VPC A at 10.0.0.5 wants to call a service fronted by the ELB in VPC B.

Step 1. The instance sends a packet to 10.2.0.10, the ELB's routable address. It uses this address because that is what it was configured with; it never learns the overlapping address behind the load balancer. Source is 10.0.0.5, destination is 10.2.0.10.

Step 2. The instance subnet's route table matches 10.2.0.0/16 and forwards the packet to the private NAT gateway.

Step 3. The private NAT gateway performs source NAT. Source becomes 10.1.0.125, the ENI address. Destination is still 10.2.0.10. Both addresses are now routable. The gateway records the mapping.

Step 4. The NAT gateway subnet's route table matches 10.2.0.0/16 and forwards the packet to the Transit Gateway attachment.

Step 5. The TGW route table matches 10.2.0.0/16, sends the packet to VPC B's attachment, and it arrives in VPC B's routable subnet.

Step 6. The packet reaches the ELB at 10.2.0.10. The load balancer performs destination NAT, forwarding to a real target on VPC B's overlapping 10.0.0.0/16. The service receives the request and sees a source of 10.1.0.125, a perfectly routable address.

Step 7. The reply is addressed to 10.1.0.125. VPC B's route table sends 10.1.0.0/16 back across the TGW, the TGW forwards to VPC A's attachment, and the private NAT gateway reverses its mapping, rewriting the destination back to 10.0.0.5. The original instance receives its answer.

Every hop resolved to exactly one place, because at no point on the wire between the two VPCs did an overlapping address appear as either source or destination.

Instance A NAT GW src NAT TGW ELB dst NAT Target B src 10.0.0.5 → 10.2.0.10 src 10.1.0.125 to 10.2.0.10 dst → 10.0.0.x reply to 10.1.0.125 to 10.1.0.125 dst → 10.0.0.5 Fig 5. End-to-end lifecycle. Solid = request, dashed = reply. No overlapping address ever on the wire between VPCs.

Failure modes and fault tolerance

A design is only as good as its behavior when something breaks. Several failure modes are worth understanding here.

Single-AZ NAT gateway failure. A private NAT gateway lives in one Availability Zone. If that AZ fails, the gateway and its ENI go with it, and cross-VPC traffic from that VPC stops. The mitigation is one NAT gateway per AZ, each in its own routable subnet, with per-AZ route tables directing local traffic to the local gateway. This keeps translation available even when one zone is lost.

Asymmetric routing. NAT is stateful. If a request leaves through one NAT gateway but the reply is routed back through a different one, the second gateway has no translation record and drops the packet. This is why per-AZ designs must keep each flow pinned to a consistent path, and why sloppy route-table entries cause intermittent, maddening failures rather than clean outages.

Missing return route. The single most common configuration error is forgetting the return path in the destination VPC. Traffic flows out, reaches the target, and the reply has nowhere to go because VPC B has no route for 10.1.0.0/16 back to the TGW. The connection hangs. When a private-NAT setup half-works, suspect a one-directional route table first.

Port exhaustion. A single NAT gateway supports a large but finite number of simultaneous connections to a unique destination. Very high-fan-out workloads can exhaust available source ports for a given destination. This rarely bites in exam scenarios, but it is a real ceiling in production.


The complete architecture

Assembled, the full picture is the one from the reference diagram, and now every element has a reason. Overlapping primaries stay in place. Unique secondaries provide routable identity. The private NAT gateway translates the overlapping source outbound. The ELB translates the overlapping destination inbound. The Transit Gateway carries only unique ranges between the two. VPC route tables steer traffic through the translation devices in the correct order, and the TGW route table stays free of any overlap.

VPC A Instance subnet 10.0.0.5 (overlapping) route: 10.2.0.0/16 → NAT GW Private NAT GW ENI 10.1.0.125 route: 10.2.0.0/16 → TGW VPC B ELB (routable subnet) 10.2.0.10 route: 10.1.0.0/16 → TGW Target subnet 10.0.0.x (overlapping) ELB targets TGW 10.1.0.0/16 10.2.0.0/16 unique only Fig 6. Complete architecture. Overlapping addresses stay local; only unique ranges cross the TGW.

Summary

The overlapping-CIDR problem is not really a networking limitation. It is the routing model doing exactly what it should: refusing to deliver a packet when the destination could mean two different places. Every part of this architecture exists to remove that ambiguity before traffic reaches a router that would choke on it.

The single insight that makes the whole design click is that you never fix the overlap. You route around it. The overlapping 10.0.0.0/16 stays exactly where it is, workloads untouched. What you add is a routable identity, a unique secondary range, and two translations that swap overlapping addresses for routable ones at both ends of the journey. The private NAT gateway launders the source, the ELB launders the destination, and the Transit Gateway only ever sees addresses that point to exactly one place. Get that principle, and the tangle of route tables in the exam diagram stops being a memorization exercise and becomes obvious.


Related on this blog: Architecture series



×