When Multi-Cloud Networking Gets Real
Most multi-cloud networking discussions start with a nice architecture diagram showing three clouds connected by clean lines. Reality looks nothing like that. You're dealing with overlapping IP ranges from acquisitions, different MTU defaults across providers, asymmetric routing through transit gateways, and VPN tunnels that flap at 3 AM because one provider's edge router decided to rotate its IKE key.
I've built interconnects between all three major clouds, and the honest assessment is this: every option involves tradeoffs between cost, latency, bandwidth, and operational complexity. There's no "just connect them" solution.
Dedicated Interconnect Options
Each cloud provider offers a dedicated physical interconnect product. AWS has Direct Connect, GCP has Cloud Interconnect, and Azure has ExpressRoute. These give you private, high-bandwidth connections between your on-premises infrastructure (or a colocation facility) and the cloud provider's network.
For multi-cloud specifically, the trick is finding a colocation facility where all three providers have a point of presence. Equinix is the most common choice — their fabric product lets you establish cross-connects to multiple cloud providers from the same cage. But you're still managing three separate interconnect products with three different APIs, three different billing models, and three different SLAs.
# AWS Direct Connect via Terraform
resource "aws_dx_connection" "equinix_dc" {
name = "equinix-ashburn-dx"
bandwidth = "1Gbps"
location = "EqDC2"
provider_name = "Equinix"
tags = {
Environment = "production"
Purpose = "multi-cloud-interconnect"
}
}
resource "aws_dx_private_virtual_interface" "transit" {
connection_id = aws_dx_connection.equinix_dc.id
name = "transit-vif"
vlan = 100
address_family = "ipv4"
bgp_asn = 65001
dx_gateway_id = aws_dx_gateway.main.id
}VPN-Based Interconnects Between Clouds
For teams that don't need dedicated bandwidth or can't justify the cost, IPsec VPN tunnels between clouds work. They're cheaper, faster to set up, and run over the public internet (or through the cloud provider's backbone, depending on the product).
AWS Site-to-Site VPN connects to GCP's Cloud VPN or Azure's VPN Gateway. The setup isn't complicated, but there are subtleties. GCP's Cloud VPN defaults to IKEv2 and requires specific transform sets that AWS doesn't negotiate automatically. Azure's VPN Gateway has route-based and policy-based modes, and picking the wrong one breaks dynamic routing with BGP.
# GCP HA VPN to AWS - Terraform configuration
resource "google_compute_ha_vpn_gateway" "to_aws" {
name = "ha-vpn-to-aws"
region = "us-east1"
network = google_compute_network.main.id
}
resource "google_compute_external_vpn_gateway" "aws_gw" {
name = "aws-vpn-gateway"
redundancy_type = "TWO_IPS_REDUNDANCY"
interface {
id = 0
ip_address = aws_vpn_connection.to_gcp.tunnel1_address
}
interface {
id = 1
ip_address = aws_vpn_connection.to_gcp.tunnel2_address
}
}
resource "google_compute_vpn_tunnel" "to_aws_0" {
name = "tunnel-to-aws-0"
region = "us-east1"
vpn_gateway = google_compute_ha_vpn_gateway.to_aws.id
peer_external_gateway = google_compute_external_vpn_gateway.aws_gw.id
shared_secret = var.vpn_shared_secret
router = google_compute_router.vpn_router.id
vpn_gateway_interface = 0
peer_external_gateway_interface = 0
}BGP Configuration Across Clouds
Dynamic routing with BGP is essential for multi-cloud. Static routes break whenever you add a new subnet, and they don't handle failover. All three cloud VPN products support BGP, but they each handle it slightly differently.
AWS uses a private ASN for its VPN endpoints (default 64512). GCP lets you choose the ASN for the Cloud Router. Azure assigns an ASN per VPN Gateway. You'll need to plan your ASN allocation before building anything — overlapping ASNs cause route advertisement failures that are painful to debug.
I'd recommend reserving a contiguous block from the private ASN range (64512-65534) and documenting the allocation: one ASN per cloud account per region. It sounds like overkill early on, but it prevents headaches when you're adding the fourth or fifth interconnect.
IP Address Planning for Multi-Cloud
This is where most implementations go wrong. Each cloud defaults to RFC 1918 ranges that overlap. AWS defaults to 172.31.0.0/16 for default VPCs. GCP defaults to auto-mode subnets across 10.128.0.0/9. Azure defaults to 10.0.0.0/16 for new VNets. If you don't plan your CIDR allocation upfront, you end up with overlaps that require NAT translation between clouds — and cross-cloud NAT adds latency, complexity, and failure modes.
What works: allocate a /8 from RFC 1918 space and carve it up by cloud and region.
# Example CIDR allocation plan
# 10.0.0.0/8 - Total allocation
# AWS: 10.0.0.0/10 (10.0.0.0 - 10.63.255.255)
# us-east-1: 10.0.0.0/12
# us-west-2: 10.16.0.0/12
# eu-west-1: 10.32.0.0/12
# GCP: 10.64.0.0/10 (10.64.0.0 - 10.127.255.255)
# us-east1: 10.64.0.0/12
# us-central1: 10.80.0.0/12
# europe-west1: 10.96.0.0/12
# Azure: 10.128.0.0/10 (10.128.0.0 - 10.191.255.255)
# eastus: 10.128.0.0/12
# westus2: 10.144.0.0/12
# westeurope: 10.160.0.0/12
# Reserved: 10.192.0.0/10 (future use)DNS Resolution Across Clouds
Private DNS is the piece that makes cross-cloud networking actually usable for application teams. Without it, services need to know IP addresses directly, which defeats the purpose of having DNS in the first place.
The pattern that's worked for us: run a central DNS zone (we use Route 53 as the primary) with conditional forwarding to cloud-specific private zones. GCP's Cloud DNS forwards queries for *.gcp.internal to GCP's private zone. Azure Private DNS zones handle *.azure.internal. Route 53 resolves everything else and acts as the forwarding hub.
This requires DNS forwarding endpoints in each cloud's VPC/VNet, and those endpoints need network connectivity back to the central DNS. It's circular — you need DNS to find services, but you need network connectivity to reach DNS. The bootstrap solution is to hardcode the DNS forwarder IPs (they're stable within a reservation) and configure them via DHCP options in each cloud.
Monitoring Cross-Cloud Connectivity
Don't rely on VPN tunnel status alone. A tunnel can show "up" while dropping 30% of packets due to MTU mismatches or asymmetric routing. You need active probing between endpoints in each cloud — synthetic ping tests, TCP connection checks, and throughput measurements.
We run a lightweight agent in each VPC that sends probes to its counterparts every 30 seconds. The probes test ICMP, TCP on port 443, and a 1400-byte UDP payload (to catch MTU issues). Results go to a central Prometheus instance, and we alert on packet loss exceeding 1% or latency exceeding 2x the baseline for that link.
The MTU testing catches a specific class of problems. AWS VPN tunnels default to 1500 MTU but the IPsec overhead reduces the effective payload. GCP HA VPN uses 1460 MTU by default. If you don't set mss_clamping on the VPN tunnels, large packets get fragmented or dropped silently. TCP connections work fine (MSS negotiation handles it), but UDP-based protocols break in confusing ways.
Security Considerations for Cross-Cloud Traffic
Traffic between clouds traverses either the public internet (VPN) or a shared colocation fabric (interconnect). Neither is inherently private. VPN tunnels encrypt traffic with IPsec, which handles confidentiality, but you still need to think about authentication and authorization at the application layer.
Service mesh solutions like Istio can extend mTLS across cloud boundaries, but the setup is considerably more complex than single-cluster mTLS. Each cloud's service mesh deployment needs to share a common root certificate authority, and the mesh control planes need to discover services across clouds. Consul Connect handles this better than most alternatives because it was designed for multi-datacenter operation from the start, but it adds another piece of infrastructure to manage.
A pragmatic approach for most teams: encrypt at the transport layer (VPN tunnels or interconnect with MACsec) and authenticate at the application layer (API keys, JWT tokens, or mutual TLS between specific services). Don't try to build a unified service mesh across all three clouds unless you have a dedicated team to maintain it — the operational cost outweighs the security benefit for most organizations.
Cost Modeling for Multi-Cloud Networking
Networking costs are the sneaky part of multi-cloud. Each cloud charges for egress differently. AWS charges $0.09/GB for internet egress, GCP charges $0.08-$0.12/GB depending on destination, and Azure charges $0.05-$0.087/GB. VPN traffic counts as internet egress on most clouds. Dedicated interconnect pricing varies by bandwidth and location but typically runs $0.02-$0.05/GB — a significant saving at high volumes.
For a workload transferring 10 TB/month between AWS and GCP via VPN, the egress alone costs roughly $900/month from AWS plus $800 from GCP. The same traffic over a dedicated interconnect through Equinix costs around $200-$500/month in egress plus the interconnect port fee. At volumes above 5 TB/month, dedicated interconnect usually breaks even within a few months.
Data transfer between VPCs within the same cloud region is cheaper than cross-cloud transfer but not free. AWS charges $0.01/GB for cross-AZ traffic and free for same-AZ. GCP charges for cross-region but not cross-zone within a region. These differences affect architecture decisions — placing services that communicate heavily in the same cloud and region saves more than most optimization efforts in the application layer.