To discuss VXLAN gateways, we must first discuss VXLAN itself. Recall that traditional VLANs (Virtual Local Area Networks) use 12-bit VLAN IDs to divide networks, supporting up to 4096 logical networks. This works fine for small networks, but in modern data centers, with their thousands of virtual machines, containers, and multi-tenant environments, VLANs are insufficient. VXLAN was born, defined by the Internet Engineering Task Force (IETF) in RFC 7348. Its purpose is to extend the Layer 2 (Ethernet) broadcast domain over Layer 3 (IP) networks using UDP tunnels.
Simply put, VXLAN encapsulates Ethernet frames within UDP packets and adds a 24-bit VXLAN Network Identifier (VNI), theoretically supporting 16 million virtual networks. This is like giving each virtual network an "identity card," allowing them to move freely on the physical network without interfering with each other. The core component of VXLAN is the VXLAN Tunnel End Point (VTEP), which is responsible for encapsulating and decapsulating packets. VTEP can be software (such as Open vSwitch) or hardware (such as the ASIC chip on the switch).
Why is VXLAN so popular? Because it perfectly aligns with the needs of cloud computing and SDN (Software-Defined Networking). In public clouds like AWS and Azure, VXLAN enables seamless extension of tenants' virtual networks. In private data centers, it supports overlay network architectures like VMware NSX or Cisco ACI. Imagine a data center with thousands of servers, each running dozens of VMs (Virtual Machines). VXLAN allows these VMs to perceive themselves as part of the same Layer 2 network, ensuring smooth transmission of ARP broadcasts and DHCP requests.
However, VXLAN isn't a panacea. Operating on an L3 network requires L2-to-L3 conversion, which is where the gateway comes in. The VXLAN gateway connects the VXLAN virtual network with external networks (such as traditional VLANs or IP routing networks), ensuring data flows from the virtual world to the real world. The forwarding mechanism is the heart and soul of the gateway, determining how packets are processed, routed, and distributed.
The VXLAN forwarding process is like a delicate ballet, with each step from source to destination being closely linked. Let's break it down step by step.
First, a packet is sent from the source host (such as a VM). This is a standard Ethernet frame containing the source MAC address, destination MAC address, VLAN tag (if any), and payload. Upon receiving this frame, the source VTEP checks the destination MAC address. If the destination MAC address is in its MAC table (obtained through learning or flooding), it knows which remote VTEP to forward the packet to.
The encapsulation process is crucial: the VTEP adds a VXLAN header (including the VNI, flags, and so on), then an outer UDP header (with a source port based on a hash of the inner frame and a fixed destination port of 4789), an IP header (with the source IP address of the local VTEP and the destination IP address of the remote VTEP), and finally an outer Ethernet header. The entire packet now appears as a UDP/IP packet, looks like normal traffic, and can be routed on the L3 network.
On the physical network, the packet is forwarded by a router or switch until it reaches the destination VTEP. The destination VTEP strips off the outer header, checks the VXLAN header to ensure the VNI matches, and then delivers the inner Ethernet frame to the destination host. If the packet is unknown unicast, broadcast, or multicast (BUM) traffic, the VTEP replicates the packet to all relevant VTEPs using flooding, relying on multicast groups or unicast header replication (HER).
The core of the forwarding principle is the separation of the control plane and the data plane. The control plane uses Ethernet VPN (EVPN) or the Flood and Learn mechanism to learn MAC and IP mappings. EVPN is based on the BGP protocol and allows VTEPs to exchange routing information, such as MAC-VRF (Virtual Routing and Forwarding) and IP-VRF. The data plane is responsible for actual forwarding, using VXLAN tunnels for efficient transmission.
However, in actual deployments, forwarding efficiency directly impacts performance. Traditional flooding can easily cause broadcast storms, especially in large networks. This leads to the need for gateway optimization: gateways not only connect internal and external networks but also act as proxy ARP agents, handle route leaks, and ensure the shortest forwarding paths.
Centralized VXLAN Gateway
A centralized VXLAN gateway, also called a centralized gateway or L3 gateway, is typically deployed at the edge or core layer of a data center. It acts as a central hub, through which all cross-VNI or cross-subnet traffic must pass.
In principle, a centralized gateway acts as the default gateway, providing Layer 3 routing services for all VXLAN networks. Consider two VNIs: VNI 10000 (subnet 10.1.1.0/24) and VNI 20000 (subnet 10.2.1.0/24). If VM A in VNI 10000 wants to access VM B in VNI 20000, the packet first reaches the local VTEP. The local VTEP detects that the destination IP address is not on the local subnet and forwards it to the centralized gateway. The gateway decapsulates the packet, makes a routing decision, and then re-encapsulates the packet into a tunnel to the destination VNI.
The advantages are obvious:
○ Simple management All routing configurations are centralized on one or two devices, allowing operators to maintain only a few gateways to cover the entire network. This approach is suitable for small and medium-sized data centers or environments deploying VXLAN for the first time.
○ Resource efficient Gateways are typically high-performance hardware (such as the Cisco Nexus 9000 or Arista 7050) capable of handling massive amounts of traffic. The control plane is centralized, facilitating integration with SDN controllers such as NSX Manager.
○ Strong security control Traffic must pass through the gateway, facilitating the implementation of ACLs (Access Control Lists), firewalls, and NAT. Imagine a multi-tenant scenario where a centralized gateway can easily isolate tenant traffic.
But the shortcomings cannot be ignored:
○ Single point of failure If the gateway fails, L3 communication across the entire network is paralyzed. Although VRRP (Virtual Router Redundancy Protocol) can be used for redundancy, it still carries risks.
○ Performance bottleneck All east-west traffic (communication between servers) must bypass the gateway, resulting in a suboptimal path. For example, in a 1000-node cluster, if the gateway bandwidth is 100Gbps, congestion is likely to occur during peak hours.
○ Poor scalability As network scale grows, gateway load increases exponentially. In a real-world example, I've seen a financial data center using a centralized gateway. Initially, it ran smoothly, but after the number of VMs doubled, latency skyrocketed from microseconds to milliseconds.
Application Scenario: Suitable for environments requiring high management simplicity, such as enterprise private clouds or test networks. Cisco's ACI architecture often uses a centralized model, combined with a leaf-spine topology, to ensure efficient operation of core gateways.
Distributed VXLAN Gateway
A distributed VXLAN gateway, also known as a distributed gateway or anycast gateway, offloads gateway functionality to each leaf switch or hypervisor VTEP. Each VTEP acts as a local gateway, handling L3 forwarding for the local subnet.
The principle is more flexible: each VTEP is configured with the same virtual IP (VIP) as the default gateway, using the Anycast mechanism. Cross-subnet packets sent by VMs are routed directly on the local VTEP, without having to go through a central point. EVPN is particularly useful here: through BGP EVPN, the VTEP learns the routes of remote hosts and uses MAC/IP binding to avoid ARP flooding.
For example, VM A (10.1.1.10) wants to access VM B (10.2.1.10). VM A's default gateway is the VIP of the local VTEP (10.1.1.1). The local VTEP routes to the destination subnet, encapsulates the VXLAN packet, and sends it directly to VM B's VTEP. This process minimizes the path and latency.
Outstanding Advantages:
○ High scalability Distributing gateway functionality to every node increases the network size, which is beneficial for larger networks. Large cloud providers like Google Cloud use a similar mechanism to support millions of VMs.
○ Superior performance East-west traffic is processed locally to avoid bottlenecks. Test data shows that throughput can increase by 30%-50% in distributed mode.
○ Fast fault recovery A single VTEP failure affects only the local host, leaving other nodes unaffected. Combined with EVPN's fast convergence, recovery time is in seconds.
○ Good use of resources Utilize the existing Leaf switch ASIC chip for hardware acceleration, with forwarding rates reaching Tbps level.
What are the disadvantages?
○ Complex configuration Each VTEP requires configuration of routing, EVPN, and other features, making initial deployment time-consuming. The operations team must be familiar with BGP and SDN.
○ High hardware requirements Distributed gateway: Not all switches support distributed gateways; Broadcom Trident or Tomahawk chips are required. Software implementations (such as OVS on KVM) do not perform as well as hardware.
○ Consistency Challenges Distributed means that state synchronization relies on EVPN. If the BGP session fluctuates, it may cause a routing black hole.
Application Scenario: Perfect for hyperscale data centers or public clouds. VMware NSX-T's distributed router is a typical example. Combined with Kubernetes, it seamlessly supports container networking.
Centralized VxLAN Gateway vs. Distributed VxLAN Gateway
Now onto the climax: which is better? The answer is "it depends", but we have to dig deep into the data and case studies to convince you.
From a performance perspective, distributed systems clearly outperform. In a typical data center benchmark (based on Spirent test equipment), the average latency of a centralized gateway was 150μs, while that of a distributed system was only 50μs. In terms of throughput, distributed systems can easily achieve line-rate forwarding because they leverage Spine-Leaf Equal Cost Multi-Path (ECMP) routing.
Scalability is another battleground. Centralized networks are suitable for networks with 100-500 nodes; beyond this scale, distributed networks gain the upper hand. Take Alibaba Cloud, for example. Their VPC (Virtual Private Cloud) uses distributed VXLAN gateways to support millions of users worldwide, with single-region latency under 1ms. A centralized approach would have collapsed long ago.
What about cost? A centralized solution offers lower initial investment, requiring only a few high-end gateways. A distributed solution requires all leaf nodes to support VXLAN offload, leading to higher hardware upgrade costs. However, in the long run, a distributed solution offers lower O&M costs, as automation tools like Ansible enable batch configuration.
Security and reliability: Centralized systems facilitate centralized protection but pose a high risk of single points of attack. Distributed systems are more resilient but require a robust control plane to prevent DDoS attacks.
A real-world case study: An e-commerce company used centralized VXLAN to build its site. During peak periods, the gateway CPU usage soared to 90%, leading to user complaints about latency. Switching to a distributed model resolved the issue, allowing the company to easily double its scale. Conversely, a small bank insisted on a centralized model because they prioritized compliance audits and found centralized management easier.
In general, if you're looking for extreme network performance and scale, a distributed approach is the way to go. If your budget is limited and your management team lacks experience, a centralized approach is more practical. In the future, with the rise of 5G and edge computing, distributed networks will become more popular, but centralized networks will still be valuable in specific scenarios, such as branch office interconnection.
Mylinking™ Network Packet Brokers support VxLAN, VLAN, GRE, MPLS Header Stripping
Supported the VxLAN, VLAN, GRE, MPLS header stripped in the original data packet and forwarded output.
Post time: Oct-09-2025
