Recently I was part of the lab proof of concept as a practical learning experience to evaluate the East West switching in a Kubernetes infrastructure where we had tested 3 conditions.
- Internode communication with VXLAN
- Internode communication with IPIP
- Internode communication using BGP.
| Mode | Mechanism | Encapsulation | Best For | Overhead (bytes) |
| IPIP | IP packets inside another IP packet (Protocol 4) | Yes (lightweight) | Private/baremetal clusters | ~20 bytes |
| VXLAN | IP packets inside UDP packets (UDP 4789) | Yes (heavy) | Public cloud (AWS, GCP, Azure) or complex networks | ~50 bytes |
| BGP Native Routing | Advertise pod CIDRs over BGP between nodes | No | High-performance clusters, advanced networking (Telecom networks) | 0 bytes |
In this blog we will focus on BGP implementation in large scale telecom networks and talk about our lab implementation. BGP Peering Between Kubernetes Nodes is Default in Calico. In a Calico cluster without Route Reflectors (RRs),Every node establishes a BGP session with every other node. The formula for calculating peering session is
Total Sessions=n×(n−1)/2
Why Use a Route Reflector (RR)
By default, Calico establishes a full-mesh BGP — meaning each node peers with every other node. The number of peering sessions grows rapidly:
To scale better:
-
Introduce a Route Reflector (RR).
-
Nodes peer only with the RR, not with each other.
-
RR redistributes routes across the cluster.
This dramatically reduces BGP session complexity.
Calico is a popular Container Network Interface (CNI) plugin for Kubernetes. OpenShift uses OVN-Kubernetes as default CNI , but we install Calico manually for telecom grade large clusters where tight performance is requirements.
Calico provides network connectivity for pods and also supports network security policies.
- Unlike some other CNIs, Calico uses pure Layer 3 (routing-based) networking — no overlay like VXLAN, unless you explicitly enable it. It can integrate directly with the underlay network making it very high performance and simple. A Route Reflector (RR) is a special node that acts like a BGP hub.
- Instead of full mesh peering, nodes only peer with RR.
- RR receives routes from all nodes, and reflects (redistributes) them to other nodes.
In Kubernetes clusters using Calico, each pod is assigned a unique /32 IP address, and all nodes keep track of the pod IPs running across the cluster. When a pod on Node A needs to communicate with a pod on Node B, the Calico agent on Node A checks the destination IP and uses the routing table to figure out the next hop — which is simply Node B’s host IP. The traffic is sent directly over the network without any VXLAN or IP-in-IP encapsulation by default, keeping things simple and efficient. When the packet reaches Node B, its Calico agent delivers it straight to the correct pod. To make this all seamless, Calico uses BGP (Border Gateway Protocol) between nodes: each node advertises the pod IPs it manages, and nodes share this information with each other using a BGP daemon like BIRD or GoBGP. As a result, routing tables across the cluster are dynamically updated, so every node knows exactly where each pod lives. In bare-metal environments, this direct BGP peering makes pod-to-pod communication super-fast, with minimal latency and no need for an overlay network — unless your setup specifically requires it. This is ideally best for telecom networks.
The below diagram depicts the connection between worker nodes and the Cisco router .
Our Cisco IOS router configuration is simple and straight forward.
! Configure router BGP process router bgp 64512 bgp log-neighbor-changes ! Make this router a Route Reflector bgp cluster-id 1 ! Calico Node A Peering neighbor 192.168.1.10 remote-as 64512 neighbor 192.168.1.10 route-reflector-client ! Calico Node B Peering neighbor 192.168.1.11 remote-as 64512 neighbor 192.168.1.11 route-reflector-client ! Calico Node C Peering neighbor 192.168.1.12 remote-as 64512 neighbor 192.168.1.12 route-reflector-client ! Optional tuning timers bgp 10 30 address-family ipv4 neighbor 192.168.1.10 activate neighbor 192.168.1.11 activate neighbor 192.168.1.12 activate exit-address-family
| Command | Purpose |
|---|---|
| router bgp 64512 | Enable BGP with AS 64512 |
| bgp cluster-id 1 | Set Cluster ID (RR requirement) |
| neighbor X.X.X.X remote-as 64512 | Define Calico node peers (same AS) |
| neighbor X.X.X.X route-reflector-client | Mark them as RR clients |
| address-family ipv4 + activate | Activate BGP IPv4 address family |
Manifest file calico-external-rr.yaml
# Disable node-to-node mesh (Important: else nodes will also peer with each other)
apiVersion: projectcalico.org/v3 kind: BGPConfiguration metadata: name: default spec: asNumber: 64512 nodeToNodeMeshEnabled: false
—
# Peering all Calico nodes to external Route Reflector (Cisco Router)
apiVersion: projectcalico.org/v3 kind: BGPPeer metadata: name: peer-to-external-rr spec: peerIP: 192.168.1.1 # IP address of Cisco Router RR asNumber: 64512 # Same AS Number as the Router and Nodes nodeSelector: all() # Apply to all Kubernetes nodes
kubectl apply -f calico-external-rr.yaml
All nodes have single BGP session to external router. Native Pod IP routing — no overlay. Highly scalable to thousands of nodes.
We have to use calicoctl to Check Calico BGP Peering. calicoctl is the command-line tool to interact with Calico resources. It can show BGP peering status, IP Pool configs, Node info, etc. It talks directly to the Calico datastore (Kubernetes API server or etcd).
Here are the steps to install calicoctl
curl -O -L https://github.com/projectcalico/calicoctl/releases/download/v3.27.0/calicoctl chmod +x calicoctl sudo mv calicoctl /usr/local/bin/
Set Up Access. Create a config file /etc/calico/calicoctl.cfg
apiVersion: projectcalico.org/v3 kind: CalicoAPIConfig metadata: spec: datastoreType: "kubernetes" kubeconfig: "/etc/kubernetes/admin.conf" export CALICOCTL_CFG_FILE=/etc/calico/calicoctl.cfg calicoctl node status +--------------+----------------+-------------+ | PEER ADDRESS | PEER TYPE | STATE | +--------------+----------------+-------------+ | 192.168.1.1 | Global peer | Established | +--------------+----------------+-------------+
Conclusion
Using Calico BGP native routing for Kubernetes clusters, especially when paired with an external Route Reflector like a Cisco router, provides a highly scalable, overlay-free, and telecom-grade performance solution for pod networking.
This setup is particularly ideal for large bare-metal Kubernetes/OpenShift deployments in the telecom industry, where every microsecond of network efficiency counts.