Kubernetes - Networking
Kubernetes networking in a nutshell:
- Every Kubernetes pod receives an IP address.
kube-dnsresolves Kubernetes service DNS names to IP addresses.
iptablesrules in order to do random load balancing.
1. you make a request to my-svc.my-namespace.svc.cluster.local 2. it resolves to 10.23.1.2 3. iptables on your local host redirect it to 10.1.0.3 or 10.2.3.5 at random
Kubernetes requires that each container has an IP address. So contrary to normal "one IP, multiple ports", Kubernetes uses "multiple IP addresses, one port" approach.
AWS computer has IP 22.214.171.124 You want a container that has IP 10.4.4.4 VPC Route Table rules that 10.4.4.* packets go to 126.96.36.199 (this works up to 50 instances as the rule limit is 50)
Prefer to automate the network setup. There are a couple of software that help to automate container networking, but both are limited to communication between instances in the same availability zone.
host-gw(just set route tables)
ip-in-ipencapsulation or regular mode (just set route tables)
Networking software often relies very heavily on the Linux kernel. It's worth your while to get more familiar with low level networking on Linux so you can efficiently debug and fix problems, especially
Use MAC addresses if source is in the same network as the destination. LANs and AWS availability zones utilize MAC addresses, not IP addresses. You can "ignore" the actual IP and just use the MAC address to send packages.
ip route add 10.4.4.0/24 via 172.23.1.1 dev eth0 # now all traffic to 10.4.4.* is sent to MAC address of 172.23.1.1
Use destination host instance IP, if source and destination separate AZs. Here you have to encapsulate the network package inside another network packet, e.g.
vxlan: encapsulates the packet, including MAC address, inside a UDP packet ip-in-ip: adds an extra IP header on the packet, so it won't keep the MAC # setup a new network interface ip tunnel add mytun mode ipip remote 188.8.131.52 local 10.4.4.4 ttl 255 ifconfig mytun 10.42.1.1 # route packets to the magic interface route add -net 10.42.2.0/24 dev mytun route list
You usually configure the Linux route tables using one of the following ways:
- a program reads the routes from Kubernetes cluster
etcdand adds them.
- a program reads the routes from BGP protocol gossip between nodes.
Major changes to production networking infrastructure is dangerous. Configure it the right way from the start. If you ever need to do major changes, be sure to have a couple people who know the infrastructure in-and-out.