Introduction
On our journey to build and define IPv6 only kubernetes clusters we came accross some principles that seem awkward in the IPv6 only world. Let us today have a look at the LoadBalancer and Ingress concepts.
Ingress
Let's have a look at the Ingress definition definiton from the kubernetes website:
Ingress exposes HTTP and HTTPS routes from outside the cluster to
services within the cluster. Traffic routing is controlled by rules
defined on the Ingress resource.
So the ingress basically routes from outside to inside. But, in the IPv6 world, services are already publicly reachable. It just depends on your network policy.
Update 2021-06-13: Ingress vs. Service
As some people pointed out (thanks a lot!), a public service is not the same as an Ingress. Ingress has also the possibility to route based on layer 7 information like the path, domain name, etc.
However, if all of the traffic from an Ingress points to a single IPv6 HTTP/HTTPS Service, effectively the IPv6 service will do the same, with one hop less.
Services
Let's have a look at how services in IPv6 only clusters look like:
% kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
etherpad ClusterIP 2a0a:e5c0:13:e2::a94b <none> 9001/TCP 19h
nginx-service ClusterIP 2a0a:e5c0:13:e2::3607 <none> 80/TCP 43h
postgres ClusterIP 2a0a:e5c0:13:e2::c9e0 <none> 5432/TCP 19h
...
All these services are world reachable, depending on your network policy.
ServiceTypes
While we are at looking at the k8s primitives, let's have a closer look at the Service, specifically at 3 of the ServiceTypes supported by k8s, including it's definition:
ClusterIP
The k8s website says
Exposes the Service on a cluster-internal IP. Choosing this value
makes the Service only reachable from within the cluster. This is the
default ServiceType.
So in the context of IPv6, this sounds wrong. There is nothing that makes an global IPv6 address be "internal", besides possible network policies. The concept is probably coming from the strict difference of RFC1918 space usually used in k8s clusters and not public IPv4.
This difference does not make a lot of sense in the IPv6 world though. Seeing services as public by default, makes much more sense. And simplifies your clusters a lot.
NodePort
Let's first have a look at the definition again:
Exposes the Service on each Node's IP at a static port (the
NodePort). A ClusterIP Service, to which the NodePort Service routes,
is automatically created. You'll be able to contact the NodePort
Service, from outside the cluster, by requesting <NodeIP>:<NodePort>.
Conceptually this can be similarily utilised in the IPv6 only world like it does in the IPv4 world. However given that there are enough addresses available with IPv6, this might not be such an interesting ServiceType anymore.
LoadBalancer
Before we have a look at this type, let's take some steps back first to ...
... Load Balancing
There are a variety of possibilities to do load balancing. From simple round robin, to ECMP based load balancing, to application aware, potentially weighted load balancing.
So for load balancing, there is usually more than one solution and there is likely not one size fits all.
So with this said, let.s have a look at the ServiceType LoadBalancer definition:
Exposes the Service externally using a cloud provider's load
balancer. NodePort and ClusterIP Services, to which the external load
balancer routes, are automatically created.
So whatever the cloud provider offers, can be used, and that is a good thing. However, let's have a look at how you get load balancing for free in IPv6 only clusters:
Load Balancing in IPv6 only clusters
So what is the most easy way of reliable load balancing in network? ECMP (equal cost multi path) comes to the mind right away. Given that kubernetes nodes can BGP peer with the network (upstream or the switches), this basically gives load balancing to the world for free:
[ The Internet ]
|
[ k8s-node-1 ]-----------[ network ]-----------[ k8s-node-n]
[ ECMP ]
|
[ k8s-node-2]
In the real world on a bird based BGP upstream router this looks as follows:
[18:13:02] red.place7:~# birdc show route
BIRD 2.0.7 ready.
Table master6:
...
2a0a:e5c0:13:e2::/108 unicast [place7-server1 2021-06-07] * (100) [AS65534i]
via 2a0a:e5c0:13:0:225:b3ff:fe20:3554 on eth0
unicast [place7-server4 2021-06-08] (100) [AS65534i]
via 2a0a:e5c0:13:0:225:b3ff:fe20:3564 on eth0
unicast [place7-server2 2021-06-07] (100) [AS65534i]
via 2a0a:e5c0:13:0:225:b3ff:fe20:38cc on eth0
unicast [place7-server3 2021-06-07] (100) [AS65534i]
via 2a0a:e5c0:13:0:224:81ff:fee0:db7a on eth0
...
Which results into the following kernel route:
2a0a:e5c0:13:e2::/108 proto bird metric 32
nexthop via 2a0a:e5c0:13:0:224:81ff:fee0:db7a dev eth0 weight 1
nexthop via 2a0a:e5c0:13:0:225:b3ff:fe20:3554 dev eth0 weight 1
nexthop via 2a0a:e5c0:13:0:225:b3ff:fe20:3564 dev eth0 weight 1
nexthop via 2a0a:e5c0:13:0:225:b3ff:fe20:38cc dev eth0 weight 1 pref medium
TL;DR
We know, a TL;DR at the end is not the right thing to do, but hey, we are at ungleich, aren't we?
In a nutshell, with IPv6 the concept of Ingress, Service and the LoadBalancer ServiceType types need to be revised, as IPv6 allows direct access without having to jump through hoops.
If you are interesting in continuing the discussion, we are there for you in the #hacking:ungleich.ch Matrix channel you can signup here if you don't have an account.
Or if you are interested in an IPv6 only kubernetes cluster, drop a mail to support-at-ungleich.ch.