Making kubernetes kube-dns publicly reachable

Posted on 2021-06-13 by ungleich

Introduction

If you have seen our article about running kubernetes Ingress-less, you are aware that we are pushing IPv6 only kubernetes clusters at ungleich.

Today, we are looking at making the "internal" kube-dns service world reachable using IPv6 and global DNS servers.

The kubernetes DNS service

If you have a look at your typical k8s cluster, you will notice that you usually have two coredns pods running:

% kubectl -n kube-system get pods -l k8s-app=kube-dns
NAME                       READY   STATUS    RESTARTS   AGE
coredns-558bd4d5db-gz5c7   1/1     Running   0          6d
coredns-558bd4d5db-hrzhz   1/1     Running   0          6d

These pods are usually served by the kube-dns service:

% kubectl -n kube-system get svc -l k8s-app=kube-dns
NAME       TYPE        CLUSTER-IP           EXTERNAL-IP   PORT(S)                  AGE
kube-dns   ClusterIP   2a0a:e5c0:13:e2::a   <none>        53/UDP,53/TCP,9153/TCP   6d1h

As you can see, the kube-dns service is running on a publicly reachable IPv6 address.

IPv6 only DNS

IPv6 only DNS servers have one drawback: they cannot be reached via DNS recursions, if the resolver is IPv4 only.

At ungleich we run a variety of services to make IPv6 only services usable in the real world. In case of DNS, we are using DNS forwarders. They are acting similar to HTTP proxies, but for DNS.

So in our main DNS servers, dns1.ungleich.ch, dns2.ungleich.ch and dns3.ungleich.ch we have added the following configuration:

zone "k8s.place7.ungleich.ch"  {
   type forward;
   forward only;
   forwarders { 2a0a:e5c0:13:e2::a; };
};

This tells the DNS servers to forward DNS queries that come in for k8s.place7.ungleich.ch to 2a0a:e5c0:13:e2::a.

Additionally we have added DNS delegation in the place7.ungleich.ch zone:

k8s NS dns1.ungleich.ch.
k8s NS dns2.ungleich.ch.
k8s NS dns3.ungleich.ch.

Using the kubernetes DNS service in the wild

With this configuration, we can now access IPv6 only kubernetes services directly from the Internet. Let's first discover the kube-dns service itself:

% dig kube-dns.kube-system.svc.k8s.place7.ungleich.ch. aaaa

; <<>> DiG 9.16.16 <<>> kube-dns.kube-system.svc.k8s.place7.ungleich.ch. aaaa
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 23274
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
; COOKIE: f61925944f5218c9ac21e43960c64f254792e60f2b10f3f5 (good)
;; QUESTION SECTION:
;kube-dns.kube-system.svc.k8s.place7.ungleich.ch. IN AAAA

;; ANSWER SECTION:
kube-dns.kube-system.svc.k8s.place7.ungleich.ch. 27 IN AAAA 2a0a:e5c0:13:e2::a

;; AUTHORITY SECTION:
k8s.place7.ungleich.ch. 13  IN  NS  kube-dns.kube-system.svc.k8s.place7.ungleich.ch.

As you can see, the kube-dns service in the kube-system namespace resolves to 2a0a:e5c0:13:e2::a, which is exactly what we have configured.

At the moment, there is also an etherpad test service named "ungleich-etherpad" running:

% kubectl get svc -l app=ungleichetherpad
NAME                TYPE        CLUSTER-IP              EXTERNAL-IP   PORT(S)    AGE
ungleich-etherpad   ClusterIP   2a0a:e5c0:13:e2::b7db   <none>        9001/TCP   3d19h

Let's first verify that it resolves:

% dig +short ungleich-etherpad.default.svc.k8s.place7.ungleich.ch aaaa
2a0a:e5c0:13:e2::b7db

And if that works, well, then we should also be able to access the service itself!

% curl -I http://ungleich-etherpad.default.svc.k8s.place7.ungleich.ch:9001/
HTTP/1.1 200 OK
X-Powered-By: Express
X-UA-Compatible: IE=Edge,chrome=1
Referrer-Policy: same-origin
Content-Type: text/html; charset=utf-8
Content-Length: 6039
ETag: W/"1797-Dq3+mr7XP0PQshikMNRpm5RSkGA"
Set-Cookie: express_sid=s%3AZGKdDe3FN1v5UPcS-7rsZW7CeloPrQ7p.VaL1V0M4780TBm8bT9hPVQMWPX5Lcte%2BzotO9Lsejlk; Path=/; HttpOnly; SameSite=Lax
Date: Sun, 13 Jun 2021 18:36:23 GMT
Connection: keep-alive
Keep-Alive: timeout=5

(attention, this is a test service and might not be running when you read this article at a later time)

IPv6 vs. IPv4

Could we have achived the same with IPv4? The answere here is "maybe": If the kubernetes service is reachable from globally reachable nameservers via IPv4, then the answer is yes. This could be done via public IPv4 addresses in the kubernetes cluster, via tunnels, VPNs, etc.

However, generally speaking, the DNS service of a kubernetes cluster running on RFC1918 IP addresses, is probably not reachable from globally reachable DNS servers by default.

For IPv6 the case is a bit different: we are using globally reachable IPv6 addresses in our k8s clusters, so they can potentially be reachable without the need of any tunnel or whatsoever. Firewalling and network policies can obviously prevent access, but if the IP addresses are properly routed, they will be accessible from the public Internet.

And this makes things much easier for DNS servers, which are also having IPv6 connectivity.

The following pictures shows the practical difference between the two approaches:

Does this make sense?

That clearly depends on your use-case. If you want your service DNS records to be publicly accessible, then the clear answer is yes.

If your cluster services are intended to be internal only (see previous blog post, then exposing the DNS service to the world might not be the best option.

Note on security

CoreDNS inside kubernetes is by default configured to allow resolving for any client that can reach it. Thus if you make your kube-dns service world reachable, you also turn it into an open resolver.

At the time of writing this blog article, the following coredns configuration does NOT correctly block requests:

  Corefile: |
    .:53 {
        acl k8s.place7.ungleich.ch {
             allow net ::/0
        }
        acl . {
              allow net 2a0a:e5c0:13::/48
              block
        }
        forward . /etc/resolv.conf {
           max_concurrent 1000
        }
...

Until this is solved, we recommend to place a firewall before your public kube-dns service to only allow requests from the forwarding DNS servers.

More of this

We are discussing kubernetes and IPv6 related topics in the #hacking:ungleich.ch Matrix channel (you can signup here if you don't have an account) and will post more about our k8s journey in this blog. Stay tuned!