Serving dns in docker with coredns and consul agent

Context

Coredns is a highly extensible dns server, commonly used in k8s and other orchestrators, and allows for very complex manipulation of dns requests. It is also very lightweight and can be used in docker containers.

Consul on the other hand is a common usage in this day and age of microservices and service discovery hell.

In this post we will create a docker image that will run both consul and coredns, and will be able to serve dns requests for consul services as if they were registered in the local dns server, similar to how you would do service discovery in a docker network bridge by simply using the service name.

Dockerfile

Ok, so let’s start by creating our dockerfile. We will use the official golang image as a base image to build our coredns binary as we will want to use a specific plugin for coredns named fanout, that will enable us to forward dns requests to consul and to google dns servers in the case we are not looking for a consul service, but rather a public domain.

In order to make our coredns more flexible, we will need to use consul-template to generate the coredns configuration file Corefile.

So let’s start by creating our dockerfile:

FROM golang:alpine AS COREDNS
RUN apk add git make
RUN git clone https://github.com/coredns/coredns.git && \
    echo fanout:github.com/networkservicemesh/fanout >> coredns/plugin.cfg && \
    cd coredns && \
    make
FROM hashicorp/consul-template:0.25.2-scratch as CONSUL_TEMPLATE
FROM consul as CONSUL
COPY --from=COREDNS /go/coredns/coredns /usr/bin/coredns
COPY --from=CONSUL_TEMPLATE /consul-template /bin/consul-template
COPY entrypoint.sh /entrypoint.sh
COPY Corefile.tpl /coredns/Corefile.tpl
RUN chmod +x /entrypoint.sh
ENTRYPOINT ["/entrypoint.sh"]

So what are we doing here?

• We are using the official golang image as a base image, and we are installing git and make in order to be able to clone the coredns repository and build the binary.

• We are copying the coredns binary from the coredns image onto our final consul image

• We are copying the consul-template binary from the official consul-template image onto our final consul image

• we will copy the entrypoint script that will run consul and coredns in the same container as background processes.

• we will copy the Corefile template that will be used by consul-template to generate the coredns configuration file.

Entrypoint

Now let’s create our entrypoint script that will run consul and coredns in the same container as background processes and will also template out the Corefile for our coredns configuration.

First let’s add the template

#!/bin/sh
# Set coredns in background
consul-template -once -template "/coredns/Corefile.tpl:/coredns/Corefile"

This makes sure consul-template will generate the coredns configuration file once and exit. Otherwise it will run in a loop and will keep generating the configuration file.

Now we will start coredns with the generated configuration file

echo "Setting up coredns, definition bellow"
cat /coredns/Corefile
sudo coredns -conf /coredns/Corefile -quiet &

We are just printing the coredns configuration file to the console for debugging purposes.

Next we need to add the consul agent configuration and start the consul agent as our main process

# Set consul agent as main process
consul agent -data-dir=/var/consul -node=client-1 -retry-join=172.17.0.1

Given we are running inside a docker container, we set the retry-join to the docker host ip address, which is 172.17.0.1 by default.

And that’s it, we are done with our entrypoint script.

Corefile template

Now let’s create our Corefile template that will be used by consul-template to generate the coredns configuration file.

.:53 {
  rewrite continue {
    name regex (^.[^.]+)(\.$) {1}.service.consul answer auto
  }
  fanout . :8600 8.8.8.8
}

Ok, so what does this template mean?

We are rewriting any incoming requests on port 53 for a domain that does not end with .service.consul to a domain that ends with .service.consul, and we are adding the .service.consul suffix to the request.

Then we are using the fanout plugin to forward the request to consul and to google dns servers in the case our request doesn’t fall under a registered consul service.

We are using the CONSUL_ADDRESS environment variable to set the consul address, to where we should forward the dns requests.

Building the image

So now all we need to do is to build our image.

docker build -t coredns-consul .

This process will take a while, as we are building coredns from source along with fanout plugin.

Running the image

Let’s run our image and check if everything is working as expected.

docker run --rm -p 53:53 -e CONSUL_ADDRESS=172.17.0.1 coredns-consul

Ok, we are saying to run our image in a container, and to expose port 53, and to set the CONSUL_ADDRESS environment variable to docker host ip address. However we don’t have a consul server running so we will get an error similar to the one below:

[WARN]  agent: Join cluster failed, will retry: cluster=LAN retry_interval=30s
  error=
  | 1 error occurred:
  | 	* Failed to join 172.17.0.1:8301: dial tcp 172.17.0.1:8301: connect: connection refused
  |

Testing the image

Now let’s run our image and test it.

But in order to do that, we need to run consul first, so let’s run consul in a docker container and a consul registed service.

For simplicity we will be using a docker-compose file to run both consul and our coredns-consul image.

version: '3.8'
services:
  consul:
    image: consul
    ports:
      - 8500:8500
      - 8600:8600/udp
      - 8301:8301
  command: agent -server -bootstrap-expect=1 -ui -client 0.0.0.0
  coredns:
    image: coredns-consul
    build: .
    ports:
      - 8053:53/udp
    environment:
      CONSUL_ADDRESS: 172.17.0.1
    depends_on:
      - consul

Ok, so now we are running consul in a docker container and we are exposing the consul ports 8500, 8600 and 8301, and we are running consul in server mode and we are bootstrapping the cluster with one server, while connecting to it from coreosn-consul service

So our logs now should look like this:

coredns-consul-dockerfile-coredns-1  | ==> Starting Consul agent...
coredns-consul-dockerfile-coredns-1  |               Version: '1.14.3'
coredns-consul-dockerfile-coredns-1  |            Build Date: '2022-12-13 17:13:55 +0000 UTC'
coredns-consul-dockerfile-coredns-1  |               Node ID: '11549e9a-9d8d-9298-ef22-b97f5c38b066'
coredns-consul-dockerfile-coredns-1  |             Node name: 'client-1'
coredns-consul-dockerfile-coredns-1  |            Datacenter: 'dc1' (Segment: '')
coredns-consul-dockerfile-coredns-1  |                Server: false (Bootstrap: false)
coredns-consul-dockerfile-coredns-1  |           Client Addr: [127.0.0.1] (HTTP: 8500, HTTPS: -1, gRPC: -1, gRPC-TLS: -1, DNS: 8600)
coredns-consul-dockerfile-coredns-1  |          Cluster Addr: 172.22.0.3 (LAN: 8301, WAN: 8302)
coredns-consul-dockerfile-coredns-1  |     Gossip Encryption: false
coredns-consul-dockerfile-coredns-1  |      Auto-Encrypt-TLS: false
coredns-consul-dockerfile-coredns-1  |             HTTPS TLS: Verify Incoming: false, Verify Outgoing: false, Min Version: TLSv1_2
coredns-consul-dockerfile-coredns-1  |              gRPC TLS: Verify Incoming: false, Min Version: TLSv1_2
coredns-consul-dockerfile-coredns-1  |      Internal RPC TLS: Verify Incoming: false, Verify Outgoing: false (Verify Hostname: false), Min Version: TLSv1_2
coredns-consul-dockerfile-coredns-1  |
coredns-consul-dockerfile-coredns-1  | ==> Log data will now stream in as it occurs:
coredns-consul-dockerfile-coredns-1  |
coredns-consul-dockerfile-coredns-1  |  [INFO]  agent.client.serf.lan: serf: EventMemberJoin: client-1 172.22.0.3
coredns-consul-dockerfile-coredns-1  |  [INFO]  agent.router: Initializing LAN area manager
coredns-consul-dockerfile-coredns-1  |  [INFO]  agent: Started DNS server: address=127.0.0.1:8600 network=udp
coredns-consul-dockerfile-coredns-1  |  [INFO]  agent: Started DNS server: address=127.0.0.1:8600 network=tcp
coredns-consul-dockerfile-coredns-1  |  [INFO]  agent: Starting server: address=127.0.0.1:8500 network=tcp protocol=http

If we now inspect the consul ui, we should see that our consul server is up and running and has a service registered named consul, as this is registed to consul by default.

Now let’s try to run a dig command to see if we can resolve the consul service. Beware that we are using the port 8053, as we are exposing the coredns-consul service on port 8053, but we must ensure we are using the udp protocol as dns is a udp protocol, otherwise docker will expose a tcp port, and our dig will be unable to resolve the dns query. Also ensure that the consul service is using the udp protocol.

dig +short @127.0.0.1 -p 8053 consul

So now we should see the consul service ip address

$dig +short @127.0.0.1 -p 8053 consul
172.27.0.2

And that’s the internal address for the consul service inside the bridge network.

If we do the same query to consul itself, we should see the same ip address.

$dig +short @127.0.0.1 -p 8053 consul.service.consul
172.27.0.2

But now we can’t use the short name consul, as we are using the consul service as a dns server, so we must use the full name consul.service.consul.

If we had other services registed we would be able to core them using the short name, as long as we are using the consul service as a dns server.

Conclusion

And that’s it, we have now a coredns image alongside a consul that is able to resolve consul services via a short name, and fallback to the google dns server if we are not using a known address to consul.

As usual, the code is available on github