Haproxy ssl termination for Jekyll - Part 2

So in our previous post Haproxy ssl termination for Jekyll we learned how to create a docker container capable of creating self-signed certificates or use previously created certificates to create our haproxy ssl termination to our backends, and always make sure our certificates were re-evaluated by haproxy on each change.

In this post, we will learn how to have letsencrypt create our valid ssl/tls certificates to use with blog domain, or any other web application that we have using a real domain.

Notes of caution and gotchas

Be aware that if you are trying with some Fully Qualified domain Names generated by AWS or Azure, some of them might be blacklisted by letsencrypt, such as AWS, and Azure ( except for the domains with regions.azure.com ). This had to be implemented by the folks at letsencrypt to avoid misuse of free ssl certificates.

Also make sure not to request certicates on an abusive manner, as you are limited to the number of certificate emissions weekly. Please refer to letsencrypt website for more information.

Creating a letsencrypt container

So since we will be making use of letsencrypt also inside a Docker container, for simplicity of usage.

Why not use a letsencrypt plugin such as the one used by nginx?

Well first of all, this is a post regarding haproxy, and there isn’t a plugin directly available to use with haproxy, and in order to avoid lua programming of haproxy, we will use a container to make our live easier.

Letsencrypt Dockerfile

FROM certbot/certbot:v0.20.0
RUN apk add --update --no-cache bash certbot tini \
	&& rm -rf /var/cache/apk/*
ENTRYPOINT [ "/bin/bash" ]
CMD [ "docker-entrypoint.sh" ]
COPY docker-entrypoint.sh docker-entrypoint.sh
COPY certbot.sh certbot.sh
RUN chmod +x docker-entrypoint.sh
RUN chmod +x certbot.sh

So once again, we are making use of a few shell scripts, and the official base image for the certbot ( The tool that is responsible for certificate signing requests and emission)

So what is the contents of our entrypoint.

Our docker container entrypoint script

docker-entrypoint.sh

#!/bin/bash

set -e

# Validate required environment variables.
[[ -z "$DOMAINS" ]] && MISSING="$MISSING DOMAINS"
[[ -z "$EMAIL" ]] && MISSING="$MISSING EMAIL"
[[ -z "$LOAD_BALANCER_SERVICE_NAME" ]] && MISSING="$MISSING LOAD_BALANCER_SERVICE_NAME"

if [[ -n "$MISSING" ]]; then
  echo "Missing required environment variables: $MISSING" >&2
  exit 1
fi

# Create a web server right away, otherwise HAProxy might not see that
# the server is listening and you won't be able to authenticate
echo "Creating http server port 80"
mkdir -p /opt/www
(pushd /opt/www && python -m SimpleHTTPServer 80) &

# Wait for the local server to be listening
while ! nc -z localhost 80;
do
  echo "SimpleHTTPServer not up yet, waiting 1 second";
  sleep 1;
done

echo "waiting for service \"${LOAD_BALANCER_SERVICE_NAME}\""

# Wait for HAproxy to start before updating certificates on startup.
while ! nc -z $LOAD_BALANCER_SERVICE_NAME 80;
do
  echo "Loadbalancer service \"${LOAD_BALANCER_SERVICE_NAME}\" is not up yet, waiting 5 second";
  sleep 5;
done

echo "Loadbalancer service \"${LOAD_BALANCER_SERVICE_NAME}\" is online, updating certificates..."
(./certbot.sh) &

echo "$RENEW_CRON $PWD/certbot.sh >> $PWD/certbot.log" > $PWD/crontab.txt
/usr/bin/crontab $PWD/crontab.txt
# start cron on foreground with verbosity set to 8
/usr/sbin/crond -f -l 8

Yes, that is a long and seemingly complex shell script.

What it is doing exactly?

Well since we will be using a standalone version of certbot, and letsencrypt emits certificates after being able to verify the challenges that should live under /.well-known endpoint we will make use of Python simple server to be able to respond to the challenge on this endpoint.

If we go back to our haproxy.cfg from our previous post

We now are able to understand our acl directive pointing to a backend named certificates

acl certificates_check url_beg /.well-known

This directive means that any incoming requests starting with our challenge endpoint will be tagged with certficates_check

And

use_backend certificates if certificates_check

Means that any tagged request with certificates_check will be redirected to our backend named certificates

And our backend content is:

backend certificates
mode http
balance leastconn
server letsencrypt letsencrypt:80 check init-addr last,libc,none resolvers docker_resolver

Meaning that our container whose name will be letsencrypt (in our docker-compose.yml that we will define in the) is listening on port 80 for any of request, and we are redirecting our challenges incoming request there.

This server is actually started in our docker-entrypoint.sh

python -m SimpleHTTPServer 80

Ok, moving on.

Our certbot script responsible for requesting certificate emission

certbot.sh

#!/bin/bash

set -e

# Certificates are separated by semi-colon (;). Domains on each certificate are
# separated by comma (,).
CERTS=(${DOMAINS//;/ })

# Create or renew certificates. Don't exit on error. It's likely that certbot
# will fail on first run, if HAproxy is not running.
for DOMAINS in "${CERTS[@]}"; do
	certbot certonly \
		--agree-tos \
		--domains "$DOMAINS" \
		--email "$EMAIL" \
		--expand \
		--noninteractive \
		--webroot \
		--webroot-path /opt/www \
		$OPTIONS || true
done

If you wish to know more about certbot tool, please visit certbot

In its essence, what is doing is that it will create .csr and privateKey.pem (under /opt/www ) and send a request to letsencrypt to solve the challenge and give us our fullchain.pem certificate.

tl;dr -> Its creating the certificate and associated files

What about all these variables that I keep seeing under these scripts, like domains.

Well these will all be passed by compose, and actually to make our life easier. With a .env file that will populate docker-compose.yml file.

Ok that’s it for our Letsencrypt container

Putting all things togther from Part1 and Part2

Let’s find out to put all this together.

First of all.

You should have the Part 2 under a folder name /letsencrypt and part 1 under a folder name /haproxy for the scripts and /haproxy/haproxy-cfg for the haproxy configuration.

Our final folder structure, for blog

haproxy

letsencrypt should be:

├── blog
├── docker-compose.yml
├── letsencrypt
├── haproxy
    |── haproxy-cfg
|_.env

And our final docker-compose.yml

version: "3"
services:
  blog:
    command: jekyll serve
    image: jekyll/jekyll:latest
    volumes:
      - $PWD/blog:/srv/jekyll
      - $PWD/vendor/bundle:/usr/local/bundle
    environment:
      - JEKYLL_ENV=production
    ports:
      - 4000
      - 35729
      - 3000
      -   80
  lb:
    build:
        context: ./haproxy
        dockerfile: Dockerfile
    ports:
        - 80:80
        - 443:443
    links:
        - letsencrypt
    environment:
        - DOMAINS=${DOMAINS}
    volumes:
        - "./haproxy/haproxy-cfg:/usr/local/etc/haproxy:ro"
        - certificates:/etc/letsencrypt
        - certificates:/var/lib/letsencrypt
  letsencrypt:
      build:
        context: ./letsencrypt
      volumes:
        - certificates:/etc/letsencrypt
        - certificates:/var/lib/letsencrypt
      ports:
        - 80
        - 443
      environment:
        - TERM=xterm
        - DOMAINS=${DOMAINS}
        - EMAIL=${EMAIL}
        - OPTIONS=${OPTIONS}
        - KEY_SIZE=${KEY_SIZE}
        - RENEW_CRON=${RENEW_CRON}
        - LOAD_BALANCER_SERVICE_NAME=lb
volumes:
   certificates:

In order to make our life easier, and instead of defining all this variables when doing docker-compose up -d.

We will create a .env file

EMAIL=mycontactemail@example.com
DOMAINS=example.com
OPTIONS=
KEY_SIZE=4096
#Cron job format - Default example 6 AM daily
RENEW_CRON=0 6 * * *

And that’s it.

All you need now is to docker-compose up -d on any server your domain is poiting too. And the certificate will be generated and valid by a simple docker-compose up.

What about certificate renewal. Well there is a cron job running for certificate renewal, and with the default value it will run daily.

And will renew when there are less than 4 weeks in your letsencrypt certificate.

You can also run docker-compose up locally without any issues, and use self signed certificate for testing purposes.

That’s it hope I could help you with haproxy ssl termination, even if not being used for jekyll, as it can be used for any other service.

Questions? Doubts?