Home Haproxy ssl termination for Jekyll
Post
Cancel

Haproxy ssl termination for Jekyll

Posted Apr 7, 2018
By André Ilhicas dos Santos
8 min read

For this blog’s first post I described a simple start for people getting into Jekyll blogging by using docker and how to start a blog using a Jekyll docker based image with a ready to use theme.

In this post I’ll show how to create an haproxy ssl termination, and with a valid ssl certificate by using letsencrypt, or you can change it for your own already created certificate.

Dockerfile for haproxy

So first of all we need an haproxy based docker image.

For this blog I’m currently using an alpine linux based image for haproxy.

Alpine images are usually lightweight and are as straightforward as any other linux based image in Docker, with a very reduced footprint and image size, as base is around ~ 3mb.

Since we will be using letsencrypt for the certificate generation below is the defined our Haproxy Dockerfile

1
2
3
4
5
6
7
8
9
10
11
12
13
14

FROM haproxy:1.7-alpine
RUN apk add --update bash inotify-tools openssl \
    && rm -rf /var/cache/apk/*
COPY docker-entrypoint.sh docker-entrypoint.sh
COPY install-certs.sh install-certs.sh
COPY watch-certs.sh watch-certs.sh
COPY reload.sh reload.sh
ENV LIVE_CERT_FOLDER="/etc/letsencrypt/live"
ENTRYPOINT ["/docker-entrypoint.sh"]
CMD ["haproxy", "-f", "/usr/local/etc/haproxy/haproxy.cfg"]
RUN chmod +x docker-entrypoint.sh \
    && chmod +x watch-certs.sh \
    && chmod +x install-certs.sh \
    && chmod +x reload.sh

Wow what is going on here?

So for those not very familiar with docker, we are using haproxy official alpine image in :

1

FROM haproxy:1.7-alpine

The next we will need is to install a few libraries, and the package manager for alpine linux is apk which we are using and removing cache in the end to reduce our final image size.

1
2

RUN apk add --update bash inotify-tools openssl \
    && rm -rf /var/cache/apk/*

Besides the obvious openssl, and bash (because it isn’t present in base alpine image for reduced size)

We will make use of inotify-tools, since haproxy has no way to re-evaluate its ssl certificate on any change to the certificate file used, and unless reloaded, it will keep using the same certificate file that was passed on first run, even if the file gets deleted.

Why?

For performance reasons, the certificate is load onto memory and used on each sequential request, and not evaluated on each.

So for such reasons, its imperative we reload haproxy each time we actually change our certificate, and let’s encrypt certificates only have 3 months validity, so we will need to “regularly” renew our certificate.

So what exactly is happening on the COPY directives? What do those files contain?

1
2
3
4

COPY docker-entrypoint.sh docker-entrypoint.sh
COPY install-certs.sh install-certs.sh
COPY watch-certs.sh watch-certs.sh
COPY reload.sh reload.sh

Ok so first lets create a docker-entrypoint.sh to live in the same folder as our Dockerfile above.

Our docker container entrypoint

With the following contents:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

#!/bin/bash
set -e
bash install-certs.sh
bash watch-certs.sh &
LETSENCRYPT_CERT="$(cat /certs/cert.pem)"
export DEFAULT_SSL_CERT="${DEFAULT_SSL_CERT:-$LETSENCRYPT_CERT}"
# first arg is `-f` or `--some-option`
if [ "${1#-}" != "$1" ]; then
	set -- haproxy "$@"
fi

if [ "$1" = 'haproxy' ]; then
	shift # "haproxy"
	# if the user wants "haproxy", let's add a couple useful flags
	#   haproxy-systemd-wrapper -- "master-worker mode" (similar to the new "-W" flag; allows for reload via "SIGUSR2")
	#   -db -- disables background mode
	set -- "$(which haproxy-systemd-wrapper)" -p /run/haproxy.pid -db "$@"
fi
exec "$@"

So first we will invoke our also passed scripts (read more below for each)

The rest of the script is just to allow us to run our docker container with a different haproxy entrypoint argument.

SSL certificates - Self-Signed | Existing

So our next bash script is the install-certs.sh

Whose contents are:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

#!/bin/bash
set -e

mkdir -p /certs
mkdir -p "$LIVE_CERT_FOLDER"

# Create a self signed default certificate, so HAproxy can start before we have
# any real certificates.
if [[ ! -f /certs/temp-cert.pem ]]; then
	apk --update add openssl ca-certificates
    openssl genrsa -out cert.key 4096
    openssl req -new -key cert.key -out cert.csr -nodes -subj "/C=US/ST=CA/L=SF/O=ilhicas/OU=ilhicas /CN=www.ilhicas.com/EMAIL=info@ilhicas.com"
    openssl x509 -req -days 365 -in cert.csr -signkey cert.key -out cert.crt
    cat cert.key cert.crt > /certs/cert.pem
fi


# Install combined certificates for HAproxy.
if [[ -n "$(ls -A $LIVE_CERT_FOLDER)" ]]; then
	COUNT=0
	for DIR in "$LIVE_CERT_FOLDER"/*; do
		cat "$DIR/privkey.pem" "$DIR/fullchain.pem" > /certs/cert.pem
		(( COUNT += 1 ))
	done
fi

So on this script we are actually validating if we already the certificate file present otherwise, If we don’t we will first be creating a self-signed certificate to use so that haproxy won’t complain on its entrypoint and finds out that the file we are passing its non-existant.

We will also be using a combined .pem file for each, if it exists, or not.

By using a form of :

1

cat certificate.key certificateFILE.crt > certificate.pem

Since haproxy requires the ssl certificate termination file to contain both the key and the certificate chain.

Watch certificates for change and reload

Now we will need the script that will make use of the inotify package to verify if the contents of our certificate has changed so that it triggers the reload of haproxy

Contents of watch-certs.sh

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41

    #!/bin/bash

# See: https://github.com/tartley/rerun2

set -e

mkdir -p "$LIVE_CERT_FOLDER"

# Abort, if already running.
if [[ -n "$(ps | grep inotifywait | grep -v grep)" ]]; then
	echo "Already watching directory: $LIVE_CERT_FOLDER" >&2
	exit y
fi

# Debounce for 60 seconds, which we assume is enough time to create or renew
# all certifies and avoid multiple restarts.
IGNORE_SECS=60
IGNORE_UNTIL="$(date +%s)"

# Watch the live certificates directory. When changes are detected, install
# combined certificates and reload HAproxy.
echo "Watching directory: $LIVE_CERT_FOLDER"
inotifywait \
	--event create \
	--event delete \
	--event modify \
	--event move \
	--format "%e %w%f" \
	--monitor \
	--quiet \
	--recursive \
	"$LIVE_CERT_FOLDER" |
while read CHANGED
do
	echo "$CHANGED"
	NOW="$(date +%s)"
	if (( NOW > IGNORE_UNTIL )); then
		(( IGNORE_UNTIL = NOW + IGNORE_SECS ))
		(sleep $IGNORE_SECS; ./install-certs.sh; ./reload.sh) &
	fi
done

And finally the script to actually reload haproxy

Contents of reload.sh

1
2
3

#!/bin/sh
#Sends a Sighup to haproxy-system-wrapper in order to reload
kill -USR2 1

This is actually a rather simple script.

How is it working?

Well haproxy reloads itself each time it receives as sighup and that is kill -USR2 $PID part, where for simplicity in our case since haproxy is actually our initial process ( from the docker-entrypoint), what is happening is that is actually our process with ID=1, our entry process. Killing process id=1 would actually resulting in stopping our Docker container, but we are just sending a sighup in this case.

Setting up our Haproxy.cfg

So how does our haproxy.cfg looks like so that we can create set our jekyll blog behind it?

Well its actually pretty simple:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44

global
    maxconn 4096
    maxpipes 1024
    log 127.0.0.1 local0
    log 127.0.0.1 local1 notice

defaults
    log global
    mode    http
    option httplog
    option dontlognull
    timeout connect 1000000
    timeout client 3600000
    timeout server 3600000

listen stats
  bind 0.0.0.0:70
  stats enable
  stats uri /

resolvers docker_resolver
    nameserver dns 127.0.0.11:53


frontend http
    bind *:80
    mode http
    redirect scheme https code 307 if !{ ssl_fc }

frontend https
    bind *:443 ssl crt /certs/cert.pem
    acl certificates_check url_beg /.well-known
    use_backend certificates if certificates_check
    default_backend blog

backend certificates
    mode http
    balance leastconn
    server letsencrypt letsencrypt:80 check init-addr last,libc,none resolvers docker_resolver

backend blog
    mode http
    balance leastconn
    server blog blog:4000 check init-addr last,libc,none resolvers docker_resolver

Redirect to https always

For now dont worry about the certificates backend and the acl directive leading to it, we will appproach let’s encrypt certificate generation on our post part 2.

For now focus on the Front http that is redirect with 307 whenever it receives a call on port 80 without ssl. This guarantees that all requests are https.

Why 307 and not 302?

Because both nginx and haproxy actually convert POST to GET while making the redirect with permanent redirect (302) , so using 307 is to keep the HTTP verb of the request intact.

Backend for blog

So since we are using docker, and we will actually use this inside a docker-compose.yml file, all services in the same network created by docker-compose, will actually know all other services by name.

Since on our first post we named it blog, so we shall name it the same way in haproxy.cfg

Haproxy resolvers: docker

We also use the docker-resolver so that even if one of our other container goes down, haproxy won’t crash, and can start up on his own without any of the other services running.

This is actually very usefull as we will be restarting our blog container several times, and can’t have our haproxy container falling on us, because it can’t find a service.

Ok so this was a long post.

If you wish, rest now, or continue to Part 2 - Haproxy SSL termination Jekyll

load balancers, certificates
haproxy ssl jekyll docker letsencrypt
This post is licensed under CC BY 4.0 by the author.
Contents