This tutorial is part of the effort to deploy a complex business application on a docker swarm and is focused on deploying a swarm accessible secured private registry using letsencrypt certificates.
The first step is to create the swarm, see Create and manage a Docker Swarm with Docker CE and register it to Docker Cloud
Another step is to set up a way to monitor the swarm nodes and the deployed containers. My prefered choise is WeaveScope as described in Monitor #Docker #swarm nodes with #WeaveScope
Deploying an application environment to a docker swarm requires that the docker images used to construct that application environment are visible to all the swarm nodes. If we use standard images from main docker repository (docker.io) or from a private repository hosted in the Docker Cloud all the swarm nodes will be able to pull an image when a service that needs it migrates to the node.
In case we do not want to share private docker images (due to legal restrictions, licensing, company trademark etc.) to outside hosted repositories we can setup a self hosted repository.
The simplest solution is to use the docker repository “registry:2” image. We can also use some more complex solution as Nexus Repository, but I prefer to have a dedicated registry for the swarm.
Because of swarm mode it is very inconvenient to use a private unsecure registry as the swarm registry. We do not want to configure unsecure registry for each new node added to the swarm. This would make adding a new swarm node a long an tedious manual step.
The easiest way to secure a registry is to use letsencrypt certificates for the registry. I already use them for the blog and several other web services Use #letsencrypt.org certificates with an apache server behind a secure reverse proxy.
In fact I am going to use for the registry the certificates of my store.voina.org domain.
STEP 1: Install letsencrypt support (fedora linux)
dnf install python-certbot-apache
dnf install certbot
STEP 2: Start certbot as standalone
On nas2 (store server):
certbot --standalone certonly --standalone-supported-challenges tls-sni-01
I answered to all the questions and asked for a certificate for store.voina.org. As a result I got a certificate, a private key and a full chain CA certificate generated under /etc/letsencrypt/home.voina.org directory
STEP 3: Prepare certificates for the docker registry
Note that we are making a copy of the private key and we have to put in one file the whole certificate chain.
# cd /etc/letsencrypt/live/home.voina.org/
# cp privkey.pem domain.key
# cat cert.pem chain.pem > domain.crt
# chmod 777 domain.crt
# chmod 777 domain.key
It is important to remember that this step must be done again when we renew the certificates.
To renew all the certificates just do:
certbot renew
then repeat the steps above.
STEP 4: Create an authentication token
We must create credentials to be able to login to the repository. Use the bellow to create an authentication token (myuser/mypassword)
docker run --entrypoint htpasswd registry:2 -Bbn myuser mypassword > /media/storage/work/registry/auth/htpasswd
STEP 5: Define the registry store location
Create the location where the registry data will be stored.
mkdir /media/storage/docker-registry-store
STEP 6: Start the registry
Choose one of the swarm nodes (does not have to be the swarm master), in my case nas1.voina.org and instantiate on it the registry.
docker run -d -p 5000:5000 --restart=always --name registry \
-v /media/storage/docker-registry-store:/var/lib/registry \
-v /media/storage/work/registry/auth:/auth \
-e "REGISTRY_AUTH=htpasswd" \
-e "REGISTRY_AUTH_HTPASSWD_REALM=Registry Realm" \
-e REGISTRY_AUTH_HTPASSWD_PATH=/auth/htpasswd \
-v /etc/letsencrypt/live/home.voina.org:/certs \
-e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/domain.crt \
-e REGISTRY_HTTP_TLS_KEY=/certs/domain.key \
registry:2
where:
– we mounted as an external volume the location of the registry /var/lib/registry
– we mounted as an external volume the authentication token and linked it with environment variables: REGISTRY_AUTH_HTPASSWD_PATH=/auth/htpasswd and REGISTRY_AUTH=htpasswd
– we mounted the certificate location as an external volume and linked them with environment variables: REGISTRY_HTTP_TLS_CERTIFICATE=/certs/domain.crt and REGISTRY_HTTP_TLS_KEY=/certs/domain.key
STEP 7: Login to the registry
Because this is now a secure registry we can login as we do in case of the Docker Cloud or docker.io registry. No more settings required as if of unsecure registries.
Login on the swarm master (nas2.voina.org)
docker login home.voina.org:5000
use the myuser/mypassword credentials
STEP 8: Create and add new images to the home.voina.org registry
As an example I am building a custom Jboss image on the swarm master nas2.voina.org.
The “Dockerfile” looks like:
# Use latest jboss/base-jdk:8 image as the base
FROM jboss/base-jdk:8
# Install active MQ
COPY kits/apache-activemq-5.14.3-bin.tar.gz /opt/jboss/
RUN cd /opt/jboss \
&& tar -xzf apache-activemq-5.14.3-bin.tar.gz \
&& rm apache-activemq-5.14.3-bin.tar.gz
# my custom configurations
...
To create the image execute in the same directory.
docker build --no-cache=true -t home.voina.org:5000/test-jboss .
Then push the image to the private registry
docker push home.voina.org:5000/test-jboss
Now any node of the swarm will be able to instantiate a container from this image.
STEP 9: Create a service with an image from the private registry
Define a stack.yml file for the stack containing this jboss:
version: '3'
services:
jboss:
image: home.voina.org:5000/test-jboss
ports:
- "8080:8080"
- "8443:8443"
- "9990:9990"
...
Deploy the service stack on the swarm from the swarm master nas2.voina.org:
docker stack deploy --compose-file stack.yml test-stack --with-registry-auth
Note the very important option “–with-registry-auth”. This option will forward the login credentials to all the nodes of the swarm so we do not have to login to the registry on the swarm nodes.