Simplifying TLS And Using It For Docker

Transport Layer Security (TLS) can be a complex beast and it's quite easy to skip trying to understand it when setting up a cluster or VPN. This tutorial will try to explain it whilst showing how to use it in the context of setting up secure connections between you, the client, and a Docker host/server.

Related Posts

• Use Remote Docker Host With SSH
• Create Your Own Private SSL/TLS Certificates
• Key File Formats - PEM

Table of Contents

• Terminology
  • Certificate Authority
  • Certificate Signing Request
• File Extensions
  • Docker Certificate Extension Standards
• Creating a Certificate Authority
• Setting Up A Server And Client
  • If The Requestor Is A Docker Host (Server)
  • If The Requestor Is A Client
• Testing It Works
  • Set the docker host to use certificates
  • Client Connect Using Certificates
• Conclusion
• References

Terminology

Certificate Authority

A certificate authority is a trusted party that will sign certificates, thus vouching for them. Quite often, this is a trusted third party. For example, I trust Bob and Bob vouches for Steven (shown by a signature on the certificate), so I trust Steven's certificate. This is how the world of website certificates works, with organizations such as DigiCert and Lets Encrypt acting as these trusted third parties by web browsers, that people get their certificates signed by.

You can be your own certificate authority, and this works in IT organizations, where the organization sets up the infrastructure to trust certificates issues by the organization. This s what we will be doing in this tutorial.

Certificate Signing Request (CSR)

A certificate signing request (CSR) is a message sent from an applicant to a certificate authority in order to apply for a digital identity certificate. The CSR contains information identifying the applicant, such as a distinguished name, IP or fully qualified domain name. This information must be signed using the applicant's private key and contains contains the applicant's public key accompanied by other credentials or proofs of identity required by the certificate authority.

File Extensions

Throughout the tutorial below I will use the following file extensions which may or may not be the same as other tutorial's conventions.

• .crt and .cert - a certificate file.
• .csr - a certificate signing request.
• .pem- a private key.

Docker Certificate Extension Standards

Although .crt and .cert are just both valid extensions for a PEM certificate file, the Docker daemon interprets .crt files as CA certificates and .cert files as client certificates [source]. If a CA certificate is accidentally given the extension .cert instead of the correct .crt extension, the Docker daemon logs the following error message:

Missing key KEY_NAME for client certificate CERT_NAME. CA certificates should use the extension .crt.

Creating a Certificate Authority

In this example, I am going to create a certificate authority for Programster at ca.programster.org, and use it for nodes to communicate with each other using certificates provided by this certificate authority.

First we need to generate a private key for our certificate authority. This will take a passphrase to create and then use so that if anyone else manages to steal this file, they will be unable to use it to issue certificates in the name of the certificate authority.

openssl genrsa -aes256 -out programster-ca-private-key.pem 2048

By default the generated private key is world readable. You may wish to remove this permission so that only you can view and edit it.

chmod 700 programster-ca-private-key.pem

Even though this is key encrypted, I recommend you take care with it and keep it private.

Now we are going to generate a public certificate for this certificate authority using the private key we just generated. We need to make sure to use our certificate authority's server IP or fully qualified domain name when stating the "Common Name" for the certificate.

openssl req \
  -new \
  -x509 \
  -days 365 \
  -key programster-ca-private-key.pem \
  -sha256 \
  -out programster.crt

Setting Up A Server And Client

Next we will generate a private key for a node that wishes to communicate with other nodes that use certificates issued by our certificate authority. This may be one of our servers that hosts Docker containers, or it may be a client that wishes to send Docker commands to these hosts (e.g. your personal computer). These steps are usually performed on a separate server to the certificate authority, but doesn't have to be.

openssl genrsa -out node1-private-key.pem 4096
chmod 700 node1-private-key.pem

We didn't use -aes256 in the generation of the private key this time because we want our cluster's processes to be able to use this key, hence we definitely want to change the file permissions to 700 which was "more optional" before.

Now that we have a private key for our server. We need to use it in the generation of a certificate signing request (CSR) which we will later give to our certificate authority for them to be able to provide us a public certificate that they have signed. The purpose of the CSR is to allow the authority to provide us with our public, signed certificate, without us having to give them our private key in the process.

openssl req \
  -subj "/CN=node1.programster.org" \
  -new \
  -key node1-private-key.pem \
  -out node1.csr

Replace node1.programster.org with the fully qualified domain name or IP of the node the certificate is for.

Send this CSR file over to the certificate authority for it to approve. E.g.

scp node1.csr ca.mydomain.com:/home/username/node1.csr

On our certificate authority server, we now need use the request to generate a signed certificate, in order to give the requestor our "backing".

If The Requestor Is A Docker Host (Server)

If the requestor is from a server that has a fixed IP and will be hosting containers, then create an extension file first with the following content. This will add the requestor's various IP addresses (public and private) to the certificate so that they can be verified against these.

echo subjectAltName = IP:10.10.10.20,IP:192.168.1.3 > extfile.cnf

If The Requestor Is A Client

On the other hand, if the requestor is a client with no fixed IP that you wish to authenticate when connecting to a Docker host, then use the following instead. This is typical for your laptop that you wish to send commands to your hosts from.

echo extendedKeyUsage = clientAuth > extfile.cnf

Now, using the extfile we generated, we can sign the request.

openssl x509 -req \
  -days 365 \
  -in node1.csr \
  -CA programster.crt \
  -CAkey programster-ca-private-key.pem \
  -CAcreateserial \
  -out node1.cert \
  -extfile extfile.cnf

We no longer need the certificate sign request (CSR) after having used it to generate a certificate for the requestor. Nor do we need the extfile.conf file that was used in building the certificate.

rm node1.csr extfile.conf

Now send the signed certificate, and a copy of the CA certificate back to the requestor.

scp node1.cert programster.crt user@node1.programster.org:/path/to/folder

Testing it works

After performing the steps in Setting Up A Server And Client for both a server and a client, we can set the server to listen on a public port for requests, and authenticate using the certificates we just created.

Set the docker host to use certificates

For the server, run the command below:

sudo docker daemon \
  --tlsverify \
  --tlscacert=programster.crt \
  --tlscert=node1.cert \
  --tlskey=node1-private-key.pem \
  --host=0.0.0.0:2376

• --tlsverify tells the Docker daemon to verify requests using TLS certificates.
• --tlscacert=programster.crt tells the daemon to only trust incoming connections from clients that present a certificate signed by this certificate authority public certificate.
• --tlscert=node1.cert tells the daemon the path to our public certificate for identifying ourselves and giving to clients for encrypting their communications to us.
• --tlskey=node1-private-key.pem tells the daemon what our private key is, for decrypting the communications sent to us by the clients who used our public certificate for encryption.
• --host is used to specify the host address that docker will listen to. Setting this to 0.0.0.0 configures docker to listen to all TCP connections. One can set a unix:/// socket path here instead, which is the default for Linux (unix:///var/run/docker.sock). One can use the shorthand -H instead if one desires.

Client Connect Using Certificates

For the client (your local laptop/computer), use the command below to securely connect to the host using the certificates, and fetch information about the version of docker it is running.

docker \
  --tlsverify \
  --tlscacert=programster.crt \
  --tlscert=client.crt \
  --tlskey=client-private-key.pem \
  -H=node1.programster.org:2376 \
  version

• The tlsverify and tlsacert tells the Docker client to check and trust the server if its public certificate is signed by the provided certificate-authority public certificate.
• The tlscert and tlskey parameters provides our public and private key for client authentication, for the server to trust us. We provide the server the tlscert, which it checks is signed by the certificate authority that the server expects, for it to trust us. Usually this would be the same certificate authority as we specified in tlscacert, but technically doesn't have to be. E.g. we can trust a server vouched for by Bob, but that server may only trust connections signed by Susan.

You should get output similar to below:

Client:
 Version:      1.9.1
 API version:  1.21
 Go version:   go1.4.2
 Git commit:   a34a1d5
 Built:        Fri Nov 20 13:12:04 UTC 2015
 OS/Arch:      linux/amd64

Server:
 Version:      1.9.1
 API version:  1.21
 Go version:   go1.4.2
 Git commit:   a34a1d5
 Built:        Fri Nov 20 12:59:02 UTC 2015
 OS/Arch:      linux/amd64

Conclusion

Congratulations! You can now securely send Docker commands to your hosts without having to SSH into them. This is a key step for setting up a Docker swarm.

References

• Protect the Docker daemon socket - Use TLS (HTTPS) to protect the Docker daemon socket
• Docker Docs v1.5 - Protecting the Docker daemon Socket with HTTPS
• Docker Docs - Docker Swarm