Frankly, Let’s Encrypt is possibly the best thing since sliced bread, letting you get a free SSL/TLS certificate with open source tools and an API to create and auto-renew said certificate in perpetuum.
There are already plenty of guides on getting started with Let’s Encrypt, including the canonical documentation, and Brad has talked about how excited he is by Let’s Encrypt in another article. However, today we’re going to look at something very specific that I needed for a side project: using a Let’s Encrypt certificate with a Linode NodeBalancer.
Updated 2017-02-22
This article was originally written when letsencrypt-auto was in beta, it has now been given a light update to reference the now standard Certbot tool instead.
Everything should work still, but the certificate renewal commands could be slightly shortened by using the renew action instead of certonly.
HTTPS + Node Balancing
When you have a successful web site or service, the chances are you’ll get to the point where you need to split your traffic across multiple servers in order to handle the workload. There’s a lot to be said for using multiple web servers simply to guard against server failures too. This is why a node balancer is often used to accept incoming internet traffic and quickly pass it onto the backend server best able to handle the new request, marshalling the back and forth between the client and server.
There are two common ways to handle a HTTPS request when using a node balancer in front of one or more web servers. You either pass the HTTPS request through to the web servers behind the node balancer or terminate the request at the node balancer and forward on the request as HTTP to the backend servers.
There are pros and cons to both methods. For example, at very high volumes of secure traffic, terminating at the node balancer can be problematic as such devices are generally not powerful enough to handle the decryption and encryption workload, so it is better to pass the HTTPS traffic through to the web servers. Conversely, as Let’s Encrypt certificates have a relatively short lifetime of 90 days, ensuring that your many web servers are properly configured with auto-renewed certificates can be a pain in comparison to updating a single or just a few node balancers.
First we will discuss setting up a TCP pass-through configuration for HTTPS on the node balancer as it is somewhat easier if you’re used to setting up HTTPS on a web server, and then we’ll see how to set up HTTPS termination on a Linode NodeBalancer and keep its certificate up to date when the relatively short term Let’s Encrypt certificate is renewed.
TCP Pass-through
Because setting up a configuration to accept HTTPS traffic on a Linode NodeBalancer normally requires that you add a certificate to the configuration, but with this method we want the backend web servers to terminate the HTTPS request, we instead need to set up a configuration on the node balancer that uses the TCP protocol and port 443.
In the above screenshot you’ll see the basic setup for a TCP configuration:
Port: 443
Protocol: TCP
Algorithm: Least Connections (my choice, could be Round Robin or Source IP)
Session Stickiness: Table (my choice, could be None or HTTP Cookie)
For the “Active Health Check” section I’ve stuck to the defaults for the Check Interval, Timeout and Attempts, but am specifically using “TCP Connection” as the Health Check Type as I do not then need to rely on there being a HTTP server on the backend. I could use only HTTPS if I wanted.
Also on the screenshot of my Linode NodeBalancer configuration you’ll see that I’ve already added a node: “apps2” is the Debian 8.x based server that we’re going to configure to provision and use a Let’s Encrypt certificate.
To ease setup of a Let’s Encrypt certificate it is important to be able to serve HTTP requests from the same domain name you’re going to use as the primary domain on the certificate, so I also set up a standard HTTP NodeBalancer configuration that had “apps2” as a node.
Setting Up Let’s Encrypt
As mentioned before, there are plenty of articles on how to set up a web server to use a Let’s Encrypt certificate, so I’m not going to go into great detail here. Here’s the skinny on what I did to make sure I could use Let’s Encrypt to serve le.ianmjones.com from a valid certificate.
- Created a subdomain entry in my DNS to point le.ianmjones.com to my node balancer’s IP address.
- Made sure I had a working virtual host for serving http://le.ianmjones.com
sudo su -to ensure I was the root user (Let’s Encrypt needs to write to /etc and into your web root).apt-get install letsencryptletsencrypt certonly --webroot -w /var/www/le.ianmjones.com/ \ -d le.ianmjones.com --staging
After the Let’s Encrypt tool and all its dependencies have been installed, the first run asked questions to generate a certificate signing request, such as asking for an email address to associate with the account it creates with the service. It’s important to set a real email address as this is where reminders will be sent if you let your certificate get close to its expiry date before renewing it. The letsencrypt tool isn’t just for Debian, it’ll work on many Linux distributions, FreeBSD and OpenBSD, additional platforms are continually being worked on.
When using the letsencrypt tool I passed the params certonly and --webroot, this was to ensure that a certificate was to be generated that I could then use elsewhere, and to tell Let’s Encrypt to use the webroot method of validating that I was in control of the domain. When you use this combination of parameters letsencrypt deposits a randomly named file in a .well-known/acme-challenge/ directory it creates under the web root you specify with the -w flag, and asks the Let’s Encrypt service to check for it to confirm that it is accessible from the domain. This is why I needed to have a working HTTP server for the le.ianmjones.com domain, at least at first.
We also need to use the certonly method rather than --apache or other web server specific methods of provisioning. Although they are convenient, we later want to use these certificates in a different manner with our node balancer terminating method (I’m thinking ahead).
You’ll notice that I used the --staging parameter. This is because when first setting up your certificates you might run into troubles with your configuration and need to fix things and run letsencrypt multiple times until you are confident that everything is working smoothly. The live Let’s Encrypt server has quite low rate limits (only 5 certificates issued or renewed per public domain per week at time of writing) to stop abuse by wayward processes and will refuse to work for you for a few hours or even days if you call it too often. The staging server has much higher limits and so should be used during initial set up.
With letsencrypt having been run and come back with the OK, everything I needed to configure my Apache web server was under the /etc/letsencrypt directory. For configuring Apache, the most important files for me were under /etc/letsencrypt/live/le.ianmjones.com/:
root@apps2:/etc/letsencrypt# ls -l live/le.ianmjones.com/
total 0
lrwxrwxrwx 1 root root 40 Jan 29 00:23 cert.pem -> ../../archive/le.ianmjones.com/cert3.pem
lrwxrwxrwx 1 root root 41 Jan 29 00:23 chain.pem -> ../../archive/le.ianmjones.com/chain3.pem
lrwxrwxrwx 1 root root 45 Jan 29 00:23 fullchain.pem -> ../../archive/le.ianmjones.com/fullchain3.pem
lrwxrwxrwx 1 root root 43 Jan 29 00:23 privkey.pem -> ../../archive/le.ianmjones.com/privkey3.pem
You’ll notice that live/le.ianmjones.com contained symbolic links to the actual files in an archive directory, and that the linked files all had a 3 in their name. This is a neat mechanism that means when you renew your certificate files the symbolic links are automatically updated to point to the new versions so you can continue to reference the same files in your web server configurations. In this case, once I’d run a couple of tests with the --staging flag, I then removed that flag and ran letsencrypt against the live service.
letsencrypt certonly --webroot -w /var/www/le.ianmjones.com/ -d le.ianmjones.com
To set up the Apache virtual host for serving HTTPS from le.ianmjones.com, I used the cert.pem, chain.pem and privkey.pem file paths as values for the SSLCertificateFile, SSLCertificateChainFile and SSLCertificateKeyFile directives respectively. The following is my full virtual host configuration file at /etc/apache2/sites-available/le.ianmjones.com.conf and made active by a2ensite le.ianmjones.com:
<VirtualHost *:443>
ServerName le.ianmjones.com
ServerAdmin [email protected]
DocumentRoot /var/www/le.ianmjones.com
DirectoryIndex index.html index.php
SSLEngine On
SSLCertificateKeyFile /etc/letsencrypt/live/le.ianmjones.com/privkey.pem
SSLCertificateFile /etc/letsencrypt/live/le.ianmjones.com/cert.pem
SSLCertificateChainFile /etc/letsencrypt/live/le.ianmjones.com/chain.pem
ErrorLog ${APACHE_LOG_DIR}/error.log
CustomLog ${APACHE_LOG_DIR}/access.log combined
<Directory /var/www/le.ianmjones.com>
AllowOverride All
</Directory>
</VirtualHost>
<VirtualHost *>
ServerName le.ianmjones.com
ServerAdmin [email protected]
DocumentRoot /var/www/le.ianmjones.com
DirectoryIndex index.html index.php
ErrorLog ${APACHE_LOG_DIR}/error.log
CustomLog ${APACHE_LOG_DIR}/access.log combined
<Directory /var/www/le.ianmjones.com>
AllowOverride All
</Directory>
</VirtualHost>
# vim: syntax=apache ts=4 sw=4 sts=4 sr noet
At this point I had a working https://le.ianmjones.com with a Let’s Encrypt certificate, being served from behind a Linode NodeBalancer.
Certificate Auto Renewal
If you’ve been following along and already tried running letsencrypt a few times, you’ll have noticed that it asks questions each time. That’s going to be a problem if you’re trying to set up automatic renewal of your certificate.
Luckily there is a renew-by-default parameter you can use to specify that that’s all you want to do and nothing has changed since the last time you configured the given domains. Let’s Encrypt will pick up all it needs from the relevant file in /etc/letsencrypt/renewal. You can use the -n (non-interactive) flag so that no questions are asked either, along with --agree-tos and -m [email protected] to keep everything in order.
As an aside, there is a new letsencrypt renew command on its way that should be much better. It will check to see which certificates are up for renewal in the next 30 days and renew just them It will not need to be told what domains and verification methods to use.
Before setting up my cron to renew the certificate once a week, I created a little shell script to be run, then tested it and gracefully restarted Apache (with apache2ctl graceful).
After the restart of Apache the certificate on the site reflected the new expiry date.
All that was left to do was to crontab -e and add a weekly run of the script (a better admin might add something to /etc/cron.weekly/).
5 5 * * 1 /root/renew_lets_encrypt.sh && /usr/sbin/apache2ctl graceful
If you want to be super careful with the Let’s Encrypt rate limits, I’d suggest setting your cron job to run once a month instead, maybe twice a month. The only reason I set up to renew my Let’s Encrypt certificates once a week is to make sure the feedback cycle during the Let’s Encrypt beta phase wasn’t too long.
I also of course temporarily added a duplicate entry to my crontab but with a time just a few minute in the future and then later checked that Apache had restarted at the expected time and that my SSL certificate had been updated with a new expiry date and time. This ensured that environment that cron runs the script in didn’t have any problems.
This method would be very easy to adapt to your preferred web server, and easy to extend so that you push the new certificate files out to your other servers, whether that be by scp, rsync, ansible and its brethren, or by sharing config files through GlusterFS. If you’ve already set up a bunch of node balanced web servers you’re bound to have your preferred method for syncing config files.
NodeBalancer Terminated HTTPS
To set up a Linode NodeBalancer that handles the HTTPS decryption/encryption and that automatically updates with a renewed Let’s Encrypt certificate, I first set up the new NodeBalancer port configuration manually.
Because I wanted to use the same Linode NodeBalancer to handle HTTPS that I’d already set up with a configuration that handled port 443 traffic, I first removed that configuration and added a new HTTPS configuration.
You’ll notice two new large fields in the above screenshot, SSL Certificate and Private Key, these are required for the NodeBalancer to be able to handle the HTTPS protocol. Because I’d already set up my server to obtain a certificate from Let’s Encrypt, and they provide both a fullchain.pem and privkey.pem that are perfect matches for the two new fields, I just copied their contents into those two fields and saved.
root@apps2:/etc/letsencrypt/live/le.ianmjones.com# cat fullchain.pem
...
root@apps2:/etc/letsencrypt/live/le.ianmjones.com# cat privkey.pem
After that, it was just a matter of adding in my web server node, using port 80 as the connection as the HTTPS/port 443 traffic was now being terminated on the NodeBalancer.
At this point I had a working HTTPS configuration on the Linode NodeBalancer, using the same SSL certificate that I last renewed from Let’s Encrypt.
Updating The NodeBalancer Certificate
Earlier, I set up my web server to automatically renew its certificate every week. Now that I’ve switched to using my certificate on a NodeBalancer, I need to extend the renewal process so that it updates the NodeBalancer.
Linode have a great API for managing your account, including not only your servers (Linodes) but also NodeBalancers.
At first I intended to use the raw API with PHP cURL, as the recommended PHP library is out of date and doesn’t handle setting up HTTPS on a NodeBalancer. I used the awesome Paw application to test the various API calls and see some sample PHP using its PHP cURL code generation extension.
Which worked well, but meant I’d be fiddling with storing usernames, passwords, and API keys, and then handling the flow of calls required to pick out the configuration to update and then stuff the data into it, properly encoded etc. It’s not rocket science, we do a lot of this every day as PHP developers, but, bleh.
Then I checked out the Linode CLI, and low and behold it had everything I needed and was installed with a simple:
# apt-get install linode-cli
# linode configure
A new renew_lets_encrypt.sh shell script was born, or is that bourne?
#!/bin/bash
if [ ! -d letsencrypt ]
then
git clone https://github.com/letsencrypt/letsencrypt
fi
if [ ! -d letsencrypt ]
then
echo "Could not find letsencrypt directory, exiting."
exit 1
fi
cd letsencrypt
# le.ianmjones.com
letsencrypt certonly --webroot --renew-by-default -w /var/www/le.ianmjones.com/ -d le.ianmjones.com
EXIT_STATUS=$?
if [ $EXIT_STATUS -ne 0 ]
then
exit $EXIT_STATUS
fi
/usr/bin/linode nodebalancer config-update --label apps --port 443 --ssl-cert /etc/letsencrypt/live/le.ianmjones.com/fullchain.pem --ssl-key /etc/letsencrypt/live/le.ianmjones.com/privkey.pem
exit $?
This effectively added a single line to my script to update the configuration for port 443 of NodeBalancer “apps” with a new SSL certificate and private key. All the authentication has been previously handled by the use of linode configure, so this command can run without the need for entering any user details or API keys.
All that remains is to remove all our Apache virtual host configs for port 443 and also remove the && apache2ctl graceful from the cron entry as we no longer need to reload Apache to pick up the new SSL certificates.
5 5 * * 1 /root/renew_lets_encrypt.sh
And with that I had auto-renewing Let’s Encrypt certificates being pushed out to my Linode NodeBalancer.
Adding Domains
What if I wanted to add lex.ianmjones.com to my HTTPS setup?
With my vhost already set up for HTTP, all I had to do was add the new domain and its webroot to the end of my letsencrypt command. It’s best to run it without the --renew-by-default flag for the first time as Let’s Encrypt will want to confirm that you intend to expand the certificate to include the extra domain(s).
root@apps2:~# ./letsencrypt certonly --webroot -w /var/www/le.ianmjones.com/ -d le.ianmjones.com -w /var/www/lex.ianmjones.com/ -d lex.ianmjones.com
Ensure that the renew script is updated with the new webroot and domain, and then manually run the command to update the node balancer unless you’re happy to use up another of your very limited certificate issues per week allowance.
root@apps2:~# linode nodebalancer config-update --label apps --port 443 --ssl-cert /etc/letsencrypt/live/le.ianmjones.com/fullchain.pem --ssl-key /etc/letsencrypt/live/le.ianmjones.com/privkey.pem
Updated NodeBalancer config apps 443
The certificate on lex.ianmjones.com reflects the changes.
Simple Stuff
I wrote a lot of words there, but when you look at the renew_lets_encrypt.sh script it’s all pretty simple, just a couple of CLI calls with the right parameters. Command line tools like letsencrypt and linode make our tasks so much simpler.
It’s the preparation that makes the difference, knowing what domains you need to get certificates for ahead of time will help you to minimize the number of times you request certificates (a big issue in the current Let’s Encrypt beta). If you decide to use HTTPS on the node balancer from the start, then you’ll not even need to mess with setting up HTTPS on your web server and passing TCP port 443 through.
You can now just merrily add web server nodes to your node balancer’s HTTPS configuration and not need to worry about configuring certificates.
Have you played with Let’s Encrypt yet? How about the Linode API and its CLI? We’d love to see any tips you might have in the comments below.
