Centralized Let’s Encrypt Management

Updated March 16, 2017 to reflect current webroot settings

Recently I set out to see how I could manage lets encrypt certificates from one central server, even though the actual websites didn’t live on that server. My reasoning was basically “This is how I did it with SSLMate, so let’s keep doing it” but it should also be helpful in situations where you have a cluster of webservers, and probably some other situations that I can’t think of at this time.

Before I get too in depth with how this all works, I’m going to define what I mean by two servers we have to work with:

  • Cert Manager: This is the server that actually runs Let’s Encrypt, where we run commands to issue certificates.
  • Client Server: This is the server serving the website, say… chrismarslender.com 😉

Additionally, I have a domain setup that I point to the Cert Manager. For the purposes of this article, lets just call it certmanager.mywebsite.com.

High Level Overview

At a high level, here’s how it works with the web root verification strategy:

  1. I set up nginx on the Cert Manager to listen for requests at certmanager.mywebsite.com, and if the request is for anything under the path /.well-known/ I serve up the file the request is asking for.
  2. On the client servers, I have a common nginx include that matches the /.well-known/ location, and proxies that request over to the certmanager.mywebsite.com server.

Nginx Configuration

Here’s what the configuration files look like, for both the Cert Manager Server as well as the common include for the client servers:

Cert Manager Nginx Conf:

server {
    listen 80;
    
    server_name certmanager.mywebsite.com;
    
    access_log /var/log/nginx/cert-manager.access.log;
    error_log /var/log/nginx/cert-manager.error.log;

    root /etc/letsencrypt/webroot;

    location /.well-known {
        try_files $uri $uri/ =404;
    }

    location / {
        return 403;
    }
}

Client Server Common Nginx Include:

location ~ /\.well-known {
    proxy_pass http://certmanager.mywebsite.com;
    resolver 8.8.8.8;
}

Issuing a Certificate

Now lets say I want to issue a certificate for chrismarslender.com – here is what the process would look like.
I’m assuming chrismarslender.com is already set up to serve the website on a client server by this point.

SSH to the Cert Manager server, and run the following command:

letsencrypt certonly -a webroot --webroot-path /etc/letsencrypt/webroot -d chrismarslender.com -d www.chrismarslender.com

Eventually, this command generates a verification file in the /etc/letsencrypt/live/.well-known/ directory, and then Let’s Encrypt tries to load the file to verify domain ownership at chrismarslender.com/.well-known/<file>.

Since the client server hosting chrismarslender.com is set up to proxy requests under /.well-known/ to the Cert Manager server (using the common include above), the file that was just created on the Cert Manager server is transparently served to Let’s Encrypt, and ownership of the domain is verified. Now, I have some fancy new certificates sitting in /etc/letsencrypt/live/chrismarslender.com

At this point, you just have to move the certificates to the final web server, reload nginx, and you’re in business.

In practice, I actually use ansible to manage all of this – I’ll work on a follow up post explaining how that all works as well, but generally I end up issuing SSL certificates as part of the site provisioning process on the Client Servers, in combination with `delegate_to`. Also, ansible makes steps like the moving of certificates to the final web server must less labor intensive 🙂

Things to Figure Out

I’m still trying to figure out the best strategy to keep the certificates updated. I can run the Let’s Encrypt updater on the Cert Manager server and get new certificates automatically, but since it’s not the web server that actually serves the websites, I need to figure out how I want to distribute new certificates to appropriate servers when they are updated. Feel free to comment if you have a brilliant idea 😉

23 thoughts on “Centralized Let’s Encrypt Management

  1. Hello Chris, have you figured out a way to deploy the certificates?

    I was thinking about NFS, it would be really transparent, but I would be adding an additional layer of problem, if the NFS goes down we’ve a problem.

    Suggestions?

    1. I haven’t had time to dig in and find a good solution for that yet, unfortunately. For now, I’m just issuing the certificates on the individual webservers and setting up the auto renew there.

    2. I decided to go down this route as well.

      I am on AWS and I’ve got 15,000 domains to secure. I am going to make use of Elastic File Storage (aws version of NFS) to host them.

      1. With nginx, if the underlying certificate changes, you need to restart the service to load up the new certificate. How are you planning on detecting changes to the certificates and restarting across all the servers?

        1. Hello cmmarslender,

          I think you don’t need to restart nginx. You can run “nginx reload” to re-load the certs without downtime.

          As far as detection goes: I am going to write a small service that checks new domains every 15 minutes and updates the configuration & issue reload command.

          I could not found any better way. New domains gets added/removed every day on my platform 🙁

          1. Ah yeah – reload or restart, my point was more or less the same. I was curious if you had found a better solution for detecting changes and issuing the command to reload.

  2. incron (Linux only) uses inotify to trigger commands. Using incron, you can detect the cert file changing and use scp/ssh to distribute the key(s) and reload nginx.

  3. Integrate with Ansible.

    – CertManager will get new Certificates periodically and then let Ansible to do the remaining Jobs

    – Ansbile copy new certificate to the destination server
    – Ansible restart web services in destination server

  4. Great article – and excellent input about Ansible, Dijeesh.
    I’ll be looking into a solution involving Puppet – anyone having experience with such a setup?

  5. Great article Chri! Please correct me if I am wrong but… how smart it is to set nginx root directory as your certificate root directory?

    I think this is really dangerous thing to do: “root /etc/letsencrypt/live;”

    Any small mistake in the future will open up all of your certs to outside world 😀

    1. This is a good point. In the configuration provided in the article, the location / block that returns a 403 (forbidden) was intended to prevent this from happening, but I’ve since changed how I do this a bit to avoid that requirement. My location block looks like this now:

      location ~ /\.well-known {
      	root /etc/letsencrypt/webroot;
      	try_files $uri $uri/ =404;
      }
      

      Defining the root inside of the location block leaves you free to set the other root to anything you desire. I’ve updated the article to reflect not using the /live directory.

        1. The reason they mention it as “bad” is because in their “bad” example, they don’t have any root at all outside of the location blocks, which means that you are forced into defining a root for every single location block.

          Using root inside a location block to override the default root for the server block is totally fine (and in fact, demonstrated in their “good” example).

          In my case, I have a root just inside the `server` block that points to my normal website files, and then inside the Let’s Encrypt location block, I have `root /etc/letsencrypt/webroot`thus satisfying the need to have the fallback outside the location blocks as well as a root specific to Let’s Encrypt.

  6. I am still torn on how to automate the distribution of the renewed certs. I have configured a cron job to automatically renew the lets encrypt cert; but then i dont wish to stay using ansible to distribute the files. I was considering using a repo on a private git server secured by ssh keys and pushing the entire LE folder structure as a new commit whenever a new version of the certs is available. Then from the client side I can stay polling for new commits and pull. My concern with this approach is security then..thoughts anyone?

  7. Thank you for this great article Chris. You can also use lua-resty-auto-ssl to dynamically manage ssl certificates for your websites. An added advantage of this library is that you can save certificates in redis. So that if you have a networked redis instance you can simply store all your certificates outside the main application server.

    Which means that if in future you want to create multiple app-server instances you can connect them all to the same redis-server without worrying about the location of your certificates.

    The renewal of certificates is automatic. There is a parameter that you can configure to let auto-ssl know at what intervals the renewal checks should be performed. Although the default options are quite good for a small setup.

    Furthermore since you are adding the certificates dynamically for each one of your websites there is no need for you to reload nginx configuration every time a new certificate is added.

    https://github.com/GUI/lua-resty-auto-ssl

  8. Great article and one I plan to follow for my own setup.

    What happens if you run certbot renew on your cert manager server ? Does this remember where the certs are and automatically renew them ?

    1. It does not, and that isn’t something that I’ve figured out. More recently, I’ve been tinkering with Rancher, and the certificates are managed a bit differently there. The lua-resty-auto-ssl mentioned in one of the other repos seems like it has some promise though. One day, I’ll probably give that a shot.

  9. Anyone able to set up the server that manage all Let’s Encrypt certificates and renew all certificates at once. I am trying to do that

  10. I have a central manage lets encrypt server that handles all my website and mail certs. I use a similar proxyPass to issue/renew my certs, store everything on a GlusterFS (mounted to all my servers), and inotify (or FSwatch) w/ cron to trigger nginx or postfix reloads. In order to prevent all the web or mail servers from reloading at the same time I added a ‘sleep X’ command in the cron and staggered the time.

    1. Hi Rich, could you please provide more information about your solution? I’m in a very similar situation, with HAProxies that need SSL certs and a GlusterFS cluster already used for another purpose. I’m still wondering if it’s better to create a centralized cert manager or use one of my HAProxies to request the certificates, but the idea is to put them where the othere proxies will be able to find them (Gluster).
      Thank you,
      sguido

Leave a Reply

Your email address will not be published. Required fields are marked *