Tag Archives: nginx

Centralized Let’s Encrypt Management

Updated March 16, 2017 to reflect current webroot settings

Recently I set out to see how I could manage lets encrypt certificates from one central server, even though the actual websites didn’t live on that server. My reasoning was basically “This is how I did it with SSLMate, so let’s keep doing it” but it should also be helpful in situations where you have a cluster of webservers, and probably some other situations that I can’t think of at this time.

Before I get too in depth with how this all works, I’m going to define what I mean by two servers we have to work with:

  • Cert Manager: This is the server that actually runs Let’s Encrypt, where we run commands to issue certificates.
  • Client Server: This is the server serving the website, say… chrismarslender.com ūüėČ

Additionally, I have a domain setup that I point to the Cert Manager. For the purposes of this article, lets just call it certmanager.mywebsite.com.

High Level Overview

At a high level, here’s how it works with the web root verification strategy:

  1. I set up nginx on the Cert Manager to listen for requests at certmanager.mywebsite.com, and if the request is for anything under the path /.well-known/ I serve up the file the request is asking for.
  2. On the client servers, I have a common nginx include that matches the /.well-known/ location, and proxies that request over to the certmanager.mywebsite.com server.

Nginx Configuration

Here’s what the configuration files look like, for both the Cert Manager Server as well as the common include for the client servers:

Cert Manager Nginx Conf:

server {
    listen 80;
    
    server_name certmanager.mywebsite.com;
    
    access_log /var/log/nginx/cert-manager.access.log;
    error_log /var/log/nginx/cert-manager.error.log;

    root /etc/letsencrypt/webroot;

    location /.well-known {
        try_files $uri $uri/ =404;
    }

    location / {
        return 403;
    }
}

Client Server Common Nginx Include:

location ~ /\.well-known {
    proxy_pass http://certmanager.mywebsite.com;
    resolver 8.8.8.8;
}

Issuing a Certificate

Now lets say I want to issue a certificate for chrismarslender.com – here is what the process would look like.
I’m assuming chrismarslender.com is already set up to serve the website on a client server by this point.

SSH to the Cert Manager server, and run the following command:

letsencrypt certonly -a webroot --webroot-path /etc/letsencrypt/webroot -d chrismarslender.com -d www.chrismarslender.com

Eventually, this command generates¬†a verification file in the /etc/letsencrypt/live/.well-known/ directory, and then Let’s Encrypt tries to load the file to verify domain ownership at chrismarslender.com/.well-known/<file>.

Since the client server hosting chrismarslender.com is set up to proxy requests under /.well-known/ to the Cert Manager server (using the common include above), the file that was just created on the Cert Manager server is transparently served to Let’s Encrypt, and ownership of the domain is verified. Now, I have some fancy new certificates sitting in /etc/letsencrypt/live/chrismarslender.com

At this point, you just have to move the certificates to the final web server, reload nginx, and you’re in business.

In practice, I actually use ansible to manage all of this – I’ll work on a follow up post explaining how that all works as well, but generally I end up issuing SSL certificates as part of the site provisioning process on the Client Servers, in combination with `delegate_to`. Also, ansible makes steps like the moving of certificates to the final web server must less labor intensive ūüôā

Things to Figure Out

I’m still trying to figure out the best strategy to keep the certificates updated. I can run the Let’s Encrypt updater on the Cert Manager server and get new certificates automatically, but since it’s not the web server that actually serves the websites, I need to figure out how I want to distribute new certificates to appropriate servers when they are updated. Feel free to comment if you have a brilliant idea ūüėČ

URL Based Variables in Nginx

Over the past few months, I’ve set up a few fairly complex staging environments for websites I’ve been working on.

One setup creates a new subdomain based on the ticket number so we can test just that branch of code. If the ticket number is ticket-123, the testing url might look something like ticket-123.staging.example.com. I have Jenkins set up to create a directory for each site at something like /var/www/html/ticket-123.

Another setup is a staging installation for a large multisite install that utilizes domain mapping, so there are many different domains all on the same multisite install (site1.com, site2.com, and site3.com). The staging server for this clones the production database, does some magic on the urls, and I end up with staging urls like site1.staging.example.com, site2.staging.example.com, and site3.staging.example.com. To save some disk space and avoid the headache of copying a bunch of media every time we move the database from production to staging, I proxy the images from the production site.

All of this could be set up manually, but creating a new nginx config file each time a new ticket is staged or having to set up a separate rules for each site we want to proxy images for on the multisite would be tedious work.

Here’s how I solved these issues.

Nginx allows you to use regular expressions in your server_name line. In addition,¬†you can capture certain parts of the url for use later, by giving them a name. Here’s an example of how I match for a ticket number based on a URL structure that looks like ticket-123.staging.example.com

server {
    server_name  ~^(?P<ticket>.+)\.staging\.example\.com$
}

The above should match any subdomain on staging.example.com and store the preceding segment of the URL in the $ticket variable. Now that I have the $ticket variable, I can use this information to point nginx to the correct site root.

server {
    server_name  ~^(?P<ticket>.+)\.staging\.example\.com$
    root    /var/www/html/$ticket;
}

Now any request that comes in for a staged ticket will automatically serve the files from the correct location.

Multisite Image Proxy

The multisite install uses similar techniques for a different end result. In this case, we are only ever staging one codebase at a time (not different tickets), but there are a bunch of images that we want to proxy from the production server. Here’s the catch – the production urls vary, because the main site uses domain mapping. Here’s an example of how the URLs translate from production to staging

  • www. site1.com -> site1.staging.example.com
  • www.site2.com -> site2.staging.example.com
  • www.site3.com-> site3.staging.example.com

Luckily there is a pattern to how the URLs change, so this is a problem I was able to solve again using the named variable capture in Nginx.

Here’s an example of what the server name looks like in the Nginx config (It nearly identical to above)

server {
    server_name  ~^(?P<subsite>.+)\.staging\.example\.com$
}

Again, now that I have the $subsite variable available, I can use that to construct the URL to proxy images from (See this post for more on proxying images with Nginx).

Here’s what the nginx config looks like¬†to accomplish the smart image proxy

server {
    server_name  ~^(?P<subsite>.+)\.staging\.example\.com$

    location ~* ^.+\.(svg|svgz|jpg|jpeg|gif|png|ico|bmp)$ { {
        try_files $uri @image_fallback;
    }

    location @image_fallback {
        resolver 8.8.8.8;
        proxy_pass http://www.$subsite.com;
    }
}

 

Fallback Images

When working on a website, I always develop locally – usually using Varying Vagrant Vagrants. I’ll often times pull down a copy of the production database and do a search and replace to make sure I’m dealing with local urls, so that I have some real content to develop with. This works great, except for a bunch of annoying missing images. I could download all the images from the server, but who wants all those files on their computer? I don’t.

My solution to this was to have nginx serve the images from the original server, if they are not present locally. It seems to be working great so far, and all it took was a few extra lines in the nginx config.


location ~* ^.+\.(svg|svgz|jpg|jpeg|gif|png|ico|bmp)$ {
    try_files $uri @image_fallback;
}

location @image_fallback {
    proxy_pass http://example.com;
}

For any .jpg, .gif, or .png file, nginx first tries to find the file locally. If its not there, it passes it along to example.com and tries to find it there. This could probably be expanded to try and get other file types as well, but at the time, I only needed these image types.