Friday, February 10, 2017

Easy when you know how: Let's Encrypt open source "automated" TLS/SSL certificates and keys

We recently renewed our StartCom certs for our vhost domains.

The sites immediately failed SSL:
     Chromium reported

     Firefox reported

          Trusted No NOT TRUSTED

It turns out that Apple, Mozilla, Google no longer trust StartCom:
so the CA (Certifying Authority) cert(ificate) was not being accepted.

So, what to do? We decided to go with Let's Encrypt:

Easy when you know how, except we didn't. But we do now. After working through this tutorial you shall also. 
This post provides a top level of the gotchas we found along the path.


The system is a bit short on concise documentation (hence this post) but there is a helpful wiki:
and, as always, Google is your friend. 

First, Let's Encrypt is a system put in place by the EFF (Electronic Frontier  Foundation) to provide free TLS/SSL certificates:

The objective of Let’s Encrypt and the ACME protocol is to make it possible to set up an HTTPS server and have it automatically obtain a browser-trusted certificate, without any human intervention. This is accomplished by running a certificate management agent on the web server.

The agent operates under the ACME (Automatic Certificate Management Environment) protocol. Any number of clients for the ACME server exist: the protocol is open source, so anyone can write a client for it and many have. recommends we use their “official” client CertBot:
Certbot is an easy-to-use automatic client that fetches and deploys SSL/TLS certificates for your webserver. Certbot was developed by EFF.
Certbot is part of EFF’s larger effort to encrypt the entire Internet. Websites need to use HTTPS to secure the web.

Except that we were unable to get it to work under openSUSE. It is not in the normal repositories. We found several ymp packages but all failed either to install or installed but failed to run: fails to run after installation. clones a git installation. It runs, but only checks its installation:
     certbot-auto certonly --webroot -w /srv/www/htdocs/gvhl \ -d
Bootstrapping dependencies for openSUSE-based OSes...
Loading repository data...
Reading installed packages..

xxx is already installed...
Nothing to do. lists a huge number of alternative. But which? suggests GetSSL:
     GetSSL is designed so you can also run it from a different server, providing you have SSH / SFTP to the server you want to place the certs on if needed, for servers / devices where you couldn't run most other clients.

Ultimately we went with Dehydrated:

We found
and it just runs. But it needs configuration.

Configuring Dehydrated

To begin please note that, unlike StartCom, CertBot, and GetSSL, you need to install and run Dehydrated on the target server machine, not a proxy.

These are the steps:
1. Install Dehydrated
2. Set the Staging
3. Create domains.txt
4. Create well-known
5. Create /etc/dehydrated/config

6. Configure Apache2
7. Start it:
     # dehydrated -c -f /etc/dehydrated/config
Now: Success!
8. Reverse the Staging and reexecute
9. Get the CA certs
10. Link the /etc/dehydrated/certs files to /etc/apache2/ssl.crt and ...key
11. Amend the apache .conf files to address these.
12. Now check the certs.
13. Go to production.

1. Install

openSUSE installation requires that you add the repository with yast
on the server:

Then use yast sw_single to install:

2. Staging
Let’s Encrypt has stringent rate limits in place.
If you start testing using the production endpoint (which is the default), you will quickly hit these limits and find yourself locked out.
To avoid this, please set the CA property to the Let’s Encrypt staging server URL in your
config file:
That means you issued those certificates against the staging server (possibly with --test-cert or --dry-run). You should reissue against the production server by removing those flags from your client invocation, if present.

Once you've gotten things working mostly, comment out these to get actual (not fake) certs:
Once your testing is done against the test certificate authority server (happy hacker fake CA) you need to edit the script to point to the production certificate authority server to generate a valid certificate. Edit the script and look for the variable at the top called "CA=" and uncomment out the "official server" and comment the "testing server". For example:
# The Lets Encrypt certificate authority URL
#CA="" # testing server, high rate limits. "happy hacker fake CA"
CA="" # official server, rate limited to 5 certs per 7 days

DO NOT delete the staging certs: it will break apache! Just let letsencrypt replace them when you go to production.

After all the testing is done you will need to run it with the -x option, since you will have created test certificates and it will refuse to regenerate actual certificates unless forced to do so:
     dehydrated -c -x -f /etc/dehydrated/config

3. domains.txt
Create /etc/dehydrated/domains.txt on the server:
Dehydrated uses the file domains.txt as a configuration file for which certificates should be requested.

4. well-known
n the server machine:

     # cd /var     
     # mkdir www     
     # mkdir www/dehydrated
     # mkdir www/dehydrated/.well-known
     # mkdir www/dehydrated/.well-known/acme-challenge
     # touch www/dehydrated/.well-known/acme-challenge/m4g1C-t0k3n

Change the owner and group
     # cd /var/www/dehydrated
     # chown -R wwwrun:www .well-known

Change the mod from 0755 to 0775

     # chmod -R 0775 .well-known

Copy it to each :

     # cp -R .well-known /srv/www/htdocs/
     # cp -R .well-known /srv/www/htdocs/gvhl

     # cp -R .well-known /srv/www/htdocs/nptbeyond
     # cp -R .well-known /srv/www/htdocs/privustech
     # cp -R .well-known /srv/www/htdocs/truthcourage


5. config file
is mostly commented out, showing the default values.

Two values must be uncommented and corrected:
     # E-mail to use during the registration (default: )

6. Configure Apache2

We must add a dehydrated.conf file in /etc/apache2/sysconfig.d:
Edit /etc/apache2/httpd.conf to add
     Include /etc/apache2/sysconfig.d/dehydrated.conf

Create /etc/apache2/sysconfig.d/dehydrated.conf to read:
     Alias /.well-known/acme-challenge /var/www/dehydrated     
     Options None

     AllowOverride None
     # Apache 2.x     
     Order allow,deny

     Allow from all
     # Apache 2.4     
     Require all granted

The #Apache 2.x section is deprecated so we delete it.

Restart the server.

7. Start it

The first time you run it it will be in the Staging mode.
     # dehydrated -c -f /etc/dehydrated/config
Now: Success!

8. Reverse the Staging and reexecute

9. Get the CA certs

LetsEncrypt keeps their root offline, so we need to use the active Intermediate as the CA cert. But which? They have a number of different intermediate certs:
     Our intermediate “Let’s Encrypt Authority X3” represents a single public/private key pair. The private key of that pair generates the signature for all end-entity certificates (also known as leaf certificates), i.e. the certificates we issue for use on your server.

The one that works for us is the cross-signed cert:

So we would expect that cert to verify the leaf cert:
lavarre:/etc/apache2/ssl.crt # openssl verify -verbose -CAfile lets-encrypt-x3-cross-signed.pem OK

And indeed, it does. Woohoo!

10. Links

Link the /etc/dehydrated/certs files to /etc/apache2/ssl.crt and ...key

Link to the CA cert:
# cd /etc/apache2/ssl.crt
# ln -s lets-encrypt-x3-cross-signed.pem letsencrypt_CA.pem

Link to the Domain Certificates and Keys Link to each cert in /etc/apache2/ssl.crt.
Link to each key in /etc/apache2/ssl.key.
We wrote a BASH script tldr_link that automates this:
     for DOMAIN_TLDR in {,}
     cd /etc/apache2/ssl.crt
      ln -s /etc/dehydrated/certs/$DOMAIN_TLDR/cet.pem $DOMAIN_TLDR.pem
     ln -s /etc/dehydrated/certs/$DOMAIN_TLDR/chain.pem $DOMAIN_TLDR_chain.pem
     ln -s /etc/dehydrated/certs/$DOMAIN_TLDR/fullchain.pem $DOMAIN_TLDR_fullchain.pem
     cd /etc/apache2/ssl.key
     ln -s /etc/dehydrated/certs/$DOMAIN_TLDR/privkey.pem $DOMAIN_TLDR_key.pem

11. Amend the apache.conf files

Amend the apache .conf files to address these. entities.

We check configuration files. Different files have different server names CA cert files. So we make them all point to
     SSLCACertificatePath /etc/ssl/certs
     SSLCACertificateFile /etc/ssl/certs/letsencrypt_CA.pem

Update all of these

12.Check the certs
Now you should find a folder /etc/dehydrated/certs/ with a folder for each of your domain set in your domains.txt file.
In each of those folders, you’ll find two important symbolic links that you need to use in all your application that rely on that certificate-key pair.
fullchain.pem :  /etc/dehydrated/certs/
privkey.pem :  /etc/dehydrated/certs/
The first one is your certificate will the different root certificates prepended to it, in other words, the one you need to set for your service. The second one is the private key of the certificate.

1. Verify the key-cert pairs (If more than one hash is displayed, they don't match):
# (openssl x509 -noout -modulus -in /etc/apache2/ssl.crt/ | openssl md5;openssl rsa -noout -modulus -in /etc/apache2/ssl.key/example1.com_key.pem | openssl md5) | uniq
(stdin)= 6b93be29e02041403ad7b0903bd8acc4

2. Verify the chain
# openssl verify -verbose -CAfile example1.com_chain.pem OK

3. Check the dates
# openssl x509 -startdate -enddate -noout -in
notBefore=Jan 23 20:17:30
2017 GMT
notAfter=Jan 23 20:17:30
2020 GMT

13. Go to production

Reverse the Staging in /etc/dehydrated/config and run
dehydrated -c -x -f /etc/dehydrated/config

Restart the server:
     service apache2 restart
and check your site(s) in a browser.
https:// is working!

So much for "automatic". But easy when you know how. We all now do. :-)

1 comment:

Doug Dodge said...

Rick Strahl also posted a great article on Let's Encrypt a little while back:

Rick is a great guy as well as being more than generous with his findings.