Easy when you know how: Let's Encrypt open source "automated" TLS/SSL certificates and keys

We recently renewed our StartCom certs for our vhost domains.

The sites immediately failed SSL:
     Chromium reported
          NET::ERR_CERT_AUTHORITY_INVALID
     Firefox reported
          SEC_ERROR_REVOKED_CERTIFICATE
     Qualys
          https://www.ssllabs.com/ssltest/analyze.html
          reported 
          Trusted No NOT TRUSTED

It turns out that Apple, Mozilla, Google no longer trust StartCom:
https://linustechtips.com/main/topic/688200-apple-google-and-mozilla-disavow-wosign-and-startcom-certificates/
https://serverfault.com/questions/829298/my-certificate-issued-by-startssl-is-not-accepted-by-my-clients.
so the CA (Certifying Authority) cert(ificate) was not being accepted.

So, what to do? We decided to go with Let's Encrypt:
     https://letsencrypt.org/

Easy when you know how, except we didn't. But we do now. After working through this tutorial you shall also. This post provides a top level of the gotchas we found along the path.

==========

The system is a bit short on concise documentation (hence this post) but there is a helpful wiki:
     https://community.letsencrypt.org/latest
and, as always, Google is your friend. 

First, Let's Encrypt is a system put in place by the EFF (Electronic Frontier  Foundation) to provide free TLS/SSL certificates:
     https://letsencrypt.org/
     https://letsencrypt.org/how-it-works/

The objective of Let’s Encrypt and the ACME protocol is to make it possible to set up an HTTPS server and have it automatically obtain a browser-trusted certificate, without any human intervention. This is accomplished by running a certificate management agent on the web server.

The agent operates under the ACME (Automatic Certificate Management Environment) protocol. Any number of clients for the ACME server exist: the protocol is open source, so anyone can write a client for it and many have.

letsencrypt.org recommends we use their “official” client CertBot:
https://certbot.eff.org/about/
Certbot is an easy-to-use automatic client that fetches and deploys SSL/TLS certificates for your webserver. Certbot was developed by EFF.
Certbot is part of EFF’s larger effort to encrypt the entire Internet. Websites need to use HTTPS to secure the web.

Except that we were unable to get it to work under openSUSE. It is not in the normal repositories. We found several ymp packages but all failed either to install or installed but failed to run:
     https://software.opensuse.org/package/certbot fails to run after installation.
     https://rootco.de/2016-05-16-letsencrypt-on-leap/ clones a git installation. It runs, but only checks its installation:
     certbot-auto certonly --webroot -w /srv/www/htdocs/gvhl \ -d genietvanhetleven.org
Bootstrapping dependencies for openSUSE-based OSes...
Loading repository data...
Reading installed packages..
xxx is already installed...
Nothing to do.

     https://letsencrypt.org/docs/client-options/ lists a huge number of alternative. But which?

     https://community.letsencrypt.org/t/cerbot-failing-when-installing-python-packages/22722 suggests GetSSL:
     GetSSL is designed so you can also run it from a different server, providing you have SSH / SFTP to the server you want to place the certs on if needed, for servers / devices where you couldn't run most other clients.

Ultimately we went with Dehydrated:
     https://github.com/lukas2511/dehydrated
We found 
     https://software.opensuse.org/download.html?project=security&package=dehydrated
and it just runs. But it needs configuration.

Configuring Dehydrated

To begin please note that, unlike StartCom, CertBot, and GetSSL, you need to install and run Dehydrated on the target server machine, not a proxy.

These are the steps:
1. Install Dehydrated
2. Set the Staging
3. Create domains.txt
4. Create well-known
5. Create /etc/dehydrated/config
6. Configure Apache2
7. Start it:
     # dehydrated -c -f /etc/dehydrated/config
Now: Success!
8. Reverse the Staging and reexecute
9. Get the CA certs
10. Link the /etc/dehydrated/certs files to /etc/apache2/ssl.crt and ...key
11. Amend the apache .conf files to address these.
12. Now check the certs.
13. Go to production.

1. Install

openSUSE installation requires that you add the repository with yast
on the server:

http://download.opensuse.org/repositories/security/openSUSE_Leap_42.1/

Then use yast sw_single to install:
     dehydrated
     dehydrated-apache

2. Staging

https://github.com/lukas2511/dehydrated/blob/master/docs/staging.md
Let’s Encrypt has stringent rate limits in place.
If you start testing using the production endpoint (which is the default), you will quickly hit these limits and find yourself locked out.
To avoid this, please set the CA property to the Let’s Encrypt staging server URL in your config file:
CA="https://acme-staging.api.letsencrypt.org/directory"
CA_TERMS="https://acme-staging.api.letsencrypt.org/terms"

https://community.letsencrypt.org/t/cn-fake-le-intermediate-x1/13437
That means you issued those certificates against the staging server (possibly with --test-cert or --dry-run). You should reissue against the production server by removing those flags from your client invocation, if present.

Once you've gotten things working mostly, comment out these to get actual (not fake) certs:
https://calomel.org/lets_encrypt_client.html
Once your testing is done against the test certificate authority server (happy hacker fake CA) you need to edit the script to point to the production certificate authority server to generate a valid certificate. Edit the script and look for the variable at the top called "CA=" and uncomment out the "official server" and comment the "testing server". For example:
# The Lets Encrypt certificate authority URL
#CA="https://acme-staging.api.letsencrypt.org" # testing server, high rate limits. "happy hacker fake CA"
CA="https://acme-v01.api.letsencrypt.org" # official server, rate limited to 5 certs per 7 days

DO NOT delete the staging certs: it will break apache! Just let letsencrypt replace them when you go to production.

After all the testing is done you will need to run it with the -x option, since you will have created test certificates and it will refuse to regenerate actual certificates unless forced to do so:
https://community.letsencrypt.org/t/this-is-not-going-well/27366/4
     dehydrated -c -x -f /etc/dehydrated/config

3. domains.txt

https://github.com/lukas2511/dehydrated/blob/master/docs/domains_txt.md
Create /etc/dehydrated/domains.txt on the server:
Dehydrated uses the file domains.txt as a configuration file for which certificates should be requested.
     /etc/dehydrated/domains.txt:
          example1.com www.example1.com
          example2.net www.example2.net

4. well-known

https://github.com/lukas2511/dehydrated/blob/master/docs/wellknown.md
     On the server machine:
     # cd /var     
     # mkdir www     
     # mkdir www/dehydrated
     # mkdir www/dehydrated/.well-known
     # mkdir www/dehydrated/.well-known/acme-challenge
     # touch www/dehydrated/.well-known/acme-challenge/m4g1C-t0k3n

Change the owner and group
     # cd /var/www/dehydrated
     # chown -R wwwrun:www .well-known

Change the mod from 0755 to 0775
     # chmod -R 0775 .well-known

Copy it to each :
     # cp -R .well-known /srv/www/htdocs/
     # cp -R .well-known /srv/www/htdocs/gvhl
     # cp -R .well-known /srv/www/htdocs/nptbeyond
     # cp -R .well-known /srv/www/htdocs/privustech
     # cp -R .well-known /srv/www/htdocs/truthcourage
/etc/dehydrated/config:
     WELLKNOWN=/var/www/dehydrated

5. config file

https://www.aaflalo.me/2016/09/dehydrated-bash-client-lets-encrypt/
     /etc/dehydrated/config is mostly commented out, showing the default values.

Two values must be uncommented and corrected:
     # E-mail to use during the registration (default: )
          CONTACT_EMAIL=alavarre@gmail.com
      HOOK=/etc/dehydrated/hook.sh

6. Configure Apache2

We must add a dehydrated.conf file in /etc/apache2/sysconfig.d:
Edit /etc/apache2/httpd.conf to add
     Include /etc/apache2/sysconfig.d/dehydrated.conf

Create /etc/apache2/sysconfig.d/dehydrated.conf to read:
     Alias /.well-known/acme-challenge /var/www/dehydrated     
     Options None
     AllowOverride None
     # Apache 2.x     
     Order allow,deny
     Allow from all
     # Apache 2.4     
     Require all granted

The #Apache 2.x section is deprecated so we delete it.

Restart the server.

7. Start it

The first time you run it it will be in the Staging mode.
     # dehydrated -c -f /etc/dehydrated/config
Now: Success!

8. Reverse the Staging and reexecute

9. Get the CA certs

LetsEncrypt keeps their root offline, so we need to use the active Intermediate as the CA cert. But which? They have a number of different intermediate certs:
     https://letsencrypt.org/certificates/
     Our intermediate “Let’s Encrypt Authority X3” represents a single public/private key pair. The private key of that pair generates the signature for all end-entity certificates (also known as leaf certificates), i.e. the certificates we issue for use on your server.

The one that works for us is the cross-signed cert:
lets-encrypt-x3-cross-signed.pem
https://letsencrypt.org/certs/lets-encrypt-x3-cross-signed.pem

So we would expect that cert to verify the leaf cert:
lavarre:/etc/apache2/ssl.crt # openssl verify -verbose -CAfile lets-encrypt-x3-cross-signed.pem genietvanhetleven.org.pem
genietvanhetleven.org.pem: OK 
And indeed, it does. Woohoo!

10. Links

Link the /etc/dehydrated/certs files to /etc/apache2/ssl.crt and ...key

Link to the CA cert:
# cd /etc/apache2/ssl.crt
# ln -s lets-encrypt-x3-cross-signed.pem letsencrypt_CA.pem

Link to the Domain Certificates and Keys Link to each cert in /etc/apache2/ssl.crt.
Link to each key in /etc/apache2/ssl.key.
We wrote a BASH script tldr_link that automates this:
     for DOMAIN_TLDR in {example1.com,example2.net}
     do
          cd /etc/apache2/ssl.crt
      ln -s /etc/dehydrated/certs/$DOMAIN_TLDR/cet.pem $DOMAIN_TLDR.pem
     ln -s /etc/dehydrated/certs/$DOMAIN_TLDR/chain.pem $DOMAIN_TLDR_chain.pem
     ln -s /etc/dehydrated/certs/$DOMAIN_TLDR/fullchain.pem $DOMAIN_TLDR_fullchain.pem
     cd /etc/apache2/ssl.key
     ln -s /etc/dehydrated/certs/$DOMAIN_TLDR/privkey.pem $DOMAIN_TLDR_key.pem
     done



11. Amend the apache.conf files

Amend the apache .conf files to address these. entities.

We check configuration files. Different files have different server names CA cert files. So we make them all point to
     ServerName privustech.com:993
     ServerAdmin alavarre@privustech.com
     SSLCACertificatePath /etc/ssl/certs
     SSLCACertificateFile /etc/ssl/certs/letsencrypt_CA.pem

Update all of these
     /etc/apache2/default-server.conf
     /etc/apache2/default-vhost-ssl.conf
     /etc/apache2/default-vhost.conf
     /etc/apache2/vhost.conf



12.Check the certs

https://www.aaflalo.me/2016/09/dehydrated-bash-client-lets-encrypt/
Now you should find a folder /etc/dehydrated/certs/ with a folder for each of your domain set in your domains.txt file.
In each of those folders, you’ll find two important symbolic links that you need to use in all your application that rely on that certificate-key pair.
fullchain.pem :  /etc/dehydrated/certs/example.com/fullchain.pem
privkey.pem :  /etc/dehydrated/certs/example.com/privkey.pem
The first one is your certificate will the different root certificates prepended to it, in other words, the one you need to set for your service. The second one is the private key of the certificate.
1. Verify the key-cert pairs (If more than one hash is displayed, they don't match):
# (openssl x509 -noout -modulus -in /etc/apache2/ssl.crt/example1.com.pem | openssl md5;openssl rsa -noout -modulus -in /etc/apache2/ssl.key/example1.com_key.pem | openssl md5) | uniq
(stdin)= 6b93be29e02041403ad7b0903bd8acc4

2. Verify the chain
# openssl verify -verbose -CAfile example1.com_chain.pem example1.com.pem
example1.com.pem: OK

3. Check the dates
# openssl x509 -startdate -enddate -noout -in example1.com.pem
notBefore=Jan 23 20:17:30 2017 GMT
notAfter=Jan 23 20:17:30 2020 GMT

13. Go to production

Reverse the Staging in /etc/dehydrated/config and run
dehydrated -c -x -f /etc/dehydrated/config

Restart the server:
     service apache2 restart
and check your site(s) in a browser. https:// is working!

So much for "automatic". But easy when you know how. We all now do. :-)