The sites immediately failed SSL:
Chromium reported
NET::ERR_CERT_AUTHORITY_INVALID
Firefox reported
SEC_ERROR_REVOKED_CERTIFICATE
Qualys
https://www.ssllabs.com/ssltest/analyze.html
reported
Trusted No NOT TRUSTED
It turns out that Apple, Mozilla, Google no longer trust StartCom:
https://linustechtips.com/main/topic/688200-apple-google-and-mozilla-disavow-wosign-and-startcom-certificates/
https://serverfault.com/questions/829298/my-certificate-issued-by-startssl-is-not-accepted-by-my-clients.
so the CA (Certifying Authority) cert(ificate) was not being accepted.
So, what to do? We decided to go with Let's Encrypt:
https://letsencrypt.org/
Easy when you know how, except we didn't. But we do now. After working through this tutorial you shall also. This post provides a top level of the gotchas we found along the path.
==========
The system is a bit short on concise documentation (hence this post) but there is a helpful wiki:
https://community.letsencrypt.org/latest
and, as always, Google is your friend.
First, Let's Encrypt is a system put in place by the EFF (Electronic Frontier Foundation) to provide free TLS/SSL certificates:
https://letsencrypt.org/
https://letsencrypt.org/how-it-works/
The objective of Let’s Encrypt and the ACME protocol is to make it possible to set up an HTTPS server and have it automatically obtain a browser-trusted certificate, without any human intervention. This is accomplished by running a certificate management agent on the web server.
The agent operates under the ACME (Automatic Certificate Management Environment) protocol. Any number of clients for the ACME server exist: the protocol is open source, so anyone can write a client for it and many have.
letsencrypt.org recommends we use their “official” client CertBot:
https://certbot.eff.org/about/
Certbot is an easy-to-use automatic client that fetches and deploys SSL/TLS certificates for your webserver. Certbot was developed by EFF.
Certbot is part of EFF’s larger effort to encrypt the entire Internet. Websites need to use HTTPS to secure the web.
Except that we were unable to get it to work under openSUSE. It is not in the normal repositories. We found several ymp packages but all failed either to install or installed but failed to run:
https://software.opensuse.org/package/certbot fails to run after installation.
https://rootco.de/2016-05-16-letsencrypt-on-leap/ clones a git installation. It runs, but only checks its installation:
certbot-auto certonly --webroot -w /srv/www/htdocs/gvhl \ -d genietvanhetleven.org
Bootstrapping dependencies for openSUSE-based OSes...
Loading repository data...
Reading installed packages..
xxx is already installed...
Nothing to do.https://letsencrypt.org/docs/client-options/ lists a huge number of alternative. But which?
https://community.letsencrypt.org/t/cerbot-failing-when-installing-python-packages/22722 suggests GetSSL:
GetSSL is designed so you can also run it from a different server, providing you have SSH / SFTP to the server you want to place the certs on if needed, for servers / devices where you couldn't run most other clients.
Ultimately we went with Dehydrated:
https://github.com/lukas2511/dehydrated
We found
https://software.opensuse.org/download.html?project=security&package=dehydrated
and it just runs. But it needs configuration.
Configuring Dehydrated
To begin please note that, unlike StartCom, CertBot, and GetSSL, you need to install and run Dehydrated on the target server machine, not a proxy.These are the steps:
1. Install Dehydrated
2. Set the Staging
3. Create domains.txt
4. Create well-known
5. Create /etc/dehydrated/config
6. Configure Apache2
7. Start it:
# dehydrated -c -f /etc/dehydrated/configNow: Success!
8. Reverse the Staging and reexecute
9. Get the CA certs
10. Link the
/etc/dehydrated/certs files to /etc/apache2/ssl.crt and ...key11. Amend the apache
.conf files to address these.12. Now check the certs.
13. Go to production.
1. Install
openSUSE installation requires that you add the repository with yast
on the server:
http://download.opensuse.org/repositories/security/openSUSE_Leap_42.1/Then use
yast sw_single to install: dehydrated
dehydrated-apache2. Staging
https://github.com/lukas2511/dehydrated/blob/master/docs/staging.mdLet’s Encrypt has stringent rate limits in place.
If you start testing using the production endpoint (which is the default), you will quickly hit these limits and find yourself locked out.
To avoid this, please set the CA property to the Let’s Encrypt staging server URL in your
config file:CA="https://acme-staging.api.letsencrypt.org/directory"
CA_TERMS="https://acme-staging.api.letsencrypt.org/terms"https://community.letsencrypt.org/t/cn-fake-le-intermediate-x1/13437
That means you issued those certificates against the staging server (possibly with --test-cert or --dry-run). You should reissue against the production server by removing those flags from your client invocation, if present.
Once you've gotten things working mostly, comment out these to get actual (not fake) certs:
https://calomel.org/lets_encrypt_client.html
Once your testing is done against the test certificate authority server (happy hacker fake CA) you need to edit the script to point to the production certificate authority server to generate a valid certificate. Edit the script and look for the variable at the top called "CA=" and uncomment out the "official server" and comment the "testing server". For example:
# The Lets Encrypt certificate authority URL
#CA="https://acme-staging.api.letsencrypt.org" # testing server, high rate limits. "happy hacker fake CA"
CA="https://acme-v01.api.letsencrypt.org" # official server, rate limited to 5 certs per 7 days
DO NOT delete the staging certs: it will break apache! Just let letsencrypt replace them when you go to production.
After all the testing is done you will need to run it with the -x option, since you will have created test certificates and it will refuse to regenerate actual certificates unless forced to do so:
https://community.letsencrypt.org/t/this-is-not-going-well/27366/4
dehydrated -c -x -f /etc/dehydrated/config3. domains.txt
https://github.com/lukas2511/dehydrated/blob/master/docs/domains_txt.mdCreate /etc/dehydrated/
domains.txt on the server:Dehydrated uses the file
domains.txt as a configuration file for which certificates should be requested./etc/dehydrated/domains.txt: example1.com www.example1.com
example2.net www.example2.net4. well-known
https://github.com/lukas2511/dehydrated/blob/master/docs/wellknown.mdOn the server machine:
# cd /var # mkdir www # mkdir www/dehydrated # mkdir www/dehydrated/.well-known # mkdir www/dehydrated/.well-known/acme-challenge # touch www/dehydrated/.well-known/acme-challenge/m4g1C-t0k3nChange the owner and group
# cd /var/www/dehydrated# chown -R wwwrun:www .well-known
Change the mod from 0755 to 0775
# chmod -R 0775 .well-known
Copy it to each
# cp -R .well-known /srv/www/htdocs/
# cp -R .well-known /srv/www/htdocs/gvhl# cp -R .well-known /srv/www/htdocs/nptbeyond# cp -R .well-known /srv/www/htdocs/privustech# cp -R .well-known /srv/www/htdocs/truthcourage/etc/dehydrated/config:
WELLKNOWN=/var/www/dehydrated5. config file
https://www.aaflalo.me/2016/09/dehydrated-bash-client-lets-encrypt/
/etc/dehydrated/configTwo values must be uncommented and corrected:
# E-mail to use during the registration (default: )
CONTACT_EMAIL=alavarre@gmail.com 6. Configure Apache2
We must add adehydrated.conf file in /etc/apache2/sysconfig.d:Edit
/etc/apache2/httpd.conf to addInclude /etc/apache2/sysconfig.d/dehydrated.confCreate
/etc/apache2/sysconfig.d/dehydrated.conf to read:Alias /.well-known/acme-challenge /var/www/dehydrated
Options NoneAllowOverride None# Apache 2.x
Order allow,denyAllow from all# Apache 2.4
Require all grantedThe #Apache 2.x section is deprecated so we delete it.
Restart the server.
7. Start it
The first time you run it it will be in theStaging mode.# dehydrated -c -f /etc/dehydrated/configNow: Success!
8. Reverse the Staging and reexecute
9. Get the CA certs
LetsEncrypt keeps their root offline, so we need to use the active Intermediate as the CA cert. But which? They have a number of different intermediate certs:https://letsencrypt.org/certificates/
Our intermediate “Let’s Encrypt Authority X3” represents a single public/private key pair. The private key of that pair generates the signature for all end-entity certificates (also known as leaf certificates), i.e. the certificates we issue for use on your server.
The one that works for us is the cross-signed cert:
lets-encrypt-x3-cross-signed.pem
https://letsencrypt.org/certs/lets-encrypt-x3-cross-signed.pem
So we would expect that cert to verify the leaf cert:
lavarre:/etc/apache2/ssl.crt # openssl verify -verbose -CAfile lets-encrypt-x3-cross-signed.pem genietvanhetleven.org.pem
genietvanhetleven.org.pem: OK And indeed, it does. Woohoo!
10. Links
Link the/etc/dehydrated/certs files to /etc/apache2/ssl.crt and ...keyLink to the CA cert:
# cd /etc/apache2/ssl.crt
# ln -s lets-encrypt-x3-cross-signed.pem letsencrypt_CA.pemLink to the Domain Certificates and Keys Link to each
/etc/apache2/ssl.crt.Link to each
/etc/apache2/ssl.key.We wrote a BASH script tldr_link that automates this:
for DOMAIN_TLDR in {example1.com,example2.net}
do
ln -s /etc/dehydrated/certs/$DOMAIN_TLDR/chain.pem $DOMAIN_TLDR_chain.pem
ln -s /etc/dehydrated/certs/$DOMAIN_TLDR/fullchain.pem $DOMAIN_TLDR_fullchain.pem
cd /etc/apache2/ssl.key
ln -s /etc/dehydrated/certs/$DOMAIN_TLDR/privkey.pem $DOMAIN_TLDR_key.pem
done
11. Amend the apache.conf files
Amend the apache.conf files to address these. entities.We check configuration files. Different files have different server names CA cert files. So we make them all point to
ServerName privustech.com:993
ServerAdmin alavarre@privustech.com
SSLCACertificatePath /etc/ssl/certs
SSLCACertificateFile /etc/ssl/certs/letsencrypt_CA.pemUpdate all of these
/etc/apache2/default-server.conf
/etc/apache2/default-vhost-ssl.conf
/etc/apache2/default-vhost.conf
/etc/apache2/vhost.conf
12.Check the certs
https://www.aaflalo.me/2016/09/dehydrated-bash-client-lets-encrypt/Now you should find a folder /etc/dehydrated/certs/ with a folder for each of your domain set in your domains.txt file.
In each of those folders, you’ll find two important symbolic links that you need to use in all your application that rely on that certificate-key pair.
fullchain.pem : /etc/dehydrated/certs/example.com/fullchain.pem
privkey.pem : /etc/dehydrated/certs/example.com/privkey.pem
The first one is your certificate will the different root certificates prepended to it, in other words, the one you need to set for your service. The second one is the private key of the certificate.
1. Verify the key-cert pairs (If more than one hash is displayed, they don't match):
# (openssl x509 -noout -modulus -in /etc/apache2/ssl.crt/example1.com.pem | openssl md5;openssl rsa -noout -modulus -in /etc/apache2/ssl.key/example1.com_key.pem | openssl md5) | uniq
(stdin)= 6b93be29e02041403ad7b0903bd8acc42. Verify the chain
# openssl verify -verbose -CAfile example1.com_chain.pem example1.com.pem
example1.com.pem: OK3. Check the dates
#
openssl x509 -startdate -enddate -noout -in example1.com.pem
notBefore=Jan 23 20:17:30 2017 GMT
notAfter=Jan 23 20:17:30 2020 GMT13. Go to production
Reverse the Staging in/etc/dehydrated/config and rundehydrated -c -x -f /etc/dehydrated/configRestart the server:
service apache2 restartand check your site(s) in a browser. https:// is working!
So much for "automatic". But easy when you know how. We all now do. :-)
