Manage server certificates via ACME

Entitled persons can use the WWU IT portal to set up ACME IDs that can be used on one or more servers to obtain organization-validated X.509 certificates for TLS servers and/or TLS clients.

Entitled is whoever is listed in the computer database of the university of Münster as the responsible person or administrator of the relevant server and who has logged in to the WWU IT portal with his or her personal digital ID.

The administration of ACME IDs can be found in the WWU IT portal under “Digital ID (certificate)”.

Please see to the detailed instructions in the WWU IT portal.

When you set up an ACME ID, you receive an key ID, an HMAC key, and the URL of the certificate server. You must then instruct your ACME-enabled software to register with the certificate server using this key data.

During this registration, key material is exchanged between your ACME-enabled software and the certificate server, which the software can then use to request and collect the desired certificates.

Key ID and HMAC key are deleted by the certificate server after successful registration, so they can be used only once. However, it is possible to copy the data stored by the ACME-enabled software to another server after registration.

Example: ACME with CertBot under CentOS 7

The following notes describe how we have tried it successfully.

There are many alternatives that are at least as good.

Abstract

We have successfully set up a server with the following commands:

yum install certbot

certbot register --server 'https://acme.sectigo.com/v2/OV' --eab-kid 'key-id' --eab-hmac-key 'hmac-key' --email 'email-address' --agree-tos --no-eff-email

certbot certonly --standalone --agree-tos --server https://acme.sectigo.com/v2/OV --domain fqdn1,fqdn2,... --cert-name 'certname' --email='email-address' --deploy-hook 'systemctl restart servicename'

setfacl -R -m u:username:rX /etc/letsencrypt/{live,archive}/certname

setfacl -m u:username:rX /etc/letsencrypt/{live,archive}

setfacl -m d:u:username:rX /etc/letsencrypt/{live,archive}/certname

systemctl enable certbot-renew.timer

systemctl start certbot-renew.timer

With:

  • key-id = the (shorter) key ID displayed by the IT portal

  • hmac-key = the (shorter) HMAC key displayed by the IT portal

  • email-address = the email address to which the CertBot sends its information

  • fqdn1,fqdn2,... = the fully qualified computer names (FQDNs) under which the service is found; these FQDNs must have been specified in the IT portal for the ACME ID; the FQDN specified first is also included in the subject of the certificate, the others only in the subject alternative names

  • certname = the freely choosable file name (without path) of the certificate

  • servicename = the name under which the service is managed by the system daemon

  • username = the username under which the server runs

Setting up the ACME ID

Validation is performed with External Account Binding (EAB).

To do this, you must first log in to the IT portal with your personal digital ID (https://xsso.uni-muenster.de/IT-Portal/) and set up an ACME ID there under “Digital ID (certificate)” | “New ACME ID ‘TCS’” for the fully qualified domain names (FQDNs) to be included in the future certificates. (The list of FQDNs of an ACME ID can be changed later in the IT portal).

This provides key data in the form of an EAB key ID and an EAB HMAC key, which are required for registering the server.

Installation of the software

The EPEL repository contains the CertBot software, this can be installed with:

yum install certbot

Registration of the CertBot

To register the server with the ACME service, the following command must be entered once. This contains the key ID and HMAC key for external account binding as well as the email address to be informed in case of problems:

certbot register --server 'https://acme.sectigo.com/v2/OV' --eab-kid 'key-id' --eab-hmac-key 'hmac-key' --email 'email-address' --agree-tos --no-eff-email

With:

  • key-id = the (shorter) key ID displayed by the IT portal

  • hmac-key = the (shorter) HMAC key displayed by the IT portal

  • email-address = the email address to which the CertBot sends its information

After successful registration, the key ID and HMAC key are deleted from the ACME server and can no longer be used.

However, if an ACME ID is to be used on multiple servers, the CertBot directory /etc/letsencrypt can be copied to the additional servers.

Generate certificate

In order to actually request a certificate via ACME, collect it, install it and restart the certified service at the same time, the following command is used:

certbot certonly --standalone --agree-tos --server https://acme.sectigo.com/v2/OV --domain fqdn1,fqdn2,... --cert-name 'certname' --email='email-address' --deploy-hook 'systemctl restart servicename'

With:

  • fqdn1,fqdn2,... = the fully qualified computer names (FQDNs) under which the service is found; these FQDNs must have been specified in the IT portal for the ACME ID; the FQDN specified first is also included in the subject of the certificate, the others only in the subject alternative names

  • certname = the freely choosable file name (without path) of the certificate

  • email-address = the email address to which the CertBot sends its information

  • servicename = the name under which the service is managed by the system daemon

This command places all required files in the /etc/letsencrypt/archive/certname/ directory tree and ensures that the following symbolic links always point to the currently active files:

  • /etc/letsencrypt/live/certname/privkey.pem

  • /etc/letsencrypt/live/certname/cert.pem

  • /etc/letsencrypt/live/certname/chain.pem

  • /etc/letsencrypt/live/certname/fullchain.pem

The service should of course be configured to use exactly these symbolic links to access the files and to restart with the command specified with --deploy-hook.

Instead of using these paths, CertBot could also be automatically built into services (e.g. with --apache or --nginx instead of --standalone), but this requires plugins.

Access permissions to the certificate files

CertBot stores the certificate files, especially the private key, with access rights only for the root user (0700). So that applications that do not run as root but under a different identifier are also allowed to read these certificates, the corresponding read rights must be set.

setfacl -R -m u:username:rX /etc/letsencrypt/{live,archive}/certname

setfacl -m u:username:rX /etc/letsencrypt/{live,archive}

setfacl -m d:u:username:rX /etc/letsencrypt/{live,archive}/certname

With:

  • username = the username under which the server runs

  • certname = the freely choosable file name (without path) of the certificate

You should definitely check if the identifier can actually read the generated private keys, for example with:

getfacl /etc/letsencrypt/archive/certname/priv*

If necessary, the ACL mask in above commands must be adjusted.

You should not edit the access rights with chown, chgrp or chmod, because this will be changed or complained about by CertBot.

A completely different approach would be to specify a script in the --deploy-hook option of the above certbot command that copies the file to another location, sets the permissions of these copies correctly with +chmod, and then restarts the server. Then, of course, the server must be configured to use these copies and not the original files.

Enable automatic certificate renewal

For this purpose only the certbot-renew timer has to be started:

systemctl enable certbot-renew.timer

systemctl start certbot-renew.timer

All generated and deposited certificates are then automatically renewed before expiry.

Multiple certificates with one ACME ID

To manage multiple certificates with the same ACME ID, select a different certname and the desired FQDNs fqdn1,fqdn2,... for each of these certificates.

The ACME ID must be enabled for all FQDNs in all certificates, but of course it is not necessary to include all FQDNs in every certificate.

For systems that span multiple servers (clusters, high availability, etc.) but are all administered by the same group of people, you can distribute the /etc/letsencrypt directory tree to the other servers after registration. Then each participating server can get its certificates under the same ACME ID.

However, as soon as different groups of people have administrative access to the various servers, separate ACME IDs should definitely be used.

Set up/change deploy-hook later on

If you specify the --deploy-hook option when retrieving a certificate for the first time (certbot certonly --standalone ...), this command/script will be called after every certificate exchange. This is an easy way to restart services automatically after a certificate change.

If this was not wanted or forgotten, there are various alternatives to realize the service restarts with CertBot:

  • add the line renew_hook = systemctl restart servicename to /etc/letsencrypt/renewal/certname.conf under [renewalparams]

  • simply repeat the certificate retrieval with --deploy-hook; this will then generate this line, but will also renew the certificate at the same time

  • call certbot renew --deploy-hook ... but this will affect all certificates and will result in this entry for all certificates

  • place an appropriate script under /etc/letsencrypt/renewal-hooks/deploy, this also affects all certificates

The latter is especially useful if you want to do more than just issue a simple command. However, --deploy-hook also accepts paths to scripts written by yourself.

If you want even more control or want to use even more complex start/stop scripts, you can also work with --pre-hook and --post-hook or with scripts in /etc/letsencrypt/renewal-hooks/pre and /etc/letsencrypt/renewal-hooks/post respectively, the latter act on all certificates.

Try certificate renewal

certbot renew --noninteractive --no-random-sleep-on-renew --force-renewal

Using with private IP addresses

In order to reach the certification server, the proxy server should be set before issuing certbot commands:

http_proxy=wwwproxy.uni-muenster.de:3128
https_proxy=wwwproxy.uni-muenster.de:3128
export http_proxy https_proxy

For the timer-controlled automatic certificate retrieval to work, the following lines should be added to the system configuration file /etc/sysconfig/certbot:

http_proxy=wwwproxy.uni-muenster.de:3128
https_proxy=wwwproxy.uni-muenster.de:3128