LetsEncrypt (60) SSL certificate problem: certificate has expired

If you're using LetsEncrypt for SSL certificates and have been paying attention, you'll know that one of their root CA certificates (DST Root CA X3) expired on 30/09/2021 and that older devices (really old devices) will have stopped working.

What might not be so clear (though is hinted at in the announcement https://letsencrypt.org/docs/dst-root-ca-x3-expiration-september-2021/) is that any system using OpenSSL 1.0.1 (which includes Debian 8 and Debian 9) will also have stopped working due to that version of OpenSSL erroring if any of the certificates in the chain have expired. So even though those systems have a valid trusted chain, they error.

The make it worse, the error isn't obvious and makes it sound like the certificate itself has expired, and will only present if OpenSSL 1.0.1 is in use (so for example, curl or file_get_contents on MacOS Mojave or Debian 9 will fail, but browsers will show the certificate is working correctly). The error from cURL is:

# curl -V
curl 7.38.0 (x86_64-pc-linux-gnu) libcurl/7.38.0 OpenSSL/1.0.1t zlib/1.2.8 libidn/1.29 libssh2/1.4.3 librtmp/2.3
# curl "https://testsite.com/"
curl: (60) SSL certificate problem: certificate has expired
More details here: http://curl.haxx.se/docs/sslcerts.html

curl performs SSL certificate verification by default, using a "bundle"
 of Certificate Authority (CA) public keys (CA certs). If the default
 bundle file isn't adequate, you can specify an alternate file
 using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
 the bundle, the certificate verification probably failed due to a
 problem with the certificate (it might be expired, or the name might
 not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
 the -k (or --insecure) option.

While Debian 8 and Debian 9 are old, they're not as old as the systems alluded to in the LetsEncrypt announcement (eg. iPhone 4).

The fix is to blacklist the expired certificate on the client server (ie. the server running the request, not the server with the SSL certificate). On Debian you can do this follows:

sed -i 's|mozilla/DST_Root_CA_X3.crt|!mozilla/DST_Root_CA_X3.crt|' /etc/ca-certificates.conf && update-ca-certificates