2

One of our raspberry pis running stretch suddenly, from one moment to the other, lost the ability to verify ssl certificates from servers. As far as I can see there were no reboots, crashes or other extraordinary happenings going on prior to that "event".

The really weird thing is, this is one Pi in a fleet of over a hundred, all with the same configurations and applications running.

This manifested in two ways: For one, the main application running on it communicating with our gateway, could not connect anymore. The gateway terminates SSL at an nginx proxy using LetsEncrypt certificates. The application runs on a JVM, and the good old not very enlightening error is thrown:

sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

First I thought we're having a truststore issue, but then I noticed that the AWS logs agent also couldn't connect to the Amazon servers anymore, with a much clearer message. From awslogs.log:

SSLError: SSL validation failed for https://logs.eu-west-1.amazonaws.com/ [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:661)

The log agent is written in python, so the issue is not limited to the JVM.

Finally, curling google runs into the same problem:
curl https://google.com
curl: (60) SSL certificate problem: unable to get local issuer certificate

Above interpretation was a fallacy. Google has moved to www.google.com a long time ago, so there is no certificate for the domain google.com. curling https://www.google.com actually works.

What does not work, however, is curling https://www.amazon.com. It appears there is some problem with amazon certificates specifically, and since our gateway domain is provided by cloudfront, there may well be a connection there.

I've started looking for possible causes of the issue, down to the raspbian level. Things I already verified:

  • The connection is fine. The pi has no problem with two-way handshakes like OpenVPN, SSH, or puppet, these all work.
  • The date and time on the Pi are correct.
  • openssl version is 1.1.0. I did an apt-get update / upgrade in case something might be outdated, but the problem persists.
  • I tried sudo apt-get -y --reinstall ca-certificates, to no avail (it didn't have any errors, either).
  • I ran openssl verify on all 4 Amazon_Root_CAs, all checked out OK (some of them seem to have multiple hashes associated though... is that normal?)
  • completely removed and reinstalled openssl
  • The pi was rebooted multiple times during all of this

As mentioned, we're observing this issue on only one of over a hundred more or less identical devices. Murphy would have it that it's the one sitting in a weather station over 2000m above sealevel, accessible by a single freight cable car that runs once per day. Physical access is... expensive. So if anybody could give me further leads on what might be wrong and what I could check, I would be very happy.

Additional information

I have been running parallel experiments with a pi at the office with the exact same configuration. I have not been able to reproduce the issue, not to fix it on the problematic pi, but there's an interesting thing I noticed. As part of my tests, I completely removed the ca-certificates package from the healthy pi, and rebooted. It didn't change its behaviour one bit, meaning everything still worked. Whatever the issue is, it seems not directly related to the ca-certificates package. Also, I can do curl https://www.amazon.com without issue on the healthy pi, while doing the same on the other one results in the error SSL certificate problem: unable to get local issuer certificate.

Here is the curl log of a healthy pi, what it looks like if the connection succeeds:

curl https://www.amazon.com -vvv
* Rebuilt URL to: https://www.amazon.com/
*   Trying 13.224.90.101...
* TCP_NODELAY set
* Connected to www.amazon.com (13.224.90.101) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=US; ST=Washington; L=Seattle; O=Amazon.com, Inc.; CN=www.amazon.com
*  start date: Jul 13 00:00:00 2020 GMT
*  expire date: Jul 10 12:00:00 2021 GMT
*  subjectAltName: host "www.amazon.com" matched cert's "www.amazon.com"
*  issuer: C=US; O=DigiCert Inc; CN=DigiCert Global CA G2
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0xf13eb8)
> GET / HTTP/1.1
> Host: www.amazon.com
> User-Agent: curl/7.52.1
> Accept: */*

And here is the log output of the same command on the pi that won't connect:

curl https://www.amazon.com -vvv
* Rebuilt URL to: https://www.amazon.com/
*   Trying 13.224.90.101...
* TCP_NODELAY set
* Connected to www.amazon.com (13.224.90.101) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS alert, Server hello (2):
* SSL certificate problem: unable to get local issuer certificate
* Curl_http_done: called premature == 1
* stopped the pause stream!
* Closing connection 0
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.haxx.se/docs/sslcerts.html

It seems like there is a second hello from the server instead of the expected key exchange, but the server is working perfectly fine for everybody else...

Kind of ironic, another server that cannot be verified is curl.haxx.se...

more weirdness The issue went away. Just like that. Hunting through the logs for explanations, I found out that this same issue had occurred sometime last week, only shorter. Then, too, it went away again by itself. Presumably, it will recur in the future. I'm completely stomped as to what might be the cause. I don't think a firewall setting could ever lead to a behaviour like this?

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.