K
Q

Debugging SSL Handshake problems of kubernetes pods with the internet

3/26/2021

I recently installed k3s on debian 10 and I am having problems connecting to the internet from inside the pods. Ping, DNS, HTTP, works but TLS is having problems with the Handshake. I made a tcpdump and see ClientHello's and then immediatly a response from the server with internal error SSL error 80. On the VM where cluster runs on everything works fine, just not inside the pods.

Example in container:

pacman -Syu
:: Synchronizing package databases...
error: failed retrieving file 'core.db' from mirror.pkgbuild.com : error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error
error: failed retrieving file 'core.db' from mirror.rackspace.com : error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error
error: failed retrieving file 'core.db' from mirror.leaseweb.net : error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error
error: failed to update core (download library error)
error: failed retrieving file 'extra.db' from mirror.pkgbuild.com : error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error
error: failed retrieving file 'extra.db' from mirror.rackspace.com : error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error
error: failed retrieving file 'extra.db' from mirror.leaseweb.net : error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error
error: failed to update extra (download library error)
error: failed retrieving file 'community.db' from mirror.pkgbuild.com : error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error
error: failed retrieving file 'community.db' from mirror.rackspace.com : error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error
error: failed retrieving file 'community.db' from mirror.leaseweb.net : error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error
error: failed to update community (download library error)
error: failed to synchronize all databases
curl -4 -v https://www.google.com
*   Trying 157.230.127.168:443...
* Connected to www.google.com (157.230.127.168) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS alert, internal error (592):
* error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error
* Closing connection 0
curl: (35) error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error

On VM:

curl -4 -v https://www.google.com
* Expire in 0 ms for 6 (transfer 0x556e74843fb0)
<snip>A lot of these expire messages</snip>
* Expire in 10 ms for 1 (transfer 0x556e74843fb0)
*   Trying 172.217.23.100...
* TCP_NODELAY set
* Expire in 200 ms for 4 (transfer 0x556e74843fb0)
* Connected to www.google.com (172.217.23.100) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=US; ST=California; L=Mountain View; O=Google LLC; CN=www.google.com
*  start date: Mar 11 15:00:19 2021 GMT
*  expire date: Jun  3 15:00:18 2021 GMT
*  subjectAltName: host "www.google.com" matched cert's "www.google.com"
*  issuer: C=US; O=Google Trust Services; CN=GTS CA 1O1
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x556e74843fb0)
> GET / HTTP/2
> Host: www.google.com
> User-Agent: curl/7.64.0
> Accept: */*
> 
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
< HTTP/2 200 
< date: Fri, 26 Mar 2021 01:32:52 GMT
< expires: -1
< cache-control: private, max-age=0
< content-type: text/html; charset=ISO-8859-1
< p3p: CP="This is not a P3P policy! See g.co/p3phelp for more info."
< server: gws
< x-xss-protection: 0
< x-frame-options: SAMEORIGIN
< set-cookie: NID=212=0Nbn1MMUvMRUNa8iBWgekw-YrWof3yeeBb22v94ZrQ4KkMeVm5wouwkK9ElA353VmZCp_TOXah6KfC_KQX7S48W-IgXlQUz1z4ytqKnDzSXM7X40rLw8tBwMi4oH7eLyE4nGGAfsSlUne28lwMNLobMxYls54iUbgM4x9-kuMCA; expires=Sat, 25-Sep-2021 01:32:52 GMT; path=/; domain=.google.com; HttpOnly
< alt-svc: h3-29=":443"; ma=2592000,h3-T051=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"
< accept-ranges: none
< vary: Accept-Encoding
<hmtl>...,</html>

Does anybody have an idea how I can debug this or fix it?

UPDATE

Here a curl from inside the VM and a pod. I fixed the Webserver to a known working one on the Internet (Vercel Hosted Website) aswell so we debug the same server. But it is always the same error inside the pod. no matter which server I contact.

kubectl run -i --tty --rm debug --image=archlinux --restart=Never -- bash

Pod:

[root@debug /]# curl -v --tlsv1.2 --tls-max 1.2 https://wegmueller.it/ > /dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 157.230.127.168:443...
* Connected to wegmueller.it (157.230.127.168) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: none
} [5 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
} [229 bytes data]
* TLSv1.2 (IN), TLS alert, internal error (592):
{ [2 bytes data]
* error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
* Closing connection 0
curl: (35) error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error
[root@debug /]#

VM:

curl -v --tlsv1.2 --tls-max 1.2 https://wegmueller.it/ > /dev/null
* Expire in 0 ms for 6 (transfer 0x559d896cafb0)
* Expire in 1 ms for 1 (transfer 0x559d896cafb0)
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* Expire in 0 ms for 1 (transfer 0x559d896cafb0)
* Expire in 1 ms for 1 (transfer 0x559d896cafb0)
<snip a ton of expire messages>
* Expire in 50 ms for 1 (transfer 0x559d896cafb0)
*   Trying 76.76.21.21...
* TCP_NODELAY set
* Expire in 200 ms for 4 (transfer 0x559d896cafb0)
* Connected to wegmueller.it (76.76.21.21) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs
} [5 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
} [223 bytes data]
* TLSv1.2 (IN), TLS handshake, Server hello (2):
{ [106 bytes data]
* TLSv1.2 (IN), TLS handshake, Certificate (11):
{ [2458 bytes data]
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
{ [300 bytes data]
* TLSv1.2 (IN), TLS handshake, Server finished (14):
{ [4 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
} [37 bytes data]
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.2 (OUT), TLS handshake, Finished (20):
} [16 bytes data]
* TLSv1.2 (IN), TLS handshake, Finished (20):
{ [16 bytes data]
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=wegmueller.it
*  start date: Mar 16 21:08:10 2021 GMT
*  expire date: Jun 14 21:08:10 2021 GMT
*  subjectAltName: host "wegmueller.it" matched cert's "wegmueller.it"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
} [5 bytes data]
* Using Stream ID: 1 (easy handle 0x559d896cafb0)
} [5 bytes data]
> GET / HTTP/2
> Host: wegmueller.it
> User-Agent: curl/7.64.0
> Accept: */*
> 
{ [5 bytes data]
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
} [5 bytes data]
< HTTP/2 200 
< date: Sun, 28 Mar 2021 15:24:59 GMT
< content-type: text/html; charset=utf-8
< content-disposition: inline; filename="index"
< cache-control: public, max-age=0, must-revalidate
< content-length: 3154
< access-control-allow-origin: *
< etag: W/"637eb49039bc1f6a7aa9e4d88a56b0708b722983f403c3c2f717371b2b25a472"
< accept-ranges: bytes
< x-vercel-cache: MISS
< age: 0
< server: Vercel
< x-vercel-id: cdg1::z6h9l-1616945098788-71384c5744be
< strict-transport-security: max-age=63072000
< 
{ [1112 bytes data]
100  3154  100  3154    0     0   3604      0 --:--:-- --:--:-- --:--:--  3600
* Connection #0 to host wegmueller.it left intact
-- Toasterson
k3s
kubernetes
kubernetes-networking
ssl

Similar Questions

How to make http call to an application running inside a kubernetes pod from outside?
Access service api exposed as clusterIP in kubernete
Revoking SSL certificate in Mutual TLS authentication
What's the benefit of using cni instead of static route table in k8s
How does a User get its kubeconfig without access to the Rancher GUI

1 Answer

3/26/2021
  • error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error

openssl error code contains 3 parts.<br>

  • First 8 bits is module serial number. 14(HEX) means ssl module which has serial number 20.
  • The next 12 bits is function SN. 094(HEX) is SSL_SRP_CTX_init func with SN 148(DEC).
  • The next 12 bits 438(HEX) is error reason SSL_R_TLSV1_ALERT_INTERNAL_ERROR with 1080(DEC).

To sum up,code 14094438 means that server error occurred when it tried to initiate SSL context.

But what's the reason?

client side

client side send CLIENT HELLO with its tls version, ciphers suite and etc to server side. what's the difference of 2 CLIENT HELLO packets on pods or VM? This need to be clear.<br> Here you use newest version tls 1.3. For more infomation, you may downgrade client version. curl has options to control the TLS version used. if you want to specify that TLS 1.2 is used but not 1.1 or 1.3 etc, you need something like

curl --tlsv1.2 --tls-max 1.2 ...

server side

I also noticed that www.google.com (157.230.127.168) is different from www.google.com (172.217.23.100). Does server side's difference lead to this situation? We can confirm with sending a curl to the same IP.

I assume either you and/or the server uses an unmatched openssl library.But before drawing conclusion, we need more debug info.

-- mariolu
Source: StackOverflow