I'm attempting to update from Traefik v1.2.3 to v1.3.6 on Kubernetes. I have my TLS certificates mounted inside of the pods from secrets. Under v1.2.3, everything works as expected. When I try to apply my v1.3.6 deployment (only change being the new docker image), the pods fail to start with the following message:
time="2017-08-22T20:27:44Z" level=error msg="Error creating TLS config: tls: failed to find any PEM data in key input"
time="2017-08-22T20:27:44Z" level=fatal msg="Error preparing server: tls: failed to find any PEM data in key input"
Below is my traefik.toml file:
defaultEntryPoints = ["http","https"]
[entryPoints]
[entryPoints.http]
address = ":80"
[entryPoints.http.redirect]
entryPoint = "https"
[entryPoints.https]
address = ":443"
[entryPoints.https.tls]
[[entryPoints.https.tls.certificates]]
CertFile = "/ssl/wildcard.foo.mydomain.com.crt"
KeyFile = "/ssl/wildcard.foo.mydomain.com.key"
[[entryPoints.https.tls.certificates]]
CertFile = "/ssl/wildcard.mydomain.com.crt"
KeyFile = "/ssl/wildcard.mydomain.com.key"
[[entryPoints.https.tls.certificates]]
CertFile = "/ssl/wildcard.local.crt"
KeyFile = "/ssl/wildcard.local.key"
[kubernetes]
labelselector = "expose=internal"
My initial impression of the errors produced by the pods are that the keys in the secret are not valid. However, I am able to base64 decode the contents of the secret and see that the values match those of the certificate files I have stored locally. Additionally, I would expect to see this error on any version of Traefik if these were in fact, invalid. In reviewing the change log for Traefik, I see that the SSL library was updated but the related PR indicates that this only added ciphers and did not remove any previously supported.
:Edit w/ additional info:
Running with --logLevel=DEBUG
provides this additional information (provided in full below in case it's helpful):
[cluster-traefik-2693375319-w67hf] time="2017-08-22T21:41:19Z" level=debug msg="Global configuration loaded {"GraceTimeOut":10000000000,"Debug":false,"CheckNewVersion":true,"AccessLogsFile":"","TraefikLogsFile":"","LogLevel":"DEBUG","EntryPoints":{"http":{"Network":"","Address":":80","TLS":null,"Redirect":{"EntryPoint":"https","Regex":"","Replacement":""},"Auth":null,"Compress":false},"https":{"Network":"","Address":":443","TLS":{"MinVersion":"","CipherSuites":null,"Certificates":[{"CertFile":"/ssl/wildcard.foo.mydomain.com.crt","KeyFile":"/ssl/wildcard.foo.mydomain.com.key"},{"CertFile":"/ssl/wildcard.mydomain.com.crt","KeyFile":"/ssl/wildcard.mydomain.com.key"},{"CertFile":"/ssl/wildcard.local.crt","KeyFile":"/ssl/wildcard.local.key"}],"ClientCAFiles":null},"Redirect":null,"Auth":null,"Compress":false}},"Cluster":null,"Constraints":[],"ACME":null,"DefaultEntryPoints":["http","https"],"ProvidersThrottleDuration":2000000000,"MaxIdleConnsPerHost":200,"IdleTimeout":180000000000,"InsecureSkipVerify":false,"Retry":null,"HealthCheck":{"Interval":30000000000},"Docker":null,"File":null,"Web":{"Address":":8080","CertFile":"","KeyFile":"","ReadOnly":false,"Statistics":null,"Metrics":{"Prometheus":{"Buckets":[0.1,0.3,1.2,5]}},"Path":"","Auth":null},"Marathon":null,"Consul":null,"ConsulCatalog":null,"Etcd":null,"Zookeeper":null,"Boltdb":null,"Kubernetes":{"Watch":true,"Filename":"","Constraints":[],"Endpoint":"","Token":"","CertAuthFilePath":"","DisablePassHostHeaders":false,"Namespaces":null,"LabelSelector":"expose=internal"},"Mesos":null,"Eureka":null,"ECS":null,"Rancher":null,"DynamoDB":null}"
[cluster-traefik-2693375319-w67hf] time="2017-08-22T21:41:19Z" level=info msg="Preparing server https &{Network: Address::443 TLS:0xc42060d800 Redirect:<nil> Auth:<nil> Compress:false}"
[cluster-traefik-2693375319-w67hf] time="2017-08-22T21:41:19Z" level=error msg="Error creating TLS config: tls: failed to find any PEM data in key input"
[cluster-traefik-2693375319-w67hf] time="2017-08-22T21:41:19Z" level=fatal msg="Error preparing server: tls: failed to find any PEM data in key input"
This issue turned out to be new validation logic in the crypto/tls library in Go 1.8. They are now validating the certificate blocks end in -----
where as before they did not. The private key for one of my certificate files ended in ----
(missing a hyphen). Adding the missing character fixed this issue.