Installing Traefik on a DigitialOcean Kubernetes cluster with Helm is failing to create an external IP address for the LoadBalancer

1/15/2019

I am trying to install Traefik on my DigitalOcean Kubernetes cluster using Helm.

$ helm install -f traefik.values.yaml stable/traefik

I own the hypothetical domain example.org and the DNS record is managed through Digital Ocean

The traefik.values.yaml values file contains (you can view the full list of options here):

---
accessLogs:
  enabled: true
dashboard:
  enabled: true
  domain: traefik.example.org
debug:
  enabled: true
ssl:
  enabled: true
  enforced: true
acme:
  enabled: true
  logging: true
  staging: true
  email: <redacted>
  challengeType: "dns-01"
  dnsProvider:
    name:  digitalocean
    digitalocean:
      DO_AUTH_TOKEN: "<redacted>"
  domains:
    enabled: true
    domainsList:
      - main: "traefik.example.org"
rbac:
  enabled: true

But the service never creates an external IP address. When I check the logs, I see:

$ k logs messy-koala-traefik-584cc9f68b-d9p6h -f
{"level":"info","msg":"Using TOML configuration file /config/traefik.toml","time":"2019-01-15T16:25:20Z"}
{"level":"info","msg":"No tls.defaultCertificate given for https: using the first item in tls.certificates as a fallback.","time":"2019-01-15T16:25:20Z"}
{"level":"info","msg":"Traefik version v1.7.6 built on 2018-12-14_06:43:37AM","time":"2019-01-15T16:25:20Z"}
{"level":"debug","msg":"Global configuration loaded {\"LifeCycle\":{\"RequestAcceptGraceTimeout\":0,\"GraceTimeOut\":10000000000},\"GraceTimeOut\":0,\"Debug\":true,\"CheckNewVersion\":true,\"SendAnonymousUsage\":false,\"AccessLogsFile\":\"\",\"AccessLog\":{\"format\":\"common\",\"fields\":{\"defaultMode\":\"keep\",\"headers\":{\"defaultMode\":\"keep\"}}},\"TraefikLogsFile\":\"\",\"TraefikLog\":{\"format\":\"json\"},\"Tracing\":null,\"LogLevel\":\"\",\"EntryPoints\":{\"http\":{\"Address\":\":80\",\"TLS\":null,\"Redirect\":{\"regex\":\"^http://(.*)\",\"replacement\":\"https://$1\"},\"Auth\":null,\"WhitelistSourceRange\":null,\"WhiteList\":null,\"Compress\":true,\"ProxyProtocol\":null,\"ForwardedHeaders\":{\"Insecure\":true,\"TrustedIPs\":null}},\"https\":{\"Address\":\":443\",\"TLS\":{\"MinVersion\":\"\",\"CipherSuites\":null,\"Certificates\":[{\"CertFile\":\"/ssl/tls.crt\",\"KeyFile\":\"/ssl/tls.key\"}],\"ClientCAFiles\":null,\"ClientCA\":{\"Files\":null,\"Optional\":false},\"DefaultCertificate\":{\"CertFile\":\"/ssl/tls.crt\",\"KeyFile\":\"/ssl/tls.key\"},\"SniStrict\":false},\"Redirect\":null,\"Auth\":null,\"WhitelistSourceRange\":null,\"WhiteList\":null,\"Compress\":true,\"ProxyProtocol\":null,\"ForwardedHeaders\":{\"Insecure\":true,\"TrustedIPs\":null}},\"traefik\":{\"Address\":\":8080\",\"TLS\":null,\"Redirect\":null,\"Auth\":null,\"WhitelistSourceRange\":null,\"WhiteList\":null,\"Compress\":false,\"ProxyProtocol\":null,\"ForwardedHeaders\":{\"Insecure\":true,\"TrustedIPs\":null}}},\"Cluster\":null,\"Constraints\":[],\"ACME\":{\"Email\":\"jeff.n.may@gmail.com\",\"Domains\":[{\"Main\":\"traefik.example.org\",\"SANs\":null}],\"Storage\":\"/acme/acme.json\",\"StorageFile\":\"\",\"OnDemand\":false,\"OnHostRule\":true,\"CAServer\":\"https://acme-staging-v02.api.letsencrypt.org/directory\",\"EntryPoint\":\"https\",\"KeyType\":\"\",\"DNSChallenge\":{\"Provider\":\"digitalocean\",\"DelayBeforeCheck\":0,\"Resolvers\":null,\"DisablePropagationCheck\":false},\"HTTPChallenge\":null,\"TLSChallenge\":null,\"DNSProvider\":\"\",\"DelayDontCheckDNS\":0,\"ACMELogging\":true,\"OverrideCertificates\":false,\"TLSConfig\":null},\"DefaultEntryPoints\":[\"http\",\"https\"],\"ProvidersThrottleDuration\":2000000000,\"MaxIdleConnsPerHost\":200,\"IdleTimeout\":0,\"InsecureSkipVerify\":false,\"RootCAs\":null,\"Retry\":null,\"HealthCheck\":{\"Interval\":30000000000},\"RespondingTimeouts\":null,\"ForwardingTimeouts\":null,\"AllowMinWeightZero\":false,\"KeepTrailingSlash\":false,\"Web\":null,\"Docker\":null,\"File\":null,\"Marathon\":null,\"Consul\":null,\"ConsulCatalog\":null,\"Etcd\":null,\"Zookeeper\":null,\"Boltdb\":null,\"Kubernetes\":{\"Watch\":true,\"Filename\":\"\",\"Constraints\":[],\"Trace\":false,\"TemplateVersion\":0,\"DebugLogGeneratedTemplate\":false,\"Endpoint\":\"\",\"Token\":\"\",\"CertAuthFilePath\":\"\",\"DisablePassHostHeaders\":false,\"EnablePassTLSCert\":false,\"Namespaces\":null,\"LabelSelector\":\"\",\"IngressClass\":\"\",\"IngressEndpoint\":null},\"Mesos\":null,\"Eureka\":null,\"ECS\":null,\"Rancher\":null,\"DynamoDB\":null,\"ServiceFabric\":null,\"Rest\":null,\"API\":{\"EntryPoint\":\"traefik\",\"Dashboard\":true,\"Debug\":true,\"CurrentConfigurations\":null,\"Statistics\":null},\"Metrics\":null,\"Ping\":null,\"HostResolver\":null}","time":"2019-01-15T16:25:20Z"}
{"level":"info","msg":"\nStats collection is disabled.\nHelp us improve Traefik by turning this feature on :)\nMore details on: https://docs.traefik.io/basics/#collected-data\n","time":"2019-01-15T16:25:20Z"}
{"level":"debug","msg":"Setting Acme Certificate store from Entrypoint: https","time":"2019-01-15T16:25:20Z"}
{"level":"debug","msg":"Add certificate for domains *.example.com","time":"2019-01-15T16:25:20Z"}
{"level":"info","msg":"Preparing server traefik \u0026{Address::8080 TLS:\u003cnil\u003e Redirect:\u003cnil\u003e Auth:\u003cnil\u003e WhitelistSourceRange:[] WhiteList:\u003cnil\u003e Compress:false ProxyProtocol:\u003cnil\u003e ForwardedHeaders:0xc0002c3120} with readTimeout=0s writeTimeout=0s idleTimeout=3m0s","time":"2019-01-15T16:25:20Z"}
{"level":"debug","msg":"Creating regex redirect http -\u003e ^http://(.*) -\u003e https://$1","time":"2019-01-15T16:25:20Z"}
{"level":"info","msg":"Preparing server http \u0026{Address::80 TLS:\u003cnil\u003e Redirect:0xc00019fdc0 Auth:\u003cnil\u003e WhitelistSourceRange:[] WhiteList:\u003cnil\u003e Compress:true ProxyProtocol:\u003cnil\u003e ForwardedHeaders:0xc0002c30c0} with readTimeout=0s writeTimeout=0s idleTimeout=3m0s","time":"2019-01-15T16:25:20Z"}
{"level":"info","msg":"Preparing server https \u0026{Address::443 TLS:0xc000221170 Redirect:\u003cnil\u003e Auth:\u003cnil\u003e WhitelistSourceRange:[] WhiteList:\u003cnil\u003e Compress:true ProxyProtocol:\u003cnil\u003e ForwardedHeaders:0xc0002c30e0} with readTimeout=0s writeTimeout=0s idleTimeout=3m0s","time":"2019-01-15T16:25:20Z"}
{"level":"debug","msg":"Add certificate for domains *.example.com","time":"2019-01-15T16:25:20Z"}
{"level":"info","msg":"Starting provider configuration.ProviderAggregator {}","time":"2019-01-15T16:25:20Z"}
{"level":"info","msg":"Starting server on :8080","time":"2019-01-15T16:25:20Z"}
{"level":"info","msg":"Starting server on :80","time":"2019-01-15T16:25:20Z"}
{"level":"info","msg":"Starting server on :443","time":"2019-01-15T16:25:20Z"}
{"level":"info","msg":"Starting provider *kubernetes.Provider {\"Watch\":true,\"Filename\":\"\",\"Constraints\":[],\"Trace\":false,\"TemplateVersion\":0,\"DebugLogGeneratedTemplate\":false,\"Endpoint\":\"\",\"Token\":\"\",\"CertAuthFilePath\":\"\",\"DisablePassHostHeaders\":false,\"EnablePassTLSCert\":false,\"Namespaces\":null,\"LabelSelector\":\"\",\"IngressClass\":\"\",\"IngressEndpoint\":null}","time":"2019-01-15T16:25:20Z"}
{"level":"info","msg":"Starting provider *acme.Provider {\"Email\":\"jeff.n.may@gmail.com\",\"ACMELogging\":true,\"CAServer\":\"https://acme-staging-v02.api.letsencrypt.org/directory\",\"Storage\":\"/acme/acme.json\",\"EntryPoint\":\"https\",\"KeyType\":\"\",\"OnHostRule\":true,\"OnDemand\":false,\"DNSChallenge\":{\"Provider\":\"digitalocean\",\"DelayBeforeCheck\":0,\"Resolvers\":null,\"DisablePropagationCheck\":false},\"HTTPChallenge\":null,\"TLSChallenge\":null,\"Domains\":[{\"Main\":\"traefik.example.org\",\"SANs\":null}],\"Store\":{}}","time":"2019-01-15T16:25:20Z"}
{"level":"info","msg":"Testing certificate renew...","time":"2019-01-15T16:25:20Z"}
{"level":"debug","msg":"Using Ingress label selector: \"\"","time":"2019-01-15T16:25:20Z"}
{"level":"info","msg":"ingress label selector is: \"\"","time":"2019-01-15T16:25:20Z"}
{"level":"info","msg":"Creating in-cluster Provider client","time":"2019-01-15T16:25:20Z"}
{"level":"debug","msg":"Configuration received from provider ACME: {}","time":"2019-01-15T16:25:20Z"}
{"level":"debug","msg":"Looking for provided certificate(s) to validate [\"traefik.example.org\"]...","time":"2019-01-15T16:25:20Z"}
{"level":"debug","msg":"Domains [\"traefik.example.org\"] need ACME certificates generation for domains \"traefik.example.org\".","time":"2019-01-15T16:25:20Z"}
{"level":"debug","msg":"Loading ACME certificates [traefik.example.org]...","time":"2019-01-15T16:25:20Z"}
{"level":"info","msg":"The key type is empty. Use default key type 4096.","time":"2019-01-15T16:25:20Z"}
{"level":"debug","msg":"Add certificate for domains *.example.com","time":"2019-01-15T16:25:20Z"}
{"level":"info","msg":"Server configuration reloaded on :80","time":"2019-01-15T16:25:20Z"}
{"level":"info","msg":"Server configuration reloaded on :443","time":"2019-01-15T16:25:20Z"}
{"level":"info","msg":"Server configuration reloaded on :8080","time":"2019-01-15T16:25:20Z"}
{"level":"debug","msg":"Received Kubernetes event kind *v1beta1.Ingress","time":"2019-01-15T16:25:21Z"}
{"level":"debug","msg":"Configuration received from provider kubernetes: {\"backends\":{\"traefik.example.org\":{\"loadBalancer\":{\"method\":\"wrr\"}}},\"frontends\":{\"traefik.example.org\":{\"entryPoints\":[\"http\",\"https\"],\"backend\":\"traefik.example.org\",\"routes\":{\"traefik.example.org\":{\"rule\":\"Host:traefik.example.org\"}},\"passHostHeader\":true,\"priority\":0,\"basicAuth\":null}}}","time":"2019-01-15T16:25:21Z"}
{"level":"debug","msg":"Received Kubernetes event kind *v1.Endpoints","time":"2019-01-15T16:25:21Z"}
{"level":"debug","msg":"Skipping Kubernetes event kind *v1.Endpoints","time":"2019-01-15T16:25:21Z"}
{"level":"debug","msg":"Add certificate for domains *.example.com","time":"2019-01-15T16:25:21Z"}
{"level":"debug","msg":"Wiring frontend traefik.example.org to entryPoint http","time":"2019-01-15T16:25:21Z"}
{"level":"debug","msg":"Creating backend traefik.example.org","time":"2019-01-15T16:25:21Z"}
{"level":"debug","msg":"Adding TLSClientHeaders middleware for frontend traefik.example.org","time":"2019-01-15T16:25:21Z"}
{"level":"debug","msg":"Creating load-balancer wrr","time":"2019-01-15T16:25:21Z"}
{"level":"debug","msg":"Creating route traefik.example.org Host:traefik.example.org","time":"2019-01-15T16:25:21Z"}
{"level":"debug","msg":"Wiring frontend traefik.example.org to entryPoint https","time":"2019-01-15T16:25:21Z"}
{"level":"debug","msg":"Creating backend traefik.example.org","time":"2019-01-15T16:25:21Z"}
{"level":"debug","msg":"Adding TLSClientHeaders middleware for frontend traefik.example.org","time":"2019-01-15T16:25:21Z"}
{"level":"debug","msg":"Creating load-balancer wrr","time":"2019-01-15T16:25:21Z"}
{"level":"debug","msg":"Creating route traefik.example.org Host:traefik.example.org","time":"2019-01-15T16:25:21Z"}
{"level":"info","msg":"Server configuration reloaded on :443","time":"2019-01-15T16:25:21Z"}
{"level":"info","msg":"Server configuration reloaded on :8080","time":"2019-01-15T16:25:21Z"}
{"level":"info","msg":"Server configuration reloaded on :80","time":"2019-01-15T16:25:21Z"}
{"level":"debug","msg":"Try to challenge certificate for domain [traefik.example.org] founded in Host rule","time":"2019-01-15T16:25:21Z"}
{"level":"debug","msg":"Looking for provided certificate(s) to validate [\"traefik.example.org\"]...","time":"2019-01-15T16:25:21Z"}
{"level":"debug","msg":"No ACME certificate generation required for domains [\"traefik.example.org\"].","time":"2019-01-15T16:25:21Z"}
{"level":"debug","msg":"Received Kubernetes event kind *v1.Secret","time":"2019-01-15T16:25:22Z"}
{"level":"debug","msg":"Skipping Kubernetes event kind *v1.Secret","time":"2019-01-15T16:25:22Z"}
{"level":"debug","msg":"Received Kubernetes event kind *v1.Secret","time":"2019-01-15T16:25:22Z"}
{"level":"debug","msg":"Skipping Kubernetes event kind *v1.Secret","time":"2019-01-15T16:25:22Z"}
{"level":"debug","msg":"Received Kubernetes event kind *v1.Secret","time":"2019-01-15T16:25:22Z"}
{"level":"debug","msg":"Skipping Kubernetes event kind *v1.Secret","time":"2019-01-15T16:25:22Z"}
{"level":"debug","msg":"Received Kubernetes event kind *v1.Secret","time":"2019-01-15T16:25:22Z"}
{"level":"debug","msg":"Skipping Kubernetes event kind *v1.Secret","time":"2019-01-15T16:25:22Z"}
{"level":"debug","msg":"Received Kubernetes event kind *v1.Secret","time":"2019-01-15T16:25:22Z"}
{"level":"debug","msg":"Skipping Kubernetes event kind *v1.Secret","time":"2019-01-15T16:25:22Z"}
{"level":"debug","msg":"Received Kubernetes event kind *v1.Secret","time":"2019-01-15T16:25:22Z"}
{"level":"debug","msg":"Skipping Kubernetes event kind *v1.Secret","time":"2019-01-15T16:25:22Z"}
{"level":"debug","msg":"Received Kubernetes event kind *v1.Endpoints","time":"2019-01-15T16:25:23Z"}
{"level":"debug","msg":"Skipping Kubernetes event kind *v1.Endpoints","time":"2019-01-15T16:25:23Z"}
{"level":"debug","msg":"Received Kubernetes event kind *v1.Endpoints","time":"2019-01-15T16:25:23Z"}
{"level":"debug","msg":"Skipping Kubernetes event kind *v1.Endpoints","time":"2019-01-15T16:25:23Z"}
{"level":"debug","msg":"Received Kubernetes event kind *v1.Endpoints","time":"2019-01-15T16:25:25Z"}
{"level":"debug","msg":"Skipping Kubernetes event kind *v1.Endpoints","time":"2019-01-15T16:25:25Z"}
{"level":"debug","msg":"Received Kubernetes event kind *v1.Endpoints","time":"2019-01-15T16:25:25Z"}
{"level":"debug","msg":"Skipping Kubernetes event kind *v1.Endpoints","time":"2019-01-15T16:25:25Z"}
{"level":"debug","msg":"Received Kubernetes event kind *v1.Endpoints","time":"2019-01-15T16:25:27Z"}
{"level":"debug","msg":"Skipping Kubernetes event kind *v1.Endpoints","time":"2019-01-15T16:25:27Z"}
{"level":"debug","msg":"Received Kubernetes event kind *v1.Endpoints","time":"2019-01-15T16:25:27Z"}
{"level":"debug","msg":"Skipping Kubernetes event kind *v1.Endpoints","time":"2019-01-15T16:25:27Z"}
{"level":"debug","msg":"Received Kubernetes event kind *v1.Endpoints","time":"2019-01-15T16:25:29Z"}
{"level":"debug","msg":"Skipping Kubernetes event kind *v1.Endpoints","time":"2019-01-15T16:25:29Z"}
{"level":"debug","msg":"Received Kubernetes event kind *v1.Endpoints","time":"2019-01-15T16:25:29Z"}
{"level":"debug","msg":"Skipping Kubernetes event kind *v1.Endpoints","time":"2019-01-15T16:25:29Z"}
{"level":"debug","msg":"Received Kubernetes event kind *v1.Endpoints","time":"2019-01-15T16:25:31Z"}
{"level":"debug","msg":"Skipping Kubernetes event kind *v1.Endpoints","time":"2019-01-15T16:25:31Z"}
{"level":"debug","msg":"Received Kubernetes event kind *v1.Endpoints","time":"2019-01-15T16:25:31Z"}
{"level":"debug","msg":"Skipping Kubernetes event kind *v1.Endpoints","time":"2019-01-15T16:25:31Z"}
{"level":"debug","msg":"Received Kubernetes event kind *v1.Endpoints","time":"2019-01-15T16:25:33Z"}
{"level":"debug","msg":"Skipping Kubernetes event kind *v1.Endpoints","time":"2019-01-15T16:25:33Z"}
{"level":"debug","msg":"Received Kubernetes event kind *v1.Endpoints","time":"2019-01-15T16:25:33Z"}
{"level":"debug","msg":"Skipping Kubernetes event kind *v1.Endpoints","time":"2019-01-15T16:25:33Z"}
{"level":"debug","msg":"Received Kubernetes event kind *v1.Endpoints","time":"2019-01-15T16:25:35Z"}
{"level":"debug","msg":"Skipping Kubernetes event kind *v1.Endpoints","time":"2019-01-15T16:25:35Z"}
{"level":"debug","msg":"Received Kubernetes event kind *v1.Endpoints","time":"2019-01-15T16:25:35Z"}
{"level":"debug","msg":"Received Kubernetes event kind *v1.Endpoints","time":"2019-01-15T16:25:35Z"}
{"level":"debug","msg":"Skipping Kubernetes event kind *v1.Endpoints","time":"2019-01-15T16:25:35Z"}
{"level":"debug","msg":"Configuration received from provider kubernetes: {\"backends\":{\"traefik.example.org\":{\"servers\":{\"messy-koala-traefik-584cc9f68b-d9p6h\":{\"url\":\"http://10.244.94.3:8080\",\"weight\":1}},\"loadBalancer\":{\"method\":\"wrr\"}}},\"frontends\":{\"traefik.example.org\":{\"entryPoints\":[\"http\",\"https\"],\"backend\":\"traefik.example.org\",\"routes\":{\"traefik.example.org\":{\"rule\":\"Host:traefik.example.org\"}},\"passHostHeader\":true,\"priority\":0,\"basicAuth\":null}}}","time":"2019-01-15T16:25:35Z"}
{"level":"debug","msg":"Add certificate for domains *.example.com","time":"2019-01-15T16:25:35Z"}
{"level":"debug","msg":"Wiring frontend traefik.example.org to entryPoint http","time":"2019-01-15T16:25:35Z"}
{"level":"debug","msg":"Creating backend traefik.example.org","time":"2019-01-15T16:25:35Z"}
{"level":"debug","msg":"Adding TLSClientHeaders middleware for frontend traefik.example.org","time":"2019-01-15T16:25:35Z"}
{"level":"debug","msg":"Creating load-balancer wrr","time":"2019-01-15T16:25:35Z"}
{"level":"debug","msg":"Creating server messy-koala-traefik-584cc9f68b-d9p6h at http://10.244.94.3:8080 with weight 1","time":"2019-01-15T16:25:35Z"}
{"level":"debug","msg":"Creating route traefik.example.org Host:traefik.example.org","time":"2019-01-15T16:25:35Z"}
{"level":"debug","msg":"Wiring frontend traefik.example.org to entryPoint https","time":"2019-01-15T16:25:35Z"}
{"level":"debug","msg":"Creating backend traefik.example.org","time":"2019-01-15T16:25:35Z"}
{"level":"debug","msg":"Adding TLSClientHeaders middleware for frontend traefik.example.org","time":"2019-01-15T16:25:35Z"}
{"level":"debug","msg":"Creating load-balancer wrr","time":"2019-01-15T16:25:35Z"}
{"level":"debug","msg":"Creating server messy-koala-traefik-584cc9f68b-d9p6h at http://10.244.94.3:8080 with weight 1","time":"2019-01-15T16:25:35Z"}
{"level":"debug","msg":"Creating route traefik.example.org Host:traefik.example.org","time":"2019-01-15T16:25:35Z"}
{"level":"info","msg":"Server configuration reloaded on :80","time":"2019-01-15T16:25:35Z"}
{"level":"info","msg":"Server configuration reloaded on :443","time":"2019-01-15T16:25:35Z"}
{"level":"info","msg":"Server configuration reloaded on :8080","time":"2019-01-15T16:25:35Z"}
{"level":"debug","msg":"Try to challenge certificate for domain [traefik.example.org] founded in Host rule","time":"2019-01-15T16:25:35Z"}
{"level":"debug","msg":"Looking for provided certificate(s) to validate [\"traefik.example.org\"]...","time":"2019-01-15T16:25:35Z"}
{"level":"debug","msg":"No ACME certificate generation required for domains [\"traefik.example.org\"].","time":"2019-01-15T16:25:35Z"}
{"level":"debug","msg":"Received Kubernetes event kind *v1.Endpoints","time":"2019-01-15T16:25:35Z"}
{"level":"debug","msg":"Skipping Kubernetes event kind *v1.Endpoints","time":"2019-01-15T16:25:35Z"}
{"level":"debug","msg":"Received Kubernetes event kind *v1.Endpoints","time":"2019-01-15T16:25:37Z"}
{"level":"debug","msg":"Skipping Kubernetes event kind *v1.Endpoints","time":"2019-01-15T16:25:37Z"}

After which the following logs are repeated forever:

{"level":"debug","msg":"Received Kubernetes event kind *v1.Endpoints","time":"2019-01-15T16:25:37Z"}
{"level":"debug","msg":"Skipping Kubernetes event kind *v1.Endpoints","time":"2019-01-15T16:25:37Z"}

Am I missing some config? I can't assign an A record to the LoadBalancer until it has an external IP address.

UPDATE

I cancelled and retried and the second time, it worked. I just didn't wait long enough. I was able to manually set an A record on Digital Ocean after it came up.

When I went to the Traefik dashboard, however, I was warned about my certificate. Automating the DNS might bring the app up in time to coordinate with Let's Encrypt CA... haven't tried this yet.

-- Jeff May
digital-ocean
kubernetes
kubernetes-helm
traefik
traefik-ingress

1 Answer

1/16/2019

Yes, the acme config for traefik is expecting you to have a DNS record exist.

You need to use something like external-dns to register a DNS record for your ingress

-- jaxxstorm
Source: StackOverflow