What is the source of strange 404 and 403 WARNINGs in GCE Loadbalancer Logs

4/6/2018

Checking the logs of a GCE Loadbalancer in the Google Cloud Platform Logs, shows a bunch of WARNINGs in the form of:

"GET https://<MY_DOMAIN>/.well-known/acme-challenge/*" 404 215 "Go-http-client/1.1"

and ..:

"GET https://<MY_SERVICE_DOMAIN>/*" 401 561 "Go-http-client/1.1"

What is causing these calls? Is it some kind of health check?

From what I gather from the docs, the readiness-probes of the backing Pods should be expected to be called. Also as far as I see the backend service groups are considered healthy.

As they appear as WARNINGs in the logs, I assume I should work on making them go away?

-- Markward Schubert
google-cloud-platform
kubernetes
kubernetes-ingress

1 Answer

4/8/2018

"GET https://<MY_DOMAIN>/.well-known/acme-challenge/*" 404 215 "Go-http-client/1.1"

That one is pretty straightforward and is caused by the Let's Encrypt protocol checking for your ownership over the domain. It's hard to say whether that's an actual error without knowing whether you were expecting Let's Encrypt to check that domain.

"GET https://<MY_SERVICE_DOMAIN>/*" 401 561 "Go-http-client/1.1"

Without knowing what MY_SERVICE_DOMAIN means, that's also hard to know, but I wouldn't expect readiness checks to involve the LoadBalancer since (as you correctly observed) that check should be at the Pod level, not from outside the cluster

As they appear as WARNINGs in the logs, I assume I should work on making them go away?

That likely is a personal preference. Without any question having those extraneous messages makes finding actual warnings harder, but I doubt they are actually hurting anything, either. The distinction to me would be whether some process is expecting a successful HTTP response to the MY_SERVICE_DOMAIN request, and when it doesn't receive one, that causes a downstream failure -- it won't be the LoadBalancer that would require action, but rather the consumer of it.

-- mdaniel
Source: StackOverflow