I maintain a SpringBoot REST service.

A large and complex Jenkinsfile build script has been set up for me, which in addition to building the code, will build and smoke test the Docker container, then run a k8s deployment to create pods and a service, and then smoketest the service.

The REST service requires more than three minutes from startup to "ready". I implemented a HealthIndicator subclass that checks for the required constraints.

When the build runs, the "/health" url on the container works as I would expect, as it takes many tries until it gets "UP". I also added a bit of temporary Jenkinsfile script that makes a call to a specific service operation that returns a json response I can use to verify that the service is ACTUALLY working.

In the build output, everything is fine so far.

When it deploys to k8s, it seems like it all works. However, I noticed that when it executed the same "/health" url on the deployed k8s service, it returned an "UP" response IMMEDIATELY after starting up the pod. That alone told me that something's not right here.

At that point, I went to a shell and tried several experiments. I verified that I could reach the same "/health" url on the service, which returned the same "UP" response. I then changed the url to make the specific service operation call that I made in the Jenkinsfile. That returned a 500 error. I then looked at the three pods associated with the service. I checked the logs for all three of them, and none of them had any indication that they failed to return a response to something. It didn't appear a request had been sent to any of those containers.

So, the behavior I'm seeing from the service's /health url, and the specific service operation call, seem to tell me that this service is not talking to the containers in my pods.

What can I do with kubectl and any other diagnostics that would give me enough information to diagnose what's going on here?