normally I think it's a best practice for an incorrectly configured application to simply die on startup with a detailed error message describing the problem.
For example, if an expected environment variable is missing, meaning the application can't run properly, instead of letting it run in a zombie state where it will never function, I advocate for failing loudly and killing the application with an error message:
Critical Error: Environment variable [REDIS_HOST] not set.
In kubernetes, this ends up in a constant CrashLoop Backoff loop. This isn't great as it's hard to get to that error message as the pod keeps getting restarted and the logs disappear.
Any thoughts or suggestions on the proper way to handle this?
thanks
You can customise a container's termination message by writing to /dev/termination-log by default. When your container terminates you can use kubectl get pods <podName> -o go-template="{{range .status.containerStatuses}}{{.lastState.terminated.message}}{{end}}" to retrieve the message. More information about this can be found here.
You can also use kubectl logs <podName> -c <containerName> --previous to view the output of the previous instance of a particular container in a Pod - this may be more useful for you as you won't have to change your application to write error messages to /dev/termination-log.