We have recently moved to micro services based architecture for our enterprise application. we are using Kubernetes cluster to host all our micro services.
Currently we didn't configure ELK for manage our logs, just storing application logs into azure blob storage.
We are facing issue, when multiple POD instances are running for one services, since all instances use same log file to update the content. due to this, instances are getting stuck and getting memory leak issue.
I have configured mount path in docker container , and my logback property , has below entry to write the logs.
<property name="DEV_HOME" value="/mnt/azure/<service-name>/logs" />
Is there a way to get the pod instance name in log configuration , so that i can add one more level down, to have separate logs for different instances.
Or is there better way to handle this scenario.
<property name="DEV_HOME" value="/mnt/azure/<service-name>/<instances>/logs" />
As mentioned by @PraveenSripati, downward api is the solution. However, there are cases where one is forced to use third-party libraries and not able to use environment variables to override the location. In past we have configured our pods to use a combination of downward API and a K8S command and args to run a custom script before starting the application.
Let's assume the log location is set to /opt/logs, in that case, we do something like below, We keep this script in the container while builing it. For instance we call this entrypoint.sh ,
#!/bin/bash
log_location="/opt/log-$MY_POD_NAME/$MY_POD_IP"
mkdir -p /opt/log-$MY_POD_NAME/$MY_POD_IP
rm -f /opt/log
ln -s /opt/log $log_location
exec <Application_start_command> Kubernetes POD definition
apiVersion: v1
kind: Pod
metadata:
name: dapi-envars-fieldref
spec:
containers:
- name: test-container
image: k8s.gcr.io/busybox
command: [ "sh", "-c"]
args:
- /opt/entrypoint.sh
env:
- name: MY_NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
- name: MY_POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: MY_POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: MY_POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
- name: MY_POD_SERVICE_ACCOUNT
valueFrom:
fieldRef:
fieldPath: spec.serviceAccountName
restartPolicy: NeverThis will create a symlink where the third-party library can write without changing any code there.
It should be possible to set the Pod information (including the name) as environment variables as mentioned here. In the application read the environment variable and log appropriately.