Kubernetes deployment - Externalizing log file

3/29/2018

I have a spring boot application with the below docker file.

FROM docker.com/base/jdk1.8:latest

MAINTAINER Application Engineering [ https://docker.com/ ]

RUN mkdir -p /opt/docker/svc

COPY application/weather-service.war /opt/docker/svc/

CMD java -jar /opt/docker/svc/weather-service.war --spring.config.location=file:/conf/application.properties -Dlogging.config=/conf/logback.xml

I can use kubernetes configMap or secrets for application.properties and use the volume mount option as below.

"spec": {
        "volumes": [
          {
            "name": "svc-prop",
            "configMap": {
              "name": "svc-app-config",
              "items": [
                {
                  "key": "application.properties",
                  "path": "application.properties"
                }
              ]
            }
          }
         ],
        "containers": [
          "volumeMounts": [
              {
                "name": "svc-prop",
                "mountPath": "/conf"
              }
         ]

How can i achieve the samething for logback.xml. Do i need to use secrets as a file in this case?

I dont want to bundle logback.xml file with the image as we might be changing the log level at runtime.

Is there any other better approach for keeping logback.xml for spring boot app in Kubernetes?

-- user1578872
kubernetes
logback
spring-boot

2 Answers

8/14/2018

Usually you do not want to provide the whole logback.xml file but rather it's logger list which requires updating at runtime most frequently. In order to achieve this you can use Logback's file inclusion feature:

  1. Write your logback.xml file as usual except logger list. Use include element instead:
    <configuration scan="true" scanPeriod="10 seconds" debug="true">
        <appender ...></appender>
        <root level="INFO">...</root>

        <!-- Import loggers configuration from external file -->
        <include file="config/mount/loggers-include.xml"/>
    </configuration>

Note those scan* attributes. They are essential for log config reloading at runtime.

  1. Define all the loggers in Kubernetes ConfigMap with loggers-include.xml data section:

    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: microservice-loggers    # the name to refer to from deployment (see below)
      namespace: upc
    data:
      loggers-include.xml: |+
        <included>
          <logger name="org.springframework.cloud.netflix.zuul" level="INFO"/>
          <logger name="com.netflix.zuul" level="INFO"/>                              
          <logger name="com.netflix.hystrix" level="INFO"/>                          
          <logger name="com.netflix.ribbon" level="DEBUG"/>
          <logger name="com.netflix.loadbalancer" level="INFO"/>                     
        </included>

    Note that all the included content must be enclosed in included tag in order to be correctly parsed by Logback.

  2. Mount your ConfigMap's data into container as config/mount/loggers-include.xml file:

    apiVersion: apps/v1
    kind: Deployment
    ...
    spec:
      ...
      template:
        ...
        spec:
          # declare the volume created from ConfigMap
          volumes:
            - name: config-volume             # used with this name below
              configMap:
                name: microservice-loggers    # declared in previous step
          containers:
          - name: microservice
            ...
            ports:
              ...
            # mount the volume declared above to container's file system
            volumeMounts:
              - mountPath: /microservice/config/mount
                name: config-volume       # declared above

    Note that mount directory must not be created neither by container itself nor its image. Moreover, if such a directory exists, all its content will be removed during mounting.

  3. Apply the ConfigMap and run the declared deployment. To check if loggers file is correctly mounted execute the following command:

    $ kubectl exec restorun-7d757b7c6-wcslx -- ls -l /microservice/config/mount
    total 0
    lrwxrwxrwx    1 root     root            26 Aug 14 05:52 loggers-include.xml -> ..data/loggers-include.xml

    Also if you've set debug=true attribute in Logback's configuration element (see step 1) then you should see the following record in STDOUT during application startup:

    05:52:17,031 |-INFO in ch.qos.logback.core.joran.util.ConfigurationWatchListUtil@6e06451e - Adding [file:/microservice/config/mount/loggers-include.xml] to configuration watch list.
  4. Now you can edit your ConfigMap (e.g. set com.netflix.hystrix to level WARN), save its file and tell Kubernetes to apply the changes to the application:

    $ kubectl apply -f microservice-log-configmap.yaml
    configmap "microservice-loggers" configured

    Again, Logback should reflect the changes by logging with the following message to standard output:

        05:59:16,974 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction - Setting level of logger [com.netflix.hystrix] to WARN

    You can also check the effective logging level by directly asking it from the Spring Boot Actuator (if you have an access to this endpoint):

    $ curl http://<k8s-external-ip>/actuator/loggers/com.netflix.hystrix
    {
      "configuredLevel" : "WARN",
      "effectiveLevel" : "WARN"
    }
    

If the level stays the same, wait for a minute and check again: it takes some time for changes to propagate through Kubernetes and Logback. More info on this topic:

-- Toparvion
Source: StackOverflow

3/30/2018

You have already done... use ConfigMap and update ConfigMap by using "kubectl edit" when logback.xml needs to be updated.

When a ConfigMap or a Secret is updated, it gets eventually reflected inside the container, and if so configured such as via scan, then the application will pick up the change eventually. If not, the pod(s) need to be restarted to pick up the updated logback.xml.

Create a configmap for logback.

apiVersion: v1
kind: ConfigMap
metadata:
  name: logback-configmap
data:
  logback.xml: |+
    <?xml version="1.0" encoding="UTF-8"?>
    <configuration>
      <include resource="org/springframework/boot/logging/logback/base.xml"/>
      <logger name="org.springframework.web" level="DEBUG"/>
    </configuration>

Specify the configmap as a volume in your deployment.

  volumes:
  - configMap:
      name: logback-configmap
    name: logback

Mount the configmap volume in your container.

    volumeMounts:
    - mountPath: /path/to/logback.xml
      name: logback
-- mon
Source: StackOverflow