Kubernetes Jobs are not Completing due to splunk dependency

10/4/2021

We have the cron job which is running every night 11:30 PM. Now issue is job is not shutting down after completing execution. We have added splunk dependency in our application. The splunk listener doesnt allow to shut down the job. The dependency which we used

<dependency>
    <groupId>com.splunk.logging</groupId>
    <artifactId>splunk-library-javalogging</artifactId>
    <version>1.7.3</version>
    <scope>runtime</scope>
</dependency>.

We have added splunk appender in logback.xml to check logs in splunk

<appender name="SPLUNK" class="com.splunk.logging.HttpEventCollectorLogbackAppender">
    <url>https://operations-hec.splunk.tescocloud.com</url>
    <index>${SPLUNK_INDEX}</index>
    <source>${SPLUNK_SOURCE}</source>
    <sourcetype>${SPLUNK_SOURCE_TYPE}</sourcetype>
    <token>${SPLUNK_TOKEN}</token>
    <batch_size_count>${SPLUNK_BATCH_SIZE}</batch_size_count>
    <batch_interval>${SPLUNK_BATCH_BATCH_INTERVALMS}</batch_interval>
    <disableCertificateValidation>true</disableCertificateValidation>
    <retries_on_error>3</retries_on_error>
    <layout class="ch.qos.logback.classic.PatternLayout">
        <pattern>[%mdc] %msg</pattern>
    </layout>
    <messageFormat>json</messageFormat>
</appender>

<if condition='"pte".equals(property("ENVIRONMENT"))'>
    <then>
        <root level="WARN">
            <appender-ref ref="CONSOLE"/>
        </root>
    </then>
    <else>
        <root level="INFO">
            <appender-ref ref="SPLUNK"/>
            <appender-ref ref="RollingFile"/>
            <appender-ref ref="CONSOLE"/>
        </root>
    </else>
</if>

In main class we have added below code to stop the job

@SpringBootApplication
@EnableFeignClients
public class MyBatchJob {

    public static void main(String[] args) {
        var context = SpringApplication.run(MyBatchJob.class, args);

        Optional.of(getILoggerFactory()).filter(LoggerContext.class::isInstance).map(LoggerContext.class::cast)
                .ifPresent(LoggerContext::stop);

        context.close();
    }
}

Jobs executions are completing but in Kubernetes the status of the job is Running. enter image description here

But The job status in Kubernetes is in Running status. enter image description here

There is a Splunk listener who is listening while writing logs to the console, the listener is still open after the job completion, it is still writing to the console. Please help me how to shut down the listener so that the job will be on completed status.

-- user3815362
cron
kubernetes
microservices
splunk
spring-boot

0 Answers