I am trying to move a Spring boot application onto a openshift 3 cluster. I am using the Maven fabric8 plugin to generate most of the openshift boilerplate config as well as performing the S2I build. When my pod starts up I can see in the log output that the application starts up but right after spring boot defaults to the default profile (I didn't set a profile yet) the app crashes and the only output I see in the Openshift log is Killed... I couldn't find anything of value googleing except it seemed that it could be openJDK that tries to grab more memory than is a available for a single pod. I added a fabric8 fragment that limits the memory a single container is able to use but I still get the same error when I start up the pod. I ran OC describe pod and saw a exit code 143. i'm running out of ideas and would appreciate Any idea on how I can debug this further/how to resolve this sort of issue? Also not sure if this is related but even though my application.yml is set up to enable SSL the route that fabric8 creates is always a HTTP URL and not a HTTPS URL. i'm wondering if that could be the cause of the pod going into a crashloop as the readinessProbe and wellnessProbe can't hit the actuator endpoints?
The console output, maven pom file, application.yml and fabric8 config is below.
console output:
. ____ _ __ _ _
/\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
\\/ ___)| |_)| | | | | || (_| | ) ) ) )
' |____| .__|_| |_|_| |_\__, | / / / /
=========|_|==============|___/=/_/_/_/
:: Spring Boot :: (v2.1.3.RELEASE)
2020-06-27 10:14:06.297 INFO 8 --- [ main] com.xib.vodacom.ans.AnsApplicationKt : Starting AnsApplicationKt v1.0.9 on vc-ans-1-bzlxc-debug with PID 8 (vc-ans-1.0.9.jar started by ? in /deployments)
2020-06-27 10:14:06.396 DEBUG 8 --- [ main] com.xib.vodacom.ans.AnsApplicationKt : Running with Spring Boot v2.1.3.RELEASE, Spring v5.1.4.RELEASE
2020-06-27 10:14:06.397 INFO 8 --- [ main] com.xib.vodacom.ans.AnsApplicationKt : No active profile set, falling back to default profiles: default
Killed
sh-4.2$
oc describe pod output:
Secret (a volume populated by a Secret)
SecretName: certs
Optional: false
default-token-8pts4:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-8pts4
Optional: false
QoS Class: Burstable
Node-Selectors: node-role.kubernetes.io/compute=true
Tolerations: node.kubernetes.io/memory-pressure:NoSchedule
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 6m default-scheduler Successfully assigned cx-ans-prod/vc-ans-1-bzlxc to pocpap8zadsas.cc.corp
Normal Created 5m (x3 over 6m) kubelet, asdf.cc.corp Created container
Normal Started 5m (x3 over 6m) kubelet, asdfgf.cc.corp Started container
Normal Killing 5m (x2 over 6m) kubelet, asdfgf.cc.corp Killing container with id docker://spring-boot:Container failed liveness probe.. Container will be killed and recreated.
Warning Unhealthy 5m (x7 over 6m) kubelet, asdfgf.cc.corp Readiness probe failed: Get https://192.168.31.110:8080/actuator/health: dial tcp 192.168.31.110:8080: connect: connection refused
Warning Unhealthy 5m (x7 over 6m) kubelet, asdfgf.cc.corp Liveness probe failed: Get https://192.168.31.110:8080/actuator/health: dial tcp 192.168.31.110:8080: connect: connection refused
Normal Pulled 1m (x7 over 6m) kubelet, asdfgf.cc.corp Container image "docker-registry.default.svc:5000/cx-ans-prod/vc-ans@sha256:4f39366bddc1e4ce7ed3c7e320453fbe6e90c4100f860a82ec87b04e7fb7e5b1" already present on machine
C:\dev\xib-ans-bo>
pom.xml:
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<modelVersion>4.0.0</modelVersion>
<groupId>com.xib</groupId>
<artifactId>vc-ans</artifactId>
<version>1.0.9</version>
<packaging>jar</packaging>
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>2.1.3.RELEASE</version>
</parent>
<scm>
<connection>scm:git:git://pgitap1zatcrh.vodacom.corp:XIB/xib-ans-bo.git</connection>
<url>git://pgitap1zatcrh.vodacom.corp:XIB/xib-ans-bo.git</url>
</scm>
<properties>
<java.version>1.8</java.version>
<kotlin.version>1.3.61</kotlin.version>
<fabric8.mode>openshift</fabric8.mode>
<fabric8.namespace>cx-ans-prod</fabric8.namespace>
</properties>
<dependencies>
<dependency>
<groupId>commons-lang</groupId>
<artifactId>commons-lang</artifactId>
<version>2.3</version>
<scope>compile</scope>
<exclusions>
<exclusion>
<artifactId>tomcat-annotations-api</artifactId>
<groupId>org.apache.tomcat</groupId>
</exclusion>
<exclusion>
<artifactId>commons-logging</artifactId>
<groupId>commons-logging</groupId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-actuator</artifactId>
<scope>compile</scope>
<exclusions>
<exclusion>
<artifactId>tomcat-annotations-api</artifactId>
<groupId>org.apache.tomcat</groupId>
</exclusion>
<exclusion>
<artifactId>commons-logging</artifactId>
<groupId>commons-logging</groupId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-jdbc</artifactId>
<scope>compile</scope>
<exclusions>
<exclusion>
<artifactId>tomcat-annotations-api</artifactId>
<groupId>org.apache.tomcat</groupId>
</exclusion>
<exclusion>
<artifactId>commons-logging</artifactId>
<groupId>commons-logging</groupId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
<scope>compile</scope>
<exclusions>
<exclusion>
<artifactId>tomcat-annotations-api</artifactId>
<groupId>org.apache.tomcat</groupId>
</exclusion>
<exclusion>
<artifactId>commons-logging</artifactId>
<groupId>commons-logging</groupId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-webflux</artifactId>
<scope>compile</scope>
<exclusions>
<exclusion>
<artifactId>tomcat-annotations-api</artifactId>
<groupId>org.apache.tomcat</groupId>
</exclusion>
<exclusion>
<artifactId>commons-logging</artifactId>
<groupId>commons-logging</groupId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-security</artifactId>
<scope>compile</scope>
<exclusions>
<exclusion>
<artifactId>tomcat-annotations-api</artifactId>
<groupId>org.apache.tomcat</groupId>
</exclusion>
<exclusion>
<artifactId>commons-logging</artifactId>
<groupId>commons-logging</groupId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.springframework.security</groupId>
<artifactId>spring-security-data</artifactId>
<scope>compile</scope>
<exclusions>
<exclusion>
<artifactId>tomcat-annotations-api</artifactId>
<groupId>org.apache.tomcat</groupId>
</exclusion>
<exclusion>
<artifactId>commons-logging</artifactId>
<groupId>commons-logging</groupId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.springframework.security</groupId>
<artifactId>spring-security-test</artifactId>
<scope>compile</scope>
<exclusions>
<exclusion>
<artifactId>tomcat-annotations-api</artifactId>
<groupId>org.apache.tomcat</groupId>
</exclusion>
<exclusion>
<artifactId>commons-logging</artifactId>
<groupId>commons-logging</groupId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>com.auth0</groupId>
<artifactId>java-jwt</artifactId>
<version>3.8.1</version>
<scope>compile</scope>
<exclusions>
<exclusion>
<artifactId>tomcat-annotations-api</artifactId>
<groupId>org.apache.tomcat</groupId>
</exclusion>
<exclusion>
<artifactId>commons-logging</artifactId>
<groupId>commons-logging</groupId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.springframework.data</groupId>
<artifactId>spring-data-commons</artifactId>
<scope>compile</scope>
<exclusions>
<exclusion>
<artifactId>tomcat-annotations-api</artifactId>
<groupId>org.apache.tomcat</groupId>
</exclusion>
<exclusion>
<artifactId>commons-logging</artifactId>
<groupId>commons-logging</groupId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.module</groupId>
<artifactId>jackson-module-kotlin</artifactId>
<scope>compile</scope>
<exclusions>
<exclusion>
<artifactId>tomcat-annotations-api</artifactId>
<groupId>org.apache.tomcat</groupId>
</exclusion>
<exclusion>
<artifactId>commons-logging</artifactId>
<groupId>commons-logging</groupId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
<version>25.1-jre</version>
<scope>compile</scope>
<exclusions>
<exclusion>
<artifactId>tomcat-annotations-api</artifactId>
<groupId>org.apache.tomcat</groupId>
</exclusion>
<exclusion>
<artifactId>commons-logging</artifactId>
<groupId>commons-logging</groupId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.liquibase</groupId>
<artifactId>liquibase-core</artifactId>
<scope>compile</scope>
<exclusions>
<exclusion>
<artifactId>tomcat-annotations-api</artifactId>
<groupId>org.apache.tomcat</groupId>
</exclusion>
<exclusion>
<artifactId>commons-logging</artifactId>
<groupId>commons-logging</groupId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.jetbrains.kotlin</groupId>
<artifactId>kotlin-stdlib-jdk8</artifactId>
<version>1.3.21</version>
<scope>compile</scope>
<exclusions>
<exclusion>
<artifactId>tomcat-annotations-api</artifactId>
<groupId>org.apache.tomcat</groupId>
</exclusion>
<exclusion>
<artifactId>commons-logging</artifactId>
<groupId>commons-logging</groupId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.jetbrains.kotlin</groupId>
<artifactId>kotlin-reflect</artifactId>
<version>1.3.21</version>
<scope>compile</scope>
<exclusions>
<exclusion>
<artifactId>tomcat-annotations-api</artifactId>
<groupId>org.apache.tomcat</groupId>
</exclusion>
<exclusion>
<artifactId>commons-logging</artifactId>
<groupId>commons-logging</groupId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpclient</artifactId>
<version>4.5.6</version>
<scope>compile</scope>
<exclusions>
<exclusion>
<artifactId>tomcat-annotations-api</artifactId>
<groupId>org.apache.tomcat</groupId>
</exclusion>
<exclusion>
<artifactId>commons-logging</artifactId>
<groupId>commons-logging</groupId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<artifactId>ojdbc8</artifactId>
<groupId>com.oracle.ojdbc</groupId>
<version>19.3.0.0</version>
<scope>compile</scope>
<exclusions>
<exclusion>
<artifactId>tomcat-annotations-api</artifactId>
<groupId>org.apache.tomcat</groupId>
</exclusion>
<exclusion>
<artifactId>commons-logging</artifactId>
<groupId>commons-logging</groupId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.jetbrains.kotlinx</groupId>
<artifactId>kotlinx-coroutines-core</artifactId>
<version>1.0.0</version>
<scope>compile</scope>
<exclusions>
<exclusion>
<artifactId>tomcat-annotations-api</artifactId>
<groupId>org.apache.tomcat</groupId>
</exclusion>
<exclusion>
<artifactId>commons-logging</artifactId>
<groupId>commons-logging</groupId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
<exclusions>
<exclusion>
<artifactId>commons-logging</artifactId>
<groupId>*</groupId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.springframework.restdocs</groupId>
<artifactId>spring-restdocs-core</artifactId>
<version>2.0.3.RELEASE</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.springframework.restdocs</groupId>
<artifactId>spring-restdocs-mockmvc</artifactId>
<version>2.0.3.RELEASE</version>
<scope>test</scope>
</dependency>
</dependencies>
<dependencyManagement>
<dependencies>
<dependency>
<groupId>org.jetbrains.kotlin</groupId>
<artifactId>kotlin-stdlib-jdk8</artifactId>
<version>1.3.21</version>
</dependency>
<dependency>
<groupId>org.jetbrains.kotlin</groupId>
<artifactId>kotlin-reflect</artifactId>
<version>1.3.21</version>
</dependency>
<dependency>
<groupId>org.jetbrains.kotlin</groupId>
<artifactId>kotlin-stdlib-jre7</artifactId>
<version>1.3.21</version>
</dependency>
<dependency>
<groupId>org.jetbrains.kotlin</groupId>
<artifactId>kotlin-stdlib-jre8</artifactId>
<version>1.3.21</version>
</dependency>
<dependency>
<groupId>org.jetbrains.kotlin</groupId>
<artifactId>kotlin-stdlib-jdk7</artifactId>
<version>1.3.21</version>
</dependency>
<dependency>
<groupId>org.jetbrains.kotlin</groupId>
<artifactId>kotlin-stdlib</artifactId>
<version>1.3.21</version>
</dependency>
<dependency>
<groupId>org.jetbrains.kotlin</groupId>
<artifactId>kotlin-runtime</artifactId>
<version>1.3.21</version>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-dependencies</artifactId>
<version>2.1.2.RELEASE</version>
<scope>import</scope>
<type>pom</type>
</dependency>
<dependency>
<groupId>com.auth0</groupId>
<artifactId>java-jwt</artifactId>
<version>3.8.1</version>
<scope>compile</scope>
<type>pom</type>
</dependency>
</dependencies>
</dependencyManagement>
<build>
<sourceDirectory>${project.basedir}main/kotlin</sourceDirectory>
<testSourceDirectory>${project.basedir}test/kotlin</testSourceDirectory>
<plugins>
<plugin>
<groupId>org.jetbrains.kotlin</groupId>
<artifactId>kotlin-maven-plugin</artifactId>
<configuration>
<args>
<arg>-Xjsr305=strict</arg>
</args>
<compilerPlugins>
<plugin>spring</plugin>
</compilerPlugins>
</configuration>
<dependencies>
<dependency>
<groupId>org.jetbrains.kotlin</groupId>
<artifactId>kotlin-maven-allopen</artifactId>
<version>${kotlin.version}</version>
</dependency>
</dependencies>
</plugin>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
<configuration>
<executable>true</executable>
</configuration>
</plugin>
<plugin>
<groupId>io.fabric8</groupId>
<artifactId>fabric8-maven-plugin</artifactId>
<version>4.4.1</version>
<configuration>
<generateRoute>true</generateRoute>
<enricher>
<config>
<fmp-service>
<name>ans-service</name>
<type>ClusterIP</type>
</fmp-service>
</config>
</enricher>
<enricher>
<config>
<fmp-maven-scm-enricher>>
<scm-tag>https</scm-tag>
</fmp-maven-scm-enricher>>
</config>
</enricher>
</configuration>
<executions>
<execution>
<goals>
<goal>resource</goal>
<goal>build</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
</project>
fabric8 deployment.yml fragment:
spec:
template:
spec:
containers:
- env:
- name: SPRING_PROFILES_ACTIVE
value: qa
volumeMounts:
- name: certs
mountPath: certs
readOnly: true
livenessProbe:
httpGet:
path: health
port: 8080
scheme: HTTPS
initialDelaySeconds: 30
readinessProbe:
httpGet:
path: health
port: 8080
scheme: HTTPS
initialDelaySeconds: 30
resources:
requests:
memory: "64Mi"
limits:
memory: "256Mi"
env:
- name: JVM_OPTS
value: "-Xms64M -Xmx256M"
volumes:
- name: certs
secret:
secretName: certs
items:
- key: ans.pfx
path: ans.pfx
- key: vb_pp_client_ans.jks
path: vb_pp_client_ans.jks
- key: vc_truststore.jks
path: vc_truststore.jks
spring boot application.yml:
app:
datasource:
driver-class-name: oracle.jdbc.driver.OracleDriver
url: jdbc:oracle:thin:@qgeno.cc.corp
username: asdfgf
password: asdfgf
connection-test-query: SELECT 1 from dual
pool-name: ANS_CP
minimum-idle: 3
maximum-pool-size: 20
alive:
file: appsvc-ans/running.txt
spring:
main:
allow-bean-definition-overriding: true
vb:
vb-url: "https://asdfgf.cc.corp"
hc-check-msisdn: "0821341111"
ssl-cert-password: "asdfgf"
ssl-cert-alias: "asdfgf"
keystore-path: "/opt/certs/vb_pp_client_ans.jks"
repeat-caller-url: https://asdfgf.cc.corp
server:
http:
port: 8080
ssl:
enabled: true
protocol: TLS
key-alias: lkjhj
key-store: certs/ans.pfx
key-store-password: asdfgf
trust-store: certs/vc_truststore.jks
trust-store-password: asdfgf
logging:
level:
com.xib: DEBUG
management:
server:
ssl:
key-store: certs/ans.pfx
key-store-password: asdfgf
endpoint:
health:
show-details: always
Error code 143 is SIGTERM which Kubernetes uses to shutdown pods. This means Kubernetes wants your pod killed for a reason. There are 3 options that I think might be the case:
1: As you said, it can be triggered by memory allocation greater than given memory limit. In that case Kubernetes will just kill your pod and keeps node stable but if it was cpu limit then Kubernetes could limit the resource allocation and wouldn't kill the pod. You can check if resource limit is lower than your app requirement by running this command: oc get deployment deployment_name -o yaml
2: AS @Arghya Sadhu said, you could have misconfigured actuator somehow, try to check if liveness and readiness probe are correctly set and gives correct output. You can also get this configuration by above command.
3: Everything works great except your app takes too much to get ready and Kubernetes decides your app is misbehaving. This could be problem if you set resource limitations too narrow and your java app has too little resource to get prepared in time.
Add below in the application.yml
management.endpoint.health.group.readiness.include=*
management.endpoint.health.group.readiness.show-details=always
management.endpoint.health.group.liveness.include=ping
management.endpoint.health.group.liveness.show-details=neverUse /actuator/health/readiness for readiness and /actuator/health/liveness for liveness in the deployment.yaml
readinessProbe:
httpGet:
port: healthcheck
path: /actuator/health/readiness
initialDelaySeconds: 10
livenessProbe:
httpGet:
port: healthcheck
path: /actuator/health/liveness
initialDelaySeconds: 60
periodSeconds: 1Additionally, if Spring Security is present, you would need to add custom security configuration that allows unauthenticated access to the endpoints as shown in the following example:
@Configuration(proxyBeanMethods = false)
public class ActuatorSecurity extends WebSecurityConfigurerAdapter {
@Override
protected void configure(HttpSecurity http) throws Exception {
http.requestMatcher(EndpointRequest.toAnyEndpoint()).authorizeRequests((requests) ->
requests.anyRequest().permitAll());
}
}
If you deploy applications behind a firewall, you may prefer that all your actuator endpoints can be accessed without requiring authentication. You can do so by changing the management.endpoints.web.exposure.include property, as follows in
application.yaml
management.endpoints.web.exposure.include=*