K
Q

Question

Keycloak crash during startup when using MySQL

3/9/2020

I'm running Keycloak in Kubernetes (Microk8s/Vagrant) and it's crashing on start up.

If I don't add the MySQL environment variables then Keycloak starts up fine.
I can successfully connect to MySQL from other containers.
Logs can vary between start ups. Sometimes with no errors at all (but still exits)

Keycloak also seems to be ignoring KEYCLOAK_LOGLEVEL and ROOT_LOGLEVEL.

Keycloak Log

Added 'admin' to '/opt/jboss/keycloak/standalone/configuration/keycloak-add-user.json', restart server to load user
-b 0.0.0.0
=========================================================================

  Using MySQL database

=========================================================================

10:11:25,069 INFO  [org.jboss.modules] (CLI command executor) JBoss Modules version 1.9.1.Final
10:11:25,210 INFO  [org.jboss.msc] (CLI command executor) JBoss MSC version 1.4.11.Final
10:11:25,239 INFO  [org.jboss.threads] (CLI command executor) JBoss Threads version 2.3.3.Final
10:11:25,445 INFO  [org.jboss.as] (MSC service thread 1-1) WFLYSRV0049: Keycloak 9.0.0 (WildFly Core 10.0.3.Final) starting
10:11:25,611 INFO  [org.jboss.vfs] (MSC service thread 1-1) VFS000002: Failed to clean existing content for temp file provider of type temp. Enable DEBUG level log to find what caused this
10:11:27,152 INFO  [org.wildfly.security] (ServerService Thread Pool -- 20) ELY00001: WildFly Elytron version 1.10.4.Final
10:11:28,589 INFO  [org.jboss.as.controller.management-deprecated] (Controller Boot Thread) WFLYCTL0028: Attribute 'security-realm' in the resource at address '/core-service=management/management-interface=http-interface' is deprecated, and may be removed in a future version. See the attribute description in the output of the read-resource-description operation to learn more about the deprecation.
10:11:28,869 INFO  [org.jboss.as.controller.management-deprecated] (Controller Boot Thread) WFLYCTL0028: Attribute 'security-realm' in the resource at address '/subsystem=undertow/server=default-server/https-listener=https' is deprecated, and may be removed in a future version. See the attribute description in the output of the read-resource-description operation to learn more about the deprecation.
10:11:29,177 INFO  [org.jboss.as.patching] (MSC service thread 1-2) WFLYPAT0050: Keycloak cumulative patch ID is: base, one-off patches include: none
10:11:29,239 WARN  [org.jboss.as.domain.management.security] (MSC service thread 1-1) WFLYDM0111: Keystore /opt/jboss/keycloak/standalone/configuration/application.keystore not found, it will be auto generated on first use with a self signed certificate for host localhost                                                                                                                                                                                                          
10:11:29,424 INFO  [org.jboss.as.server] (Controller Boot Thread) WFLYSRV0212: Resuming server
10:11:29,431 INFO  [org.jboss.as] (Controller Boot Thread) WFLYSRV0025: Keycloak 9.0.0 (WildFly Core 10.0.3.Final) started in 4333ms - Started 55 of 78 services (32 services are lazy, passive or on-demand)
The batch executed successfully
10:11:29,786 INFO  [org.jboss.as] (MSC service thread 1-1) WFLYSRV0050: Keycloak 9.0.0 (WildFly Core 10.0.3.Final) stopped in 35ms
10:11:32,602 INFO  [org.jboss.modules] (CLI command executor) JBoss Modules version 1.9.1.Final
10:11:32,733 INFO  [org.jboss.msc] (CLI command executor) JBoss MSC version 1.4.11.Final
10:11:32,742 INFO  [org.jboss.threads] (CLI command executor) JBoss Threads version 2.3.3.Final
10:11:32,953 INFO  [org.jboss.as] (MSC service thread 1-2) WFLYSRV0049: Keycloak 9.0.0 (WildFly Core 10.0.3.Final) starting
10:11:33,122 INFO  [org.jboss.vfs] (MSC service thread 1-2) VFS000002: Failed to clean existing content for temp file provider of type temp. Enable DEBUG level log to find what caused this
10:11:34,751 INFO  [org.wildfly.security] (ServerService Thread Pool -- 22) ELY00001: WildFly Elytron version 1.10.4.Final
10:11:36,595 INFO  [org.jboss.as.controller.management-deprecated] (Controller Boot Thread) WFLYCTL0028: Attribute 'security-realm' in the resource at address '/core-service=management/management-interface=http-interface' is deprecated, and may be removed in a future version. See the attribute description in the output of the read-resource-description operation to learn more about the deprecation.
10:11:36,815 INFO  [org.jboss.as.controller.management-deprecated] (Controller Boot Thread) WFLYCTL0028: Attribute 'security-realm' in the resource at address '/subsystem=undertow/server=default-server/https-listener=https' is deprecated, and may be removed in a future version. See the attribute description in the output of the read-resource-description operation to learn more about the deprecation.
10:11:37,215 INFO  [org.jboss.as.patching] (MSC service thread 1-2) WFLYPAT0050: Keycloak cumulative patch ID is: base, one-off patches include: none
10:11:37,262 WARN  [org.jboss.as.domain.management.security] (MSC service thread 1-2) WFLYDM0111: Keystore /opt/jboss/keycloak/standalone/configuration/application.keystore not found, it will be auto generated on first use with a self signed certificate for host localhost                                                                                                                                                                                                          
10:11:37,466 INFO  [org.jboss.as.server] (Controller Boot Thread) WFLYSRV0212: Resuming server
10:11:37,475 INFO  [org.jboss.as] (Controller Boot Thread) WFLYSRV0025: Keycloak 9.0.0 (WildFly Core 10.0.3.Final) started in 4848ms - Started 55 of 85 services (39 services are lazy, passive or on-demand)
The batch executed successfully
10:11:37,847 INFO  [org.jboss.as] (MSC service thread 1-1) WFLYSRV0050: Keycloak 9.0.0 (WildFly Core 10.0.3.Final) stopped in 49ms
=========================================================================

  JBoss Bootstrap Environment

  JBOSS_HOME: /opt/jboss/keycloak

  JAVA: java

  JAVA_OPTS:  -server -Xms64m -Xmx512m -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=256m -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true  --add-exports=java.base/sun.nio.ch=ALL-UNNAMED --add-exports=jdk.unsupported/sun.misc=ALL-UNNAMED --add-exports=jdk.unsupported/sun.reflect=ALL-UNNAMED

=========================================================================

10:11:39,080 INFO  [org.jboss.modules] (main) JBoss Modules version 1.9.1.Final
10:11:40,028 INFO  [org.jboss.msc] (main) JBoss MSC version 1.4.11.Final
10:11:40,053 INFO  [org.jboss.threads] (main) JBoss Threads version 2.3.3.Final
10:11:40,300 INFO  [org.jboss.as] (MSC service thread 1-2) WFLYSRV0049: Keycloak 9.0.0 (WildFly Core 10.0.3.Final) starting
10:11:40,466 INFO  [org.jboss.vfs] (MSC service thread 1-1) VFS000002: Failed to clean existing content for temp file provider of type temp. Enable DEBUG level log to find what caused this
*** JBossAS process (325) received TERM signal ***
10:11:40,686 INFO  [org.jboss.as.server] (Thread-1) WFLYSRV0220: Server shutdown has been requested via an OS signal
10:11:41,114 ERROR [org.jboss.msc.service.fail] (MSC service thread 1-2) MSC000001: Failed to start service jboss.as.server-controller: org.jboss.msc.service.StartException in service jboss.as.server-controller: Failed to start service
        at org.jboss.msc@1.4.11.Final//org.jboss.msc.service.ServiceControllerImpl$StartTask.execute(ServiceControllerImpl.java:1731)
        at org.jboss.msc@1.4.11.Final//org.jboss.msc.service.ServiceControllerImpl$ControllerTask.run(ServiceControllerImpl.java:1559)
        at org.jboss.threads@2.3.3.Final//org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
        at org.jboss.threads@2.3.3.Final//org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1982)
        at org.jboss.threads@2.3.3.Final//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486)
        at org.jboss.threads@2.3.3.Final//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1363)
        at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.lang.IllegalStateException: Container is down
        at org.jboss.msc@1.4.11.Final//org.jboss.msc.service.ServiceContainerImpl.install(ServiceContainerImpl.java:739)
        at org.jboss.msc@1.4.11.Final//org.jboss.msc.service.ServiceTargetImpl.install(ServiceTargetImpl.java:260)
        at org.jboss.msc@1.4.11.Final//org.jboss.msc.service.ServiceControllerImpl$ChildServiceTarget.install(ServiceControllerImpl.java:2065)
        at org.jboss.msc@1.4.11.Final//org.jboss.msc.service.ServiceBuilderImpl.install(ServiceBuilderImpl.java:192)
        at org.jboss.as.controller@10.0.3.Final//org.jboss.as.controller.AbstractControllerService.start(AbstractControllerService.java:357)
        at org.jboss.as.server@10.0.3.Final//org.jboss.as.server.ServerService.start(ServerService.java:292)
        at org.jboss.msc@1.4.11.Final//org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1739)
        at org.jboss.msc@1.4.11.Final//org.jboss.msc.service.ServiceControllerImpl$StartTask.execute(ServiceControllerImpl.java:1701)
        ... 6 more

java.util.concurrent.ExecutionException: JBTHR00005: Operation failed
        at org.jboss.threads@2.3.3.Final//org.jboss.threads.AsyncFutureTask.get(AsyncFutureTask.java:253)
        at org.jboss.as.server@10.0.3.Final//org.jboss.as.server.Main.main(Main.java:109)
        at org.jboss.modules.Module.run(Module.java:352)
        at org.jboss.modules.Module.run(Module.java:320)
        at org.jboss.modules.Main.main(Main.java:593)
Caused by: org.jboss.msc.service.StartException in service jboss.as.server-controller: Failed to start service
        at org.jboss.msc@1.4.11.Final//org.jboss.msc.service.ServiceControllerImpl$StartTask.execute(ServiceControllerImpl.java:1731)
        at org.jboss.msc@1.4.11.Final//org.jboss.msc.service.ServiceControllerImpl$ControllerTask.run(ServiceControllerImpl.java:1559)
        at org.jboss.threads@2.3.3.Final//org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
        at org.jboss.threads@2.3.3.Final//org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1982)
        at org.jboss.threads@2.3.3.Final//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486)
        at org.jboss.threads@2.3.3.Final//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1363)
        at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.lang.IllegalStateException: Container is down
        at org.jboss.msc@1.4.11.Final//org.jboss.msc.service.ServiceContainerImpl.install(ServiceContainerImpl.java:739)
        at org.jboss.msc@1.4.11.Final//org.jboss.msc.service.ServiceTargetImpl.install(ServiceTargetImpl.java:260)
        at org.jboss.msc@1.4.11.Final//org.jboss.msc.service.ServiceControllerImpl$ChildServiceTarget.install(ServiceControllerImpl.java:2065)
        at org.jboss.msc@1.4.11.Final//org.jboss.msc.service.ServiceBuilderImpl.install(ServiceBuilderImpl.java:192)
        at org.jboss.as.controller@10.0.3.Final//org.jboss.as.controller.AbstractControllerService.start(AbstractControllerService.java:357)
        at org.jboss.as.server@10.0.3.Final//org.jboss.as.server.ServerService.start(ServerService.java:292)
        at org.jboss.msc@1.4.11.Final//org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1739)
        at org.jboss.msc@1.4.11.Final//org.jboss.msc.service.ServiceControllerImpl$StartTask.execute(ServiceControllerImpl.java:1701)
        ... 6 more
10:11:41,134 INFO  [org.jboss.as] (MSC service thread 1-1) WFLYSRV0050: Keycloak 9.0.0 (WildFly Core 10.0.3.Final) stopped in 442ms
*** JBossAS process (325) received TERM signal ***

keycloak deployment

apiVersion: apps/v1
kind: Deployment
metadata:
   name: keycloak-deployment
   labels:
      app: keycloak
spec:
   replicas: 1
   selector:
      matchLabels:
         app: keycloak
   template:
      metadata:
         labels:
           app: keycloak
      spec:
         containers:
         - name: keyclock
           image: jboss/keycloak:latest
           env:
           - name: KEYCLOAK_LOGLEVEL
             value: ALL
           - name: ROOT_LOGLEVEL
             value: ALL
           - name: KEYCLOAK_USER
             value: admin
           - name: KEYCLOAK_PASSWORD
             value: admin
           - name: DB_VENDOR
             value: mysql
           - name: DB_ADDR
             value: mysql-service
           - name: DB_DATABASE
             value: keycloak
           - name: DB_USER
             value: keycloak
           - name: DB_PASSWORD
             value: keycloak
           ports:
           - containerPort: 8080

MySQL Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
   name: mysql-deployment
   labels:
      app: mysql
spec:
   replicas: 1
   selector:
      matchLabels:
         app: mysql
   template:
      metadata:
         labels:
           app: mysql
      spec:
         containers:
         - name: mysql
           image: mysql:5.7
           env:
           - name: MYSQL_ROOT_PASSWORD
             value: root
           ports:
           - name: mysql-port
             containerPort: 3306

MySQL Service

apiVersion: v1
kind: Service
metadata:
  name: mysql-service
spec:
  selector:
    app: mysql
  strategy:
    type: Recreate
  ports:
  - port: 3306
    targetPort: mysql-port
    protocol: TCP
  type: NodePort

-- Jroddev

jboss

keycloak

kubernetes

mysql

wildfly

1 Answer

3/11/2020

After rebuilding the cluster the DEBUG logs started to come through and so did more error messages.

The problem turned out to be an SSL issue Establishing SSL connection without server's identity verification is not recommended.

I have added this to the keycloak env

- name: JDBC_PARAMS
  value: "useSSL=false"

It's not a proper solution, but it will get me by until I setup certificates in Kubernetes.