Kubernete pod for doing maven build using best effort QoS and jdk 11 results in:
[ERROR] unable to create native thread: possibly out of memory or process/resource limits reached -> [Help 1]
java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached
at java.lang.Thread.start0 (Native Method)
at java.lang.Thread.start (Thread.java:803)
at java.util.concurrent.ThreadPoolExecutor.addWorker (ThreadPoolExecutor.java:937)
at java.util.concurrent.ThreadPoolExecutor.execute (ThreadPoolExecutor.java:1343)
at org.eclipse.aether.connector.basic.BasicRepositoryConnector.get (BasicRepositoryConnector.java:262)
at org.eclipse.aether.internal.impl.DefaultArtifactResolver.performDownloads (DefaultArtifactResolver.java:499)
at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolve (DefaultArtifactResolver.java:400)
at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolveArtifacts (DefaultArtifactResolver.java:225)
at org.jfrog.build.extractor.maven.resolver.ArtifactoryEclipseArtifactResolver.resolveArtifacts (ArtifactoryEclipseArtifactResolver.java:56)
at org.eclipse.aether.internal.impl.DefaultRepositorySystem.resolveDependencies (DefaultRepositorySystem.java:335)
at org.apache.maven.project.DefaultProjectDependenciesResolver.resolve (DefaultProjectDependenciesResolver.java:202)
at org.apache.maven.lifecycle.internal.LifecycleDependencyResolver.getDependencies (LifecycleDependencyResolver.java:243)
at org.apache.maven.lifecycle.internal.LifecycleDependencyResolver.resolveProjectDependencies (LifecycleDependencyResolver.java:147)
at org.apache.maven.lifecycle.internal.MojoExecutor.ensureDependenciesAreResolved (MojoExecutor.java:248)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:202)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:156)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:148)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)
at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)
at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)
at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)
at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
at org.jvnet.hudson.maven3.launcher.Maven35Launcher.main (Maven35Launcher.java:130)
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke (Method.java:566)
at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:282)
at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:225)
at jenkins.maven3.agent.Maven35Main.launch (Maven35Main.java:176)
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke (Method.java:566)
at hudson.maven.Maven3Builder.call (Maven3Builder.java:139)
at hudson.maven.Maven3Builder.call (Maven3Builder.java:70)
at hudson.remoting.UserRequest.perform (UserRequest.java:212)
at hudson.remoting.UserRequest.perform (UserRequest.java:54)
at hudson.remoting.Request$2.run (Request.java:369)
at hudson.remoting.InterceptingExecutorService$1.call (InterceptingExecutorService.java:72)
at java.util.concurrent.FutureTask.run (FutureTask.java:264)
at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1128)
at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:628)
at java.lang.Thread.run (Thread.java:834)Using Jenkins with Kubernetes (Azure's AKS) configured as a cloud service on a 4 CPU, 16GB vm. Pod templates are configured with a QoS as Best Effort and/or Guaranteed. The image used is from dockerhub jenkins/jnlp-slave:latest-jdk11. Maven 3.6.1. I've used a variety of JVM options from -Xmx256m to -Xmx3g to -XX:MaxRAM=3g -XX:MaxRAMPercentage=40
CPU request/limit of 0/0, 1000/0, 1000/1000, 2000/2000 Mem request/limit of 0/0, 2048/0, 2048/2048, 4096/4096
I've tried many different options, and I'll get a mix of out-of-memory issues from near the beginning of the maven build. Typically associated with lack of threads, but when I check ulimits, I have unlimited for just about everything. Sometimes I get successful builds one after another for a time, but then later I'll get nothing but out-of-memory failures.
I'm at my wits end and every article and tech document I read supposes that the jvm options I add should make jvm aware of containers and magically fix issues likes this, but they don't. Why am I constantly running into out-of-memory issues despite giving the pod almost 4 cores and 16 GB?
When I run the builds manually by connecting to the pod through kuberente's exec, the seem to run fine. This only happens with Jenkins.
ulimit -a output by request
jenkins@linux-base-image-jdk11-9w68h:~$ ulimit -a
core file size (blocks, -c) unlimited
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 64090
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1048576
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) unlimited
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited