I tried deploying Spinnaker using halyard and running into issue with clouddriver.
Have added a corporate Docker registry which needs corporate CA.
Clouddriver was failing with the below error.
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141) ~[na:1.8.0_212]
at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126) ~[na:1.8.0_212]
at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280) ~[na:1.8.0_212]
at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:392) ~[na:1.8.0_212]
... 100 common frames omitted
2019-08-07 04:53:14.237 ERROR 1 --- [0.0-7002-exec-3] c.n.s.k.w.e.GenericExceptionHandlers : Internal Server Error
com.netflix.spinnaker.clouddriver.core.AlwaysUpHealthIndicator$HealthIndicatorWrappedException: retrofit.RetrofitError: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at com.netflix.spinnaker.clouddriver.core.AlwaysUpHealthIndicator.health(AlwaysUpHealthIndicator.java:49) ~[clouddriver-core.jar:na]
at org.springframework.boot.actuate.health.CompositeHealthIndicator.health(CompositeHealthIndicator.java:95) ~[spring-boot-actuator-2.1.6.RELEASE.jar:2.1.6.RELEASE]
at org.springframework.boot.actuate.health.HealthEndpoint.health(HealthEndpoint.java:50) ~[spring-boot-actuator-2.1.6.RELEASE.jar:2.1.6.RELEASE]
at org.springframework.boot.actuate.health.HealthEndpointWebExtension.health(HealthEndpointWebExtension.java:53) ~[spring-boot-actuator-2.1.6.RELEASE.jar:2.1.6.RELEASE]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_212]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_212]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_212]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_212]
at org.springframework.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:282) ~[spring-core-5.1.8.RELEASE.jar:5.1.8.RELEASE]
at org.springframework.boot.actuate.endpoint.invoke.reflect.ReflectiveOperationInvoker.invoke(ReflectiveOperationInvoker.java:76) ~[spring-boot-actuator-2.1.6.RELEASE.jar:2.1.6.RELEASE]
at org.springframework.boot.actuate.endpoint.annotation.AbstractDiscoveredOperation.invoke(AbstractDiscoveredOperation.java:60) ~[spring-boot-actuator-2.1.6.RELEASE.jar:2.1.6.RELEASE]
at org.springframework.boot.actuate.endpoint.web.servlet.AbstractWebMvcEndpointHandlerMapping$ServletWebOperationAdapter.handle(AbstractWebMvcEndpointHandlerMapping.java:278) ~[spring-boot-actuator-2.1.6.RELEASE.jar:2.1.6.RELEASE]
So, created secret with cacerts which is imported with corporate CA based on the below link.
https://www.spinnaker.io/reference/halyard/custom/#using-custom-volumes
In ~/.hal/default/service-settings/clouddriver.yml,
kubernetes:
volumes:
- id: spin-truststore
type: secret
mountPath: /app/certs/
But, still getting the same cert error.
Also, tried the below approach.
Added the below entry ~/.hal/default/profile/clouddriver-local.yml along with the above volumes.
okHttpClient:
enabled: true
keyStore: /app/certs/cacert
keyStorePassword: changeit
trustStore: /app/certs/cacert
trustStorePassword: changeit
propagateSpinnakerHeaders: true
connectTimeoutMs: 60000
readTimeoutMs: 60000
Now, getting stream too big error.
Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [retrofit.client.OkClient]: Factory method 'okClient' threw exception; nested exception is java.io.IOException: DerInputStream.getLength(): lengthTag=109, too big.
at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:185) ~[spring-beans-5.1.8.RELEASE.jar:5.1.8.RELEASE]
at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:622) ~[spring-beans-5.1.8.RELEASE.jar:5.1.8.RELEASE]
... 107 common frames omitted
Caused by: java.io.IOException: DerInputStream.getLength(): lengthTag=109, too big.
at sun.security.util.DerInputStream.getLength(DerInputStream.java:599) ~[na:1.8.0_212]
at sun.security.util.DerValue.init(DerValue.java:391) ~[na:1.8.0_212]
at sun.security.util.DerValue.<init>(DerValue.java:332) ~[na:1.8.0_212]
at sun.security.util.DerValue.<init>(DerValue.java:345) ~[na:1.8.0_212]
at sun.security.pkcs12.PKCS12KeyStore.engineLoad(PKCS12KeyStore.java:1938) ~[na:
Update
The cert error is gone now after updating the mount path as below. Here, I did not add the file clouddriver-local.yml.
mountPath: /etc/ssl/certs/java
But, getting different errors now,
2019-08-07 06:09:55.364 ERROR 1 --- [ecutionAction-2] .d.r.p.a.DockerRegistryImageCachingAgent : Could not load tags for gcp-spinnaker/spinnaker-marketplace/front50 in https://docker.xyz.com
retrofit.RetrofitError: 429 Too Many Requests
at retrofit.RetrofitError.httpError(RetrofitError.java:40) ~[retrofit-1.9.0.jar:na]
at retrofit.RestAdapter$RestHandler.invokeRequest(RestAdapter.java:388) ~[retrofit-1.9.0.jar:na]
at retrofit.RestAdapter$RestHandler.invoke(RestAdapter.java:240) ~[retrofit-1.9.0.jar:na]
at com.sun.proxy.$Proxy134.getTags(Unknown Source) ~[na:na]
Also the below error.
Error from server (Forbidden): podsecuritypolicies.extensions is forbidden: User "XXXXXXXX" cannot list resource "podsecuritypolicies" in API group "extensions" at the cluster scope
Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "XXXXXX" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope
at com.netflix.spinnaker.clouddriver.kubernetes.v2.op.job.KubectlJobExecutor.list(KubectlJobExecutor.java:421) ~[clouddriver-kubernetes.jar:na]
at com.netflix.spinnaker.clouddriver.kubernetes.v2.security.KubernetesV2Credentials.lambda$list$18(KubernetesV2Credentials.java:464) ~[clouddriver-kubernetes.jar:na]
at com.netflix.spinnaker.clouddriver.kubernetes.v2.security.KubernetesV2Credentials.runAndRecordMetrics(KubernetesV2Credentials.java:598) ~[clouddriver-kubernetes.jar:na]
at com.netflix.spinnaker.clouddriver.kubernetes.v2.security.KubernetesV2Credentials.list(KubernetesV2Credentials.java:460) ~[clouddriver-kubernetes.jar:na]
at com.netflix.spinnaker.clouddriver.kubernetes.v2.caching.agent.KubernetesV2CachingAgent.lambda$loadPrimaryResourceList$0(KubernetesV2CachingAgent.java:88) ~[clouddriver-kubernetes.jar:na]
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) ~[na:1.8.0_212]
Solution 1:-
Create a custom image with the corporate cert imported.
I'm having the same problem. Apparently, the 'too big' error is due to a wrong crt-file. The keystore/truststore couldn't be read by keytool neither as JKS nor PKCS#12 keystore.
See here