Spinnaker setup on Oracle Kubernetes Cluster and Object Storage

11/1/2019

Fresh Spninnaker install using manifest based Oracle Kubernetes Provider with Oracle Object Storage.

I can see that all the services except spin-front50 and spin-echo are up and running.

NAME                               READY   STATUS             RESTARTS   AGE
spin-clouddriver-555c68f4f-z7clr   1/1     Running            0          5d16h
spin-deck-b58644c5f-rqlw6          1/1     Running            0          5d16h
spin-echo-5d7f95dcd9-lc4mp         0/1     CrashLoopBackOff   2048       5d16h
spin-front50-5df6f9d64f-r5gq8      0/1     CrashLoopBackOff   215        13h
spin-gate-79b5c66466-tbvpw         1/1     Running            0          5d16h
spin-orca-5b7d8b9bf7-d8q42         1/1     Running            0          5d16h
spin-redis-7bcc7bf5c4-zzlf4        1/1     Running            0          5d16h
spin-rosco-7b564755c9-25gt5        1/1     Running            0          5d16h

I'm able to hit the UI, but not able to create the application.

There are no error in the spin-front50 pod logs, on describe found readiness probe failing with below error:

  Warning  Unhealthy  16m (x14 over 19m)      kubelet, 10.0.11.2  Readiness probe failed: wget: can't connect to remote host (127.0.0.1): Connection refused
  Warning  Unhealthy  14m (x10 over 18m)      kubelet, 10.0.11.2  Liveness probe failed: wget: can't connect to remote host (127.0.0.1): Connection refused

Spin-front50 pod log:

  .   ____          _            __ _ _
 /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
 \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
  '  |____| .__|_| |_|_| |_\__, | / / / /
 =========|_|==============|___/=/_/_/_/
 :: Spring Boot ::        (v2.1.7.RELEASE)

2019-11-01 06:48:44.416  INFO 1 --- [           main] com.netflix.spinnaker.front50.Main       : Starting Main on spin-front50-5df6f9d64f-r5gq8 with PID 1 (/opt/front50/lib/front50-web.jar started by spinnaker in /)
2019-11-01 06:48:44.419  INFO 1 --- [           main] com.netflix.spinnaker.front50.Main       : The following profiles are active: test,local
2019-11-01 06:48:46.481  INFO 1 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'pollingScheduler' of type [com.netflix.config.FixedDelayPollingScheduler] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2019-11-01 06:48:46.484  INFO 1 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'polledConfigurationSource' of type [com.netflix.spinnaker.kork.archaius.SpringEnvironmentPolledConfigurationSource] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2019-11-01 06:48:46.510  WARN 1 --- [           main] c.n.c.sources.URLConfigurationSource     : No URLs will be polled as dynamic configuration sources.
2019-11-01 06:48:46.510  INFO 1 --- [           main] c.n.c.sources.URLConfigurationSource     : To enable URLs as dynamic configuration sources, define System property archaius.configurationSource.additionalUrls or make config.properties available on classpath.
2019-11-01 06:48:46.539  INFO 1 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 's3SecretEngine' of type [com.netflix.spinnaker.kork.secrets.engines.S3SecretEngine] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2019-11-01 06:48:46.540  INFO 1 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'noopSecretEngine' of type [com.netflix.spinnaker.kork.secrets.engines.NoopSecretEngine] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2019-11-01 06:48:46.542  INFO 1 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'secretEngineRegistry' of type [com.netflix.spinnaker.kork.secrets.SecretEngineRegistry] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2019-11-01 06:48:46.543  INFO 1 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'secretManager' of type [com.netflix.spinnaker.kork.secrets.SecretManager] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2019-11-01 06:48:46.620  INFO 1 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.security.config.annotation.configuration.ObjectPostProcessorConfiguration' of type [org.springframework.security.config.annotation.configuration.ObjectPostProcessorConfiguration$EnhancerBySpringCGLIB$894939b0] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2019-11-01 06:48:46.628  INFO 1 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'objectPostProcessor' of type [org.springframework.security.config.annotation.configuration.AutowireBeanFactoryObjectPostProcessor] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2019-11-01 06:48:46.630  INFO 1 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.security.access.expression.method.DefaultMethodSecurityExpressionHandler@7b4b8199' of type [org.springframework.security.access.expression.method.DefaultMethodSecurityExpressionHandler] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2019-11-01 06:48:46.633  INFO 1 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.security.config.annotation.method.configuration.GlobalMethodSecurityConfiguration' of type [org.springframework.security.config.annotation.method.configuration.GlobalMethodSecurityConfiguration$EnhancerBySpringCGLIB$ae1ddc62] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2019-11-01 06:48:46.637  INFO 1 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'methodSecurityMetadataSource' of type [org.springframework.security.access.method.DelegatingMethodSecurityMetadataSource] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2019-11-01 06:48:47.021  INFO 1 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat initialized with port(s): 8080 (http)
2019-11-01 06:48:47.062  INFO 1 --- [           main] o.apache.catalina.core.StandardService   : Starting service [Tomcat]
2019-11-01 06:48:47.062  INFO 1 --- [           main] org.apache.catalina.core.StandardEngine  : Starting Servlet engine: [Apache Tomcat/9.0.22]
2019-11-01 06:48:47.233  INFO 1 --- [           main] o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring embedded WebApplicationContext
2019-11-01 06:48:47.233  INFO 1 --- [           main] o.s.web.context.ContextLoader            : Root WebApplicationContext: initialization completed in 2657 ms
2019-11-01 06:48:47.886  INFO 1 --- [           main] c.o.b.h.s.internal.BouncyCastleHelper    : Instantiated provider: org.bouncycastle.jce.provider.BouncyCastleProvider
2019-11-01 06:48:48.644  INFO 1 --- [           main] c.n.s.f.m.a.DefaultApplicationDAO        : Warming Cache
2019-11-01 06:48:48.975  INFO 1 --- [           main] o.s.s.c.ThreadPoolTaskScheduler          : Initializing ExecutorService 'taskScheduler'
2019-11-01 06:48:49.007  INFO 1 --- [           main] c.n.s.f.model.project.DefaultProjectDAO  : Warming Cache
2019-11-01 06:48:49.093  INFO 1 --- [           main] c.n.s.f.m.pipeline.DefaultPipelineDAO    : Warming Cache
2019-11-01 06:48:49.170  INFO 1 --- [           main] c.n.s.f.m.p.DefaultPipelineStrategyDAO   : Warming Cache
2019-11-01 06:48:49.491  INFO 1 --- [           main] .s.f.m.a.DefaultApplicationPermissionDAO : Warming Cache
2019-11-01 06:48:49.798  INFO 1 --- [           main] c.n.s.f.m.p.DefaultPipelineTemplateDAO   : Warming Cache
2019-11-01 06:48:49.943  INFO 1 --- [           main] c.n.s.f.m.n.DefaultNotificationDAO       : Warming Cache
2019-11-01 06:48:50.293  INFO 1 --- [           main] c.n.s.f.m.snapshot.DefaultSnapshotDAO    : Warming Cache
2019-11-01 06:48:50.358  INFO 1 --- [           main] c.n.s.f.m.s.DefaultServiceAccountDAO     : Warming Cache
2019-11-01 06:48:50.970  INFO 1 --- [           main] o.s.b.a.e.web.EndpointLinksResolver      : Exposing 2 endpoint(s) beneath base path ''
2019-11-01 06:48:51.120  INFO 1 --- [           main] .s.s.UserDetailsServiceAutoConfiguration : 

Using generated security password: ff98d6fd-5f53-473a-9a1c-b99ec7c76537

2019-11-01 06:48:51.229  INFO 1 --- [           main] o.s.s.web.DefaultSecurityFilterChain     : Creating filter chain: org.springframework.boot.actuate.autoconfigure.security.servlet.EndpointRequest$EndpointRequestMatcher@37b5a51c, [org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter@4e782401, org.springframework.security.web.context.SecurityContextPersistenceFilter@3085befb, org.springframework.security.web.header.HeaderWriterFilter@10bcbbce, org.springframework.security.web.csrf.CsrfFilter@6fe337a5, org.springframework.security.web.authentication.logout.LogoutFilter@1a712f12, org.springframework.security.web.savedrequest.RequestCacheAwareFilter@2bc0603f, org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter@57ac236a, org.springframework.security.web.authentication.AnonymousAuthenticationFilter@2bb84aa5, org.springframework.security.web.session.SessionManagementFilter@3fdede3a, org.springframework.security.web.access.ExceptionTranslationFilter@38c1b1a7, org.springframework.security.web.access.intercept.FilterSecurityInterceptor@1abea1ed]
2019-11-01 06:48:51.231  INFO 1 --- [           main] o.s.s.web.DefaultSecurityFilterChain     : Creating filter chain: Ant [pattern='/spectator/metrics'], [org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter@3fc051ce, org.springframework.security.web.context.SecurityContextPersistenceFilter@5d449307, org.springframework.security.web.header.HeaderWriterFilter@10ae002e, org.springframework.security.web.csrf.CsrfFilter@3407ded1, org.springframework.security.web.authentication.logout.LogoutFilter@aca594d, org.springframework.security.web.savedrequest.RequestCacheAwareFilter@44e7c06b, org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter@24a807a9, org.springframework.security.web.authentication.AnonymousAuthenticationFilter@3c5bb37d, org.springframework.security.web.session.SessionManagementFilter@573f2e67, org.springframework.security.web.access.ExceptionTranslationFilter@63af52a6, org.springframework.security.web.access.intercept.FilterSecurityInterceptor@35d8ba22]
2019-11-01 06:48:51.232  INFO 1 --- [           main] o.s.s.web.DefaultSecurityFilterChain     : Creating filter chain: any request, [org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter@180fb796, com.netflix.spinnaker.fiat.shared.FiatAuthenticationFilter@7fa8fff, org.springframework.security.web.authentication.AnonymousAuthenticationFilter@4423692a, org.springframework.security.web.access.ExceptionTranslationFilter@79ae3fb1]
2019-11-01 06:48:51.266  INFO 1 --- [           main] c.n.s.f.m.d.DefaultDeliveryRepository    : Warming Cache
2019-11-01 06:48:51.736  INFO 1 --- [           main] c.n.s.config.HystrixSpectatorConfig      : Enabling HystrixSpectatorPublisher
2019-11-01 06:48:51.795  INFO 1 --- [           main] pertySourcedRequestMappingHandlerMapping : Mapped URL path [/v2/api-docs] onto method [public org.springframework.http.ResponseEntity<springfox.documentation.spring.web.json.Json> springfox.documentation.swagger2.web.Swagger2Controller.getDocumentation(java.lang.String,javax.servlet.http.HttpServletRequest)]
2019-11-01 06:48:51.919  INFO 1 --- [           main] o.s.s.concurrent.ThreadPoolTaskExecutor  : Initializing ExecutorService 'applicationTaskExecutor'
2019-11-01 06:48:52.316  INFO 1 --- [           main] d.s.w.p.DocumentationPluginsBootstrapper : Context refreshed
2019-11-01 06:48:52.338  INFO 1 --- [           main] d.s.w.p.DocumentationPluginsBootstrapper : Found 1 custom documentation plugin(s)
2019-11-01 06:48:52.422  INFO 1 --- [           main] s.d.s.w.s.ApiListingReferenceScanner     : Scanning for api listing references
2019-11-01 06:49:34.601  INFO 1 --- [           main] .d.s.w.r.o.CachingOperationNameGenerator : Generating unique operation named: batchUpdateUsingPOST_1
2019-11-01 06:49:54.459  INFO 1 --- [           main] .d.s.w.r.o.CachingOperationNameGenerator : Generating unique operation named: createUsingPOST_1
2019-11-01 06:49:54.460  INFO 1 --- [           main] .d.s.w.r.o.CachingOperationNameGenerator : Generating unique operation named: deleteUsingDELETE_1
2019-11-01 06:49:59.499  INFO 1 --- [           main] .d.s.w.r.o.CachingOperationNameGenerator : Generating unique operation named: batchUpdateUsingPOST_2
2019-11-01 06:49:59.501  INFO 1 --- [           main] .d.s.w.r.o.CachingOperationNameGenerator : Generating unique operation named: deleteUsingDELETE_2
-- Rajat Mahajan
installation
kubernetes
manifest
oracle
spinnaker

2 Answers

11/1/2019

I had liveness probe enabled, with --liveness-probe-initial-delay-seconds set to 60 seconds and after changing it to a higher value, the services worked.

-- Rajat Mahajan
Source: StackOverflow

11/1/2019

Halyard is recommended way to install Spinnaker. I had issues installing from manifests. suggest you use halyard

-- P Ekambaram
Source: StackOverflow