Spinnaker services are not starting up after HTTPS configuration

1/9/2020

I've followed spinnaker HTTPS configuration as per https://www.spinnaker.io/setup/security/ssl/#server-terminated-ssl

After that spinnaker services echo, front50 and igor are not starting up properly, getting below output.

kubectl get po -n spinnaker
NAME                                READY   STATUS             RESTARTS   AGE
spin-clouddriver-5cd79d98d6-bhd4s   0/1     Evicted            0          3h34m
spin-clouddriver-5cd79d98d6-srnwl   1/1     Running            0          9m54s
spin-deck-5bd984bd9c-cx6xn          1/1     Running            0          3h34m
spin-echo-84848bfc46-2hwml          0/1     Running            219        17h
spin-echo-866c54bb8d-p4qbs          0/1     CrashLoopBackOff   37         3h34m
spin-front50-686f577f79-xld6d       0/1     Running            218        17h
spin-front50-988ccc789-662kn        0/1     Running            39         3h34m
spin-gate-5954b6cd88-rzqmd          1/1     Running            0          3h34m
spin-igor-5c8bd8c694-lr7z6          0/1     Running            226        17h
spin-igor-fd97cd9f6-q5wfs           0/1     Running            41         3h34m
spin-orca-546d5685dc-wswkj          1/1     Running            8          3h34m
spin-redis-58d4d69d77-st4bk         1/1     Running            0          29d
spin-rosco-fd6f6596d-2ht9r          1/1     Running            0          3h34m

Echo service

kubectl describe po -n spinnaker spin-echo-84848bfc46-2hwml

Warning  Unhealthy  5m35s (x1648 over 18h)  kubelet, aks-agentpool-30093549-vmss000015  Readiness probe failed:

Front50 service

kubectl describe po -n spinnaker spin-front50-988ccc789-662kn

Warning  Unhealthy  77s (x866 over 3h51m)  kubelet, aks-agentpool-30093549-vmss000014  Readiness probe failed: wget: can't connect to remote host (127.0.0.1): Connection refused

Igor service

kubectl describe po -n spinnaker spin-igor-fd97cd9f6-q5wfs

Warning  Unhealthy  17m (x558 over 3h52m)    kubelet, aks-agentpool-30093549-vmss000015  Readiness probe failed: wget: can't connect to remote host (127.0.0.1): Connection refused

Front50 logs stops after "Scanning for api listing references" and then restarts

2020-01-09 09:29:11.974  INFO 1 --- [           main] o.s.s.concurrent.ThreadPoolTaskExecutor  : Initializing ExecutorService 'applicationTaskExecutor'
2020-01-09 09:29:20.526  INFO 1 --- [           main] d.s.w.p.DocumentationPluginsBootstrapper : Context refreshed
2020-01-09 09:29:20.894  INFO 1 --- [           main] d.s.w.p.DocumentationPluginsBootstrapper : Found 1 custom documentation plugin(s)
2020-01-09 09:29:22.149  INFO 1 --- [           main] s.d.s.w.s.ApiListingReferenceScanner     : Scanning for api listing references

Echo service logs shows, it fails because can't connect to front50 service

Caused by: java.net.ConnectException: Failed to connect to spin-front50.spinnaker/ipAddress:8080

Similarly Igor service logs shows, it fails because can't connect to echo service

Caused by: java.net.ConnectException: Failed to connect to spin-echo.spinnaker/ipAddress:8089

I tried to increase liveness probe hal config deploy edit --liveness-probe-enabled true --liveness-probe-initial-delay-seconds 180 but still same.

Other front50 service logs are, not sure if related

2020-01-09 09:37:32.066  INFO 1 --- [           main] i.g.r.utils.RxJava2OnClasspathCondition  : RxJava2 related Aspect extensions are not activated, because RxJava2 is not on the classpath.
2020-01-09 09:37:32.067  INFO 1 --- [           main] i.g.r.utils.ReactorOnClasspathCondition  : Reactor related Aspect extensions are not activated because Reactor is not on the classpath.

Spinnaker is connected to Azure storage properly as it requires for front50 service. When I do hal config storage azs I get the response of configured storage

+ Get current deployment
  Success
+ Get persistent store
  Success
+ Successfully got persistent store "azs".
AzsPersistentStore(storageAccountName=xxx, storageAccountKey=xxx, storageContainerName=xxx)

Please advise. Thanks.

-- Jaydeep Soni
https
kubernetes
spinnaker
ssl

1 Answer

1/9/2020

I've again increased --liveness-probe-initial-delay-seconds to 600 sec and then it started working.

-- Jaydeep Soni
Source: StackOverflow