I may be encountering the same issue described in Horizon: does not exit if database connection fails #898 (https://github.com/stellar/go/issues/898) but with a different set up scenario.

I am in the process of migrating https://github.com/satoshipay/docker-stellar-horizon Docker Compose definitions to Kubernetes. I have been able to migrate most of the set up but hitting a problem with Horizon where the DB is not getting created during startup. I believe I have stellar core with the dependency on Postgres working as designed and the DB created as part of startup but the set up is different for Horizon.

The current issue I am hitting is the following...

Horizon Server Pod Logs

todkapmcbookpro:kubernetes todd$ kubectl get pods
NAME                                READY     STATUS             RESTARTS   AGE
postgres-horizon-564d479db4-2xvqd   1/1       Running            0          20m
postgres-sc-9f5f7fb4-prlpr          1/1       Running            0          22m
stellar-core-7ff77b4db8-tx4mt       1/1       Running            0          18m
stellar-horizon-6cff98554b-d7djn    0/1       CrashLoopBackOff   8          18m
todkapmcbookpro:kubernetes todd$ kubectl logs stellar-horizon-6cff98554b-d7djn
Initializing Horizon database...
2019/05/02 12:58:09 connect failed: pq: database "stellar-horizon" does not exist
Horizon database initialization failed (possibly because it has been done before)
2019/05/02 12:58:09 pq: database "stellar-horizon" does not exist
todkapmcbookpro:kubernetes todd$ 

Horizon Postgres DB pod logs

todkapmcbookpro:kubernetes todd$ kubectl logs postgres-horizon-564d479db4-2xvqd
2019-05-02 12:40:06.424 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
2019-05-02 12:40:06.424 UTC [1] LOG:  listening on IPv6 address "::", port 5432
2019-05-02 12:40:06.437 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2019-05-02 12:40:06.444 UTC [23] LOG:  database system was interrupted; last known up at 2019-05-02 12:38:19 UTC
2019-05-02 12:40:06.453 UTC [23] LOG:  database system was not properly shut down; automatic recovery in progress
2019-05-02 12:40:06.454 UTC [23] LOG:  redo starts at 0/1636FB8
2019-05-02 12:40:06.454 UTC [23] LOG:  invalid record length at 0/1636FF0: wanted 24, got 0
2019-05-02 12:40:06.454 UTC [23] LOG:  redo done at 0/1636FB8
2019-05-02 12:40:06.459 UTC [1] LOG:  database system is ready to accept connections
2019-05-02 12:42:35.675 UTC [30] FATAL:  database "stellar-horizon" does not exist
2019-05-02 12:42:35.690 UTC [31] FATAL:  database "stellar-horizon" does not exist
2019-05-02 12:42:37.123 UTC [32] FATAL:  database "stellar-horizon" does not exist
2019-05-02 12:42:37.136 UTC [33] FATAL:  database "stellar-horizon" does not exist
2019-05-02 12:42:50.131 UTC [34] FATAL:  database "stellar-horizon" does not exist
2019-05-02 12:42:50.153 UTC [35] FATAL:  database "stellar-horizon" does not exist
2019-05-02 12:43:16.094 UTC [36] FATAL:  database "stellar-horizon" does not exist
2019-05-02 12:43:16.115 UTC [37] FATAL:  database "stellar-horizon" does not exist
2019-05-02 12:43:57.097 UTC [38] FATAL:  database "stellar-horizon" does not exist
2019-05-02 12:43:57.111 UTC [39] FATAL:  database "stellar-horizon" does not exist
2019-05-02 12:45:21.050 UTC [40] FATAL:  database "stellar-horizon" does not exist
2019-05-02 12:45:21.069 UTC [41] FATAL:  database "stellar-horizon" does not exist
2019-05-02 12:48:05.122 UTC [42] FATAL:  database "stellar-horizon" does not exist
2019-05-02 12:48:05.145 UTC [43] FATAL:  database "stellar-horizon" does not exist
2019-05-02 12:53:07.077 UTC [44] FATAL:  database "stellar-horizon" does not exist
2019-05-02 12:53:07.099 UTC [45] FATAL:  database "stellar-horizon" does not exist
2019-05-02 12:58:09.084 UTC [46] FATAL:  database "stellar-horizon" does not exist
2019-05-02 12:58:09.098 UTC [47] FATAL:  database "stellar-horizon" does not exist
2019-05-02 13:03:18.055 UTC [48] FATAL:  database "stellar-horizon" does not exist
2019-05-02 13:03:18.071 UTC [49] FATAL:  database "stellar-horizon" does not exist
2019-05-02 13:08:28.057 UTC [50] FATAL:  database "stellar-horizon" does not exist
2019-05-02 13:08:28.078 UTC [51] FATAL:  database "stellar-horizon" does not exist
2019-05-02 13:13:42.071 UTC [52] FATAL:  database "stellar-horizon" does not exist
2019-05-02 13:13:42.097 UTC [53] FATAL:  database "stellar-horizon" does not exist
2019-05-02 13:18:55.128 UTC [54] FATAL:  database "stellar-horizon" does not exist
2019-05-02 13:18:55.152 UTC [55] FATAL:  database "stellar-horizon" does not exist

It would be ideal if the setup for Horizon and Core were the same (especially as it relates to the DB configuration env properties). I think I have the settings correct but may be missing something subtle.

I have a branch of this WIP where the failure occurs. I have included a quick set up script as well as a minikube set up in this branch. https://github.com/todkap/stellar-testnet/tree/k8-deploy/kubernetes