I am trying to set up HBase backed by S3 on K8s as a PoC and been racking my brains over this issue for 2 days now.

So far, I have managed to set up a pod with ZK, Hbase master, Hbase regionserver, Hbase rest and Hbase thrift services in a container; they are all started by Supervisord.

Everything runs fine when the first time this is run (when the S3 bucket is empty). However, if the pod crashes for any reason or is restarted, the master fails to initialise after the restart and is in a failed state. It keeps restarting with the error Caused by: java.io.IOException: Timedout 300000ms waiting for namespace table to be assigned and enabled: tableName=hbase:namespace, state=ENABLED after every 3 mins, and any commands on the Hbase shell results in a:

ERROR: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing
    at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2989)
    at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1964)
    at org.apache.hadoop.hbase.master.MasterRpcServices.createTable(MasterRpcServices.java:636)
    at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java)
    at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
    at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133)
    at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
    at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)

Online articles say that an unexpected regionserver crash can cause this which can be resolved by cleaning up Zookeeper, however, in my case, even Zookeeper has been restarted and hence starts afresh.

Is there no way just to have a quick Hbase on S3 in this manner?

What data/metadata does Hbase actually store in the persistent store that causes it to fail in this way?

Note: I do not run any Hadoop/HDFS services as I write directly to S3; the WAL is stored locally in the pod, which gets cleared on a restart.