Writing data directly to Google Cloud Storage from Kubernetes pods

6/18/2018

We are considering the alternatives to PubSub, due to high costs. For some of our low-value and high-volume data the PubSub can get quite expensive.

The plan of using PubSub:

  • Run the service in Kubernetes pod
  • Service pushes the data to PubSub
  • We add a PubSub subscriber that reads the data from PubSub and writes it to GCS

The data has a very low-value. So even if we loose some small % of it (due to pod restarts, etc) that should not be a problem.

Is it possible to write the data directly to GCS or perhaps there are other less-expensive alternatives?

Edit: just to clarify, we don't need to write the data in real-time. We can write it in batches every few minutes or so.

-- Datageek
google-cloud-pubsub
google-cloud-storage
google-kubernetes-engine

1 Answer

6/18/2018

You can directly write data to a bucket by mounting it by using gcefuse. You can read on how you can use gcefuse within GKE by following the directions here.

-- Jason
Source: StackOverflow