My Kubernetes cluster is used for running my graph database (Dgraph). However, I have to load the initial dataset (1TB) that comes as different folders and files into Dgraph.
I've processed the data locally and can now upload the files to my 6 different SSD persistent disks -- is there a way I can do so directly to the disks or do I need to use a Compute Engine Instance and go through it by mounting the disks and then unmounting them?
I have one suggestion which may be quicker and more simple than the method you mention in the post.
Presumably you have persistent disk claims mounted to the pods which will make use of this data.
For example, let's say you have a persistent disk claim mounted to /mnt/data on a pod.
It's possible to copy files to pods using the 'kubectl cp' command. I realise that the dataset you want to upload is very large and would fill a pods standard filesystem. However, if you have the persistent disk claim mounted to the pod that will contain the data the pod utilises, presumably this mounted storage is large enough for that data. You could therefore try using 'kubectl cp' to copy the data to the mount point on the pod, so that it lands on the mounted volume.
You can run this command to try this out:
kubectl cp datafile.csv NAMESPACE_NAME/POD_NAME:/mnt/dataOther than that, you could consider uploading the data to Cloud Storage using gsutil, then install fuse on the nodes as mentioned here which would allow you to mount the Cloud Storage to pods that need access to the data, although I realise this may not suit everyone use case.