I've used the Bitnami Helm chart to install SCDF into a k8s cluster generated by kOps in AWS.
I'm trying to add my development SCDF stream apps into the installation using a file URI and cannot figure-out where or how the shared Skipper & Server mount point is. exec'ing into either instance there is no /home/cnb and I'm not seeing anything common via mount. The best I can tell the Bitnami installation is using the MariaDB instance for shared "storage".
Is there a recommended way of installing local/dev Stream apps into the cluster?
Building on the previous answer here is what I came-up with:
1. Create an EBS Volume
2. Mount it on each EC2 instance in the cluster at the same location (/cdf)
3. Install CDF using the Bitnami chart and this config file:
server.extraVolumeMounts:
# Locstion in container
- mountPath: /applications
# Refer to the volume below
name: application-volume
server.extraVolumes:
- name: application-volume
hostPath:
# Location in host filesystem
path: /cdf
# this field is optional
type: Directory
skipper.extraVolumeMounts:
# Locstion in container
- mountPath: /applications
# Refer to the volume below
name: application-volume
skipper.extraVolumes:
- name: application-volume
hostPath:
# Location in host filesystem
path: /cdf
# this field is optional
type: DirectoryThen I can copy my jars into /cdf on the host file system and install the applications using a file URI of file:///applications/<jar-file-name> and everything works.
There are a couple of parameters under the deployer section that allows you to mount volumes (link):
deployer:
## @param deployer.volumeMounts Streaming applications extra volume mounts
##
volumeMounts: {}
## @param deployer.volumes Streaming applications extra volumes
##
volumes: {}see https://github.com/bitnami/charts/tree/master/bitnami/spring-cloud-dataflow#deployer-parameters.
Then, the mounted volume is used in the ConfigMaps (both server and skipper):
Apart from that, there are also server.extraVolumes and server.extraVolumeMounts to be set on the Dataflow Server Pod, and skipper.extraVolumes and skipper.extraVolumeMounts to be set on the Skipper Pod just in case it's useful for your use case.