I have a sample SCDF dataflow stream application which am able to deploy and get it running in kubernetes.But i rely on the dataflow dashboard to create application which points to the docker images and then define stream and add properties and deploy it.Now i am trying to automate the manual steps of creating the apps in the dataflow server console , create stream definition and deploy it .Is there any sample project or reference for the same.
There are a few options here.
1) We rely on SCDF's Java DSL to automate and acceptance-test the application registration, creation and deployment of streams and tasks on all the supported platforms. You will take spring-cloud-dataflow-rest-client as a dependency in your custom Java/Boot applications; with that, you could gain access to DataFlowOperations, which will allow you to perform all the actions that you can via Shell or the UI. Have a look at TickTockTests.java to get an idea. In fact, we use this pattern internally for our integration-tests, too (see: DockerComposeIT.java).
2) REST APIs: You could use the stream/task's create, deploy/launch, or stream update APIs as part of your CI tooling and the workflows. You have the flexibility to either invoke them via scripts or by other means. This concept was presented at a webinar — it might give you some ideas.
There are a few ways to automate your stream creation and deployment processes. The Spring Cloud Data Flow Dashboard is just one of the REST clients of SCDF server. Spring Cloud Data Flow ecosystem provides few other REST client implementations.
Shell interface: https://docs.spring.io/spring-cloud-dataflow/docs/current/reference/htmlsingle/#shell
Using this, you can create a script file that has all the stream creation and deployment commands. You can execute this script from one environment to other environment (say dev to staging to production). The only caveat with this approach is that, the operations that are sent by the shell commands are mostly asynchronous in nature and hence you need to add explicit checks or timeout or sleep to make sure you run the shell commands step by step.
Java DSL
SCDF REST client also has a Java DSL as in here. You can programmatically create the stream and deploy them using an explicit application.