The data needs to be loaded periodically (like once in a day), and it should be stored in SQL format so that the API can run SQL queries. We are thinking of loading it from HDFS. Currently we are thinking of using Apache Nifi using PutIgniteCache.
I was thinking probably I can launch a remote Ignite client node and then use IgniteDataStreamer to stream the data, but I was not able to find proper documentation for that. Are there any better ways to do this?
The documentation for Nifi says that it uses the data streamer API, so unless you need more control it doesn’t seem like a bad option (with the caveat that I’d never heard of Nifi before much less used it!).