Is it possible to directly modify the file mounted by configMap? We have an application that reads a configuration file that was configMap type and the application should able to edit the file and the changes should be persisted if the configMap is shared with other pods and when the pod restarts.
If configMap is not meant for this, then what should we relay on consul to save the configuration?
Yes a configmap is not intended to be writeable. If you're interacting with files from a configmap then you could instead put the files in a writeable volume and mount the volume. Or you could, as you suggest, use centralised configuration like consul. Given, that the app is dynamically writing to this data you could consider it state rather than configuration. Then it could be stored in a database. Another option could be a distributed cache such as redis or hazelcast.
AFAIK the changes to a ConfigMap will only exist locally in-memory.
That is, changes aren't visible to other pods and on a pod restart the changes will be lost.
One solution is to use the kubectl binary or the kubernetes API from within the configuring application to recreate the ConfigMap after the configuration changes.
e.g. kubectl apply -f /path/to/updated/config.yaml
From Kubernetes doc, it can be updated see link
When a ConfigMap already being consumed in a volume is updated, projected keys are eventually updated as well. Kubelet is checking whether the mounted ConfigMap is fresh on every periodic sync. However, it is using its local ttl-based cache for getting the current value of the ConfigMap. As a result, the total delay from the moment when the ConfigMap is updated to the moment when new keys are projected to the pod can be as long as kubelet sync period + ttl of ConfigMaps cache in kubelet.