Other then secrets having to be base64 encoded (which can be easily decoded) I am unaware of the differences between those two.

Is there any reason to not only rely on ConfigMaps to set secrets?

Are there security concerns?