According to this link, the nginx.ingress.kubernetes.io/limit-connections annotation sets the number of concurrent connections allowed from a single IP address. Is there a way I can sets the number of concurrent connections allowed in total?
Maybe are mistaking the concepts. This annotation is specific for handling with multiple connections with one client. It has security purpose (mitigate DDoS attacks).
As @Crou said, if you are receiving a large amount of connections, you could scale up. Or else, if you want to limit the connections to this service, you should define settings to not scale. But, you can run a out of service issue eventually.