I have deployed the hyperledger fabric blockchain on k8s with 3 peer nodes and 2 orderer nodes and 2 external chaincodes.

Hyperledger Fabric v2.3.3 | Fabric-Peer v2.3

For the peers I have assigned resources as:

resources:
        limits:
          memory: "800Mi"
          cpu: "350m"
        requests:
          memory: "750Mi"
          cpu: "300m"

While doing some performance testing I have noticed the following:

• The memory usage keeps increasing with the number of concurrent transactions, which is expected given that the messages are basically protobufs and wrapped in an envelop.
• I invoked about 500 concurrent transactions and soon the peer pods hit the memory limit and became unavailable which led to multiple failures - greylisting of peers, failure of endorsement policies etc.

Now after sometimes, when there's no more transactions in the queue, then peer should release the memory? But, that's not happening, the peer pod is still using the maximum memory. Even after 1 hour the peer memory usage is maximum

Also I have prometheus scraping all the metrics endpoints of peer and orderer nodes at an interval of 15s. And the client invokes one of the chaincode every 5 minutes (for some running total).

Are there any any environment settings which can be configured to get the optimal performance?