I developed a .net core web API in which all requests are first inserted into database and then generated request Id is inserted into redis queue.

With Load testing application work flawless with IIS hosting on VM server with any number of load. When application is deployed on K8S with 3 PODS(each POD configuration 1 CPU and 4 GB RAM) application performance degrades when requests are greater than 750. With load of 1000 requests sometimes all requests executed successfully but takes 2 to 3 mins to complete all the requests and sometimes few requests got failed. When Load is greater than 1000 to 2000 number of failed request increase. Below are the errors I got randomly.

Error 1

No connection is available to service this operation: LPUSH GENEVAREQ; IOCP: (Busy=0,Free=1000,Min=4,Max=1000), WORKER: (Busy=12,Free=32755,Min=4,Max=32767), Local-CPU: n/a \n   at StackExchange.Redis.ConnectionMultiplexer.

Error 2

at System.Lazy`1.CreateValue()
at System.Lazy`1.get_Value()
at XYZ.Demo.RedisService.RedisConnectionFactory.Connection() in D:\TeamCity\buildAgent\work\abdf9bd4783a2774\XYZ.Demo.RedisService\RedisConnectionFactory.cs:line 26
at XYZ.Demo.RedisService.RedisQueueService`1..ctor(IRedisConnectionFactory connectionFactory) in D:\TeamCity\buildAgent\work\abdf9bd4783a2774\XYZ.Demo.RedisService\RedisQueueService.cs:line 22
at XYZ.DemoAPI.Services.RequestService.EnqueueRequest(String requestId, String name, String queueName, Boolean priority) in D:\TeamCity\buildAgent\work\abdf9bd4783a2774\DemoAPI\Services\RequestService.cs:line 205
at XYZ.DemoAPI.Services.RequestService.PostReportRequest(RequestParams p) in D:\TeamCity\buildAgent\work\abdf9bd4783a2774\DemoAPI\Services\RequestService.cs:line 127
at XYZ.DemoAPI.Controllers.BaseController`1.PushReportRequest[TResult,TRow,P](P param, IProcessRequest`3 service) in D:\TeamCity\buildAgent\work\abdf9bd4783a2774\DemoAPI\Controllers\BaseController.cs:line 54
at XYZ.DemoAPI.Controllers.DemoApiController.GetReport(String businessLine, String category, String firmId, String authorization, String authUserId, Object requestPayload, String requestTrackingId) in D:\TeamCity\buildAgent\work\abdf9bd4783a2774\DemoAPI\Controllers\DemoApiController.cs:line 57
at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.TaskOfIActionResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeActionMethodAsync>g__Awaited|12_0(ControllerActionInvoker invoker, ValueTask`1 actionResultValueTask)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeNextActionFilterAsync>g__Awaited|10_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeInnerFilterAsync()

Error 3

--- End of stack trace from previous location where exception was thrown ---

at System.Lazy`1.CreateValue()
at System.Lazy`1.get_Value()
at XYZ.Demo.RedisService.RedisConnectionFactory.Connection() in D:\TeamCity\buildAgent\work\abdf9bd4783a2774\XYZ.Demo.RedisService\RedisConnectionFactory.cs:line 26
at XYZ.Demo.RedisService.RedisQueueService`1..ctor(IRedisConnectionFactory connectionFactory) in D:\TeamCity\buildAgent\work\abdf9bd4783a2774\XYZ.Demo.RedisService\RedisQueueService.cs:line 22
at XYZ.DemoAPI.Services.RequestService.EnqueueRequest(String requestId, String name, String queueName, Boolean priority) in D:\TeamCity\buildAgent\work\abdf9bd4783a2774\GenevaAPI\Services\RequestService.cs:line 205

Error 4

A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server) \n   at Microsoft.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 waitForMultipleObjectsTimeout, Boolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternal& connection)\n   at Microsoft.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal& connection)\n   at Microsoft.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection)\n   at Microsoft.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)\n   at Microsoft.Data.SqlClient.SqlConnection.TryOpen(TaskCompletionSource`1 retry)\n   at Microsoft.Data.SqlClient.SqlConnection.Open()\n   at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.OpenDbConnection(Boolean errorsExpected)\n   at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.Open(Boolean errorsExpected)\n   at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReader(RelationalCommandParameterObject parameterObject)\n   at Microsoft.EntityFrameworkCore.Query.Internal.QueryingEnumerable`1.Enumerator.InitializeReader(DbContext _, Boolean result)\n   at Microsoft.EntityFrameworkCore.SqlServer.Storage.Internal.SqlServerExecutionStrategy.Execute[TState,TResult](TState state, Func`3 operation, Func`3 verifySucceeded)\n   at Microsoft.EntityFrameworkCore.Query.Internal.QueryingEnumerable`1.Enumerator.MoveNext()\n   at System.Linq.Enumerable.SingleOrDefault[TSource](IEnumerable`1 source)\n   at XYZ.Demo.BuisnessService.ValidateService.GetClusterServer(RequestParams requestParams, String& queueName, String& filePath) in D:\\TeamCity\\buildAgent\\work\\abdf9bd4783a2774\\XYZ.Demo.BuisnessService\\ValidateService.cs:line 220\n   at XYZ.DemoAPI.Services.RequestService.PostReportRequest(RequestParams p) in

Error 5

The timeout period elapsed during the post-login phase.  The connection could have timed out while waiting for server to complete the login process and respond; Or it could have timed out while attempting to create multiple active connections.  The duration spent while attempting to connect to this server was - [Pre-Login] initialization=7; handshake=1069; [Login] initialization=0; authentication=0; [Post-Login] complete=13032;  \n   at Microsoft.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 waitForMultipleObjectsTimeout, Boolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternal& connection)\n   at Microsoft.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal& connection)\n   at Microsoft.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection)\n   at Microsoft.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)\n   at Microsoft.Data.SqlClient.SqlConnection.TryOpen(TaskCompletionSource`1 retry)\n   at Microsoft.Data.SqlClient.SqlConnection.Open()\n   at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.OpenDbConnection(Boolean errorsExpected)\n   at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.Open(Boolean errorsExpected)\n   at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReader(RelationalCommandParameterObject parameterObject)\n   at Microsoft.EntityFrameworkCore.Query.Internal.QueryingEnumerable`1.Enumerator.InitializeReader(DbContext _, Boolean result)\n

Error 6

<html>
<head><title>504 Gateway Time-out</title></head>
<body>
<center><h1>504 Gateway Time-out</h1></center>
<hr><center>nginx/1.17.10</center>
</body>
</html>


<html><body><h1>504 Gateway Time-out</h1>
The server didn't respond in time.
</body></html>

Please suggest how to fix this issue.