My application needs to make concurrent HTTP requests to a Third Party REST API. Due to a limitation on the API's side, I'm allowed to make at most 100 concurrent requests every two seconds. My goal is for all batches of 100 requests to take around the same time to complete (~2 seconds). The problem is that with my current implementation the first batch always takes considerably more time than the rest.

The application is a .NET Core 2.2 Console app that is running in Kubernetes.

I'm using a Helper Library that makes it easy for me to use their REST API. The signature of the method that I have to call 100 times concurrently looks like this:

public static async Task CreateResourceAsync()

This method makes an HTTP Request to their API, which creates a resource on their end and takes ~2 seconds to complete.

To execute the requests in batches of 100 every 2 seconds I wrote the following Code:

private ConcurrentQueue<Resource> _resourcesToCreate;

public async Task CreateResources(List<Resource> resources)
{
    ThreadPool.GetMinThreads(out _, out var minCompletionPortThreads);
    Console.WriteLine("Minimum number of IOCP Threads: {0}", 
    minCompletionPortThreads);

    _resourcesToCreate = resources.ToConcurrentQueue()

    while (!_resourcesToCreate.IsEmpty)
    {
        await ExecuteBatch(100);
        await _delayer.Delay(TimeSpan.FromSeconds(2));
    }
}

private async Task ExecuteBatch(int batchSize)
{
    var requests = new List<Task>();

    for (var i = 0; i < batchSize && !_resourcesToCreate.IsEmpty; i++)
    {
        _resourcesToCreate.TryDequeue(out var resource);

        requests.Add(CreateResource(resource));
    }

    var stopwatch = Stopwatch.StartNew();

    await Task.WhenAll(requests);

    stopwatch.Stop();

    Console.WriteLine($"[Execute Batch] [Complete] | took 
    {stopwatch.ElapsedMilliseconds}ms");
 }

private async Task CreateResource(Resource resource)
{

    var resourceCreated = await CreateResourceAsync();

    Console.WriteLine("Current IOCP thread ID: {0}", 
    Thread.CurrentThread.ManagedThreadId);

    resource.Id = resourceCreated.Sid;
}

I run five tests that consisted of making 500 requests and here are the results:

First Batch:

• Time Taken: 13088ms, 15170ms, 13536ms, 15398ms, 11700ms
• Number of threads: 5, 6, 5, 19, 3

Second Batch:

• Time Taken: 507ms, 402ms, 600ms, 402ms, 899ms
• Number of threads: 3, 6, 4, 26, 2

Third Batch:

• Time Taken: 306ms, 98ms, 501ms, 99ms, 1301ms
• Number of threads: 2, 4, 3, 27, 1

Fourth Batch:

• Time Taken: 303ms, 98ms, 496ms, 197ms, 196ms
• Number of threads: 2, 6, 3, 7, 2

Fifth Batch:

• Time Taken: 998ms, 108ms, 299ms, 194ms, 97ms
• Number of threads: 2, 5, 3, 7, 3

I know that getting the number of distinct Ids printed by this statement:

Console.WriteLine("Current IOCP thread ID: {0}", Thread.CurrentThread.ManagedThreadId);

Is most probably the wrong way to obtain the number of threads used during the execution of a batch. I tried using the method ThreadPool.GetAvailableThreads(), but it always returns 1000 IOCP threads when running in Kubernetes.

I would like to understand why the first batch is always taking more time than the rest and could it possibly be related to the number of threads used?