.NET 5 BackgroundService in Kubernetes Doesn't Exit

10/10/2021

I'm able to successfully run a .NET 5 Console Application with a BackgroundService in an Azure Kubernetes cluster on Ubuntu 18.04. In fact, the BackgroundService is all that really runs: just grabs messages from a queue, executes some actions, then terminates when Kubernetes tells it to stop, or the occasional exception.

It's this last scenario which is giving me problems. When the BackgroundService hits an unrecoverable exception, I'd like the container to stop (complete, or whatever state will cause Kubernetes to either restart or destroy/recreate the container).

Unfortunately, any time an exception is encountered, the BackgroundService appears to hit the StopAsync() function (from what I can see in the logs and console output), but the container stays in a running state and never restarts. My Main() is as appears below:

        public static async Task Main(string[] args)
        {
            // Build service host and execute.
            var host = CreateHostBuilder(args)
                .UseConsoleLifetime()
                .Build();

            // Attach application event handlers.
            AppDomain.CurrentDomain.ProcessExit += OnProcessExit;
            AppDomain.CurrentDomain.UnhandledException += new UnhandledExceptionEventHandler(OnUnhandledException);

            try
            {
                Console.WriteLine("Beginning WebSec.Scanner.");
                await host.StartAsync();
                await host.WaitForShutdownAsync();
                Console.WriteLine("WebSec.Scanner has completed.");
            }
            finally
            {
                Console.WriteLine("Cleaning up...");

                // Ensure host is properly disposed.
                if (host is IAsyncDisposable ad)
                {
                    await ad.DisposeAsync();
                }
                else if (host is IDisposable d)
                {
                    d.Dispose();
                }
            }
        }

If relevant, those event handlers for ProcessExit and UnhandledException exist to flush the AppInsights telemetry channel (maybe that's blocking it?):

        private static void OnProcessExit(object sender, EventArgs e)
        {
            // Ensure AppInsights logs are submitted upstream.
            Console.WriteLine("Flushing logs to AppInsights");
            TelemetryChannel.Flush();
        }

        private static void OnUnhandledException(object sender, UnhandledExceptionEventArgs e)
        {
            var thrownException = (Exception)e.ExceptionObject;
            Console.WriteLine("Unhandled exception thrown: {0}", thrownException.Message);

            // Ensure AppInsights logs are submitted upstream.
            Console.WriteLine("Flushing logs to AppInsights");
            TelemetryChannel.Flush();
        }

I am only overriding ExecuteAsync() in the BackgroundService:

        protected async override Task ExecuteAsync(CancellationToken stoppingToken)
        {
            this.logger.LogInformation(
                "Service started.");

            try
            {
                // Loop until the service is terminated.
                while (!stoppingToken.IsCancellationRequested)
                {
                    // Do some work...
                }
            }
            catch (Exception ex)
            {
                this.logger.LogWarning(
                    ex,
                    "Terminating due to exception.");
            }

            this.logger.LogInformation(
                "Service ending.",
        }

My Dockerfile is simple and has this line to run the service:

ENTRYPOINT ["dotnet", "MyService.dll"]

Am I missing something obvious? I feel like there's something about running this as a Linux container that I'm forgetting in order to make this run properly.

Thank you!

-- Cory Gehr
.net
background-service
c#
docker
kubernetes

1 Answer

10/11/2021

Here is a full example of how to use IHostApplicationLifetime.StopApplication().

void Main()
{
	var host = Host.CreateDefaultBuilder()
		.ConfigureServices((context, services) =>
		{
			services.AddHostedService<MyService>();
		})
		.Build();
	
	Console.WriteLine("Starting service");

	host.Run();
	
	Console.WriteLine("Ended service");
}

// You can define other methods, fields, classes and namespaces here

public class MyService : BackgroundService
{
	private readonly IHostApplicationLifetime _lifetime;
	
	private readonly Random _rnd = new Random();
	
	public MyService(IHostApplicationLifetime lifetime)
	{
		_lifetime = lifetime;
	}
	
	protected override async Task ExecuteAsync(CancellationToken stoppingToken)
	{
		try
		{
			while (true)
			{
                stoppingToken.ThrowIfCancellationRequested();
                
				var nextNumber = _rnd.Next(10);
				if (nextNumber < 8)
				{
					Console.WriteLine($"We have number {nextNumber}");
				}
				else
				{
					throw new Exception("Number too high");
				}
				
				await Task.Delay(1000);
			}
		}
        // If the application is shutting down, ignore it
        catch (OperationCanceledException e) when (e.CancellationToken == stoppingToken)
        {
            Console.WriteLine("Application is shutting itself down");
        }
        // Otherwise, we have a real exception, so must ask the application
        // to shut itself down.
		catch (Exception e)
		{
			Console.WriteLine("Oh dear. We have an exception. Let's end the process.");
			// Signal to the OS that this was an error condition by 
            // setting the exit code.
			Environment.ExitCode = 1;
			_lifetime.StopApplication();
		}
	}
}

Typical output from this program will look like:

Starting service
We have number 0
info: Microsoft.Hosting.Lifetime[0]
      Application started. Press Ctrl+C to shut down.
info: Microsoft.Hosting.Lifetime[0]
      Hosting environment: Production
info: Microsoft.Hosting.Lifetime[0]
      Content root path: C:\Users\rowla\AppData\Local\Temp\LINQPad6\_spgznchd\shadow-1
We have number 2
Oh dear. We have an exception. Let's end the process.
info: Microsoft.Hosting.Lifetime[0]
      Application is shutting down...
Ended service
-- RB.
Source: StackOverflow