I'm developing an application in ASP.NET Core 2.1, and running it on a Kubernetes cluster. I've implemented authentication using OpenIDConnect, using Auth0 as my provider.

This all works fine. Actions or controllers marked with the [Authorize] attribute redirect anonymous user to the identity provider, they log in, redirects back, and Bob's your uncle.

The problems start occurring when I scale my deployment to 2 or more containers. When a user visits the application, they log in, and depending on what container they get served during the callback, authentication either succeeds or fails. Even in the case of authentication succeeding, repeatedly F5-ing will eventually redirect to the identity provider when the user hits a container they aren't authorized on.

My train of thought on this would be that, using cookie authentication, the user stores a cookie in their browser, that gets passed along with each request, the application decodes it and grabs the JWT, and subsequently the claims from it, and the user is authenticated. This makes the whole thing stateless, and therefore should work regardless of the container servicing the request. As described above however, it doesn't appear to actually work that way.

My configuration in Startup.cs looks like this:

services.AddAuthentication(options =>
    {
        options.DefaultAuthenticateScheme = CookieAuthenticationDefaults.AuthenticationScheme;
        options.DefaultSignInScheme = CookieAuthenticationDefaults.AuthenticationScheme;
        options.DefaultChallengeScheme = CookieAuthenticationDefaults.AuthenticationScheme;
    })
    .AddCookie()
    .AddOpenIdConnect("Auth0", options =>
    {
        options.Authority = $"https://{Configuration["Auth0:Domain"]}";

        options.ClientId = Configuration["Auth0:ClientId"];
        options.ClientSecret = Configuration["Auth0:ClientSecret"];

        options.ResponseType = "code";

        options.Scope.Clear();
        options.Scope.Add("openid");
        options.Scope.Add("profile");
        options.Scope.Add("email");

        options.TokenValidationParameters = new TokenValidationParameters
        {
            NameClaimType = "name"
        };

        options.SaveTokens = true;

        options.CallbackPath = new PathString("/signin-auth0");

        options.ClaimsIssuer = "Auth0";

        options.Events = new OpenIdConnectEvents
        {
            OnRedirectToIdentityProviderForSignOut = context =>
            {
                var logoutUri =
                    $"https://{Configuration["Auth0:Domain"]}/v2/logout?client_id={Configuration["Auth0:ClientId"]}";

                var postLogoutUri = context.Properties.RedirectUri;
                if (!string.IsNullOrEmpty(postLogoutUri))
                {
                    if (postLogoutUri.StartsWith("/"))
                    {
                        var request = context.Request;
                        postLogoutUri = request.Scheme + "://" + request.Host + request.PathBase +
                                        postLogoutUri;
                    }

                    logoutUri += $"&returnTo={Uri.EscapeDataString(postLogoutUri)}";
                }

                context.Response.Redirect(logoutUri);
                context.HandleResponse();

                return Task.CompletedTask;
            },
            OnRedirectToIdentityProvider = context =>
            {
                context.ProtocolMessage.SetParameter("audience", "https://api.myapp.com");

                // Force the scheme to be HTTPS, otherwise we end up redirecting back to HTTP in production.
                // They should seriously make it easier to make Kestrel serve over TLS in the same way ngninx does...
                context.ProtocolMessage.RedirectUri = context.ProtocolMessage.RedirectUri.Replace("http://",
                    "https://", StringComparison.OrdinalIgnoreCase);

                Debug.WriteLine($"RedirectURI: {context.ProtocolMessage.RedirectUri}");

                return Task.FromResult(0);
            }
        };
    });

I've spent hours trying to address this issue, and came up empty. The only thing I can think of that could theoretically work now is using sticky load balancing, but that's more applying a band-aid than actually fixing the problem.

One of the main reasons to use Kubernetes is its resilience and ability to handle scaling very well. As it stands, I can only scale my backing services, and my main application would have to run as a single pod. That's far from ideal.

Perhaps there is some mechanism somewhere that creates affinity with a specific instance that I'm not aware of?

I hope someone can point me in the right direction.

Thanks!