C# (dotnet core) Correlation failed. Behind ingress (GLBC) loadbalancer on GKE

8/9/2018

I have struggling with authentication now for weeks on a project, and it's killing me. Bear with me, this is my first post on stackoverflow as well.

I have been looking around and googling/searching here on stackoverflow, and have come up with some different things but none seem to help:

My Microsoft Azure authentication has the same problem, and is based off: https://github.com/microsoftgraph/aspnetcore-connect-sample

Stacktrace:

[09:58:48 INF] Error from RemoteAuthentication: Correlation failed..
[09:58:48 ERR] An unhandled exception has occurred while executing the request.
System.Exception: An error was encountered while handling the remote login. 
---> System.Exception: Correlation failed.
--- End of inner exception stack trace ---
at Microsoft.AspNetCore.Authentication.RemoteAuthenticationHandler1.HandleRequestAsync() 
at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context) 
at Microsoft.AspNetCore.StaticFiles.StaticFileMiddleware.Invoke(HttpContext context) 
at Muninn.Startup.<>c.<<Configure>b__8_0>d.MoveNext() 
in /Muninn/Startup.cs:line 180 --- End of stack trace from previous location where exception was thrown --- 
at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.Invoke(HttpContext context) 

This is the Startup.cs: public class Startup { public Startup(IConfiguration configuration) { Configuration = configuration; }

    public IConfiguration Configuration { get; }
    public const string ObjectIdentifierType = "http://schemas.microsoft.com/identity/claims/objectidentifier";
    public const string TenantIdType = "http://schemas.microsoft.com/identity/claims/tenantid";
    public readonly IDataStore dataStore = new FileDataStore(GoogleWebAuthorizationBroker.Folder);

    // This method gets called by the runtime. Use this method to add services to the container.
    public void ConfigureServices(IServiceCollection services)
    {   
        services.AddDbContext<ApplicationDbContext>(options =>
            options.UseNpgsql(Configuration.GetConnectionString("DefaultConnection")));

        services.AddIdentity<ApplicationUser, IdentityRole>()
            .AddEntityFrameworkStores<ApplicationDbContext>()
            .AddDefaultTokenProviders();

        services.AddAuthentication(sharedOptions =>
        {
            sharedOptions.DefaultAuthenticateScheme = CookieAuthenticationDefaults.AuthenticationScheme;
            sharedOptions.DefaultSignInScheme = CookieAuthenticationDefaults.AuthenticationScheme;
            sharedOptions.DefaultChallengeScheme = OpenIdConnectDefaults.AuthenticationScheme;
        })
        .AddAzureAd(options => Configuration.Bind("AzureAd", options))
        .AddGoogle(googleOptions =>
        {
            googleOptions.Scope.Add(CalendarService.Scope.Calendar);
            googleOptions.ClientId = Configuration["Google:ClientId"];
            googleOptions.ClientSecret = Configuration["Google:ClientSecret"];
            googleOptions.AccessType = "offline";
            googleOptions.SaveTokens = true;
            googleOptions.CallbackPath = "/signin-google";
            googleOptions.Events = new OAuthEvents()
            {
                OnCreatingTicket = async (context) =>
                {
                    var userEmail = context.Identity.FindFirst(ClaimTypes.Email).Value;
                    Log.Information("New user logged in with Google: " + userEmail);

                    if(string.IsNullOrEmpty(context.AccessToken))
                        Log.Error("Access token was null");
                    if (string.IsNullOrEmpty(context.RefreshToken))
                        Log.Error("Refresh token was null");

                    var tokenResponse = new TokenResponse()
                    {      
                        AccessToken = context.AccessToken,
                        RefreshToken = context.RefreshToken,
                        ExpiresInSeconds = (long)context.ExpiresIn.Value.TotalSeconds,
                        IssuedUtc = DateTime.UtcNow
                    };
                    tokenResponse.Scope = CalendarService.Scope.Calendar;
                    await dataStore.StoreAsync(userEmail, tokenResponse);
                    Log.Information("User has been saved to the system: " + userEmail);
                }
            };
        })
        .AddCookie(options =>
        {
            options.LoginPath = "/Account/SignIn";
            options.LogoutPath = "/Account/SignOff";
        });
        services.AddMvc();


        var redis = ConnectionMultiplexer.Connect(Configuration["Redis:DefaultConnection"]);
        services.AddDataProtection()
            .PersistKeysToRedis(redis, "DataProtection-Keys")
            .SetApplicationName("myapp");

        services.AddDistributedRedisCache(options =>
            {
                options.Configuration = Configuration["Redis:DefaultConnection"];
                options.InstanceName = "master";
            });


        services.Configure<MvcOptions>(options =>
        {
            options.Filters.Add(new RequireHttpsAttribute());
        });
        services.Configure<CookiePolicyOptions>(options =>
        {
            options.CheckConsentNeeded = context => false;
            options.MinimumSameSitePolicy = SameSiteMode.None;
        });

        services.AddSession(options =>
        { 
            options.Cookie.Domain = ".myapp.com";
            options.Cookie.SecurePolicy = Microsoft.AspNetCore.Http.CookieSecurePolicy.Always;
            options.Cookie.Name = ".myapp.Session";
            options.IdleTimeout = TimeSpan.FromSeconds(5);
        });

        services.AddSingleton<IGraphAuthProvider, GraphAuthProvider>();
        services.AddTransient<IGraphSdkHelper, GraphSdkHelper>();
        services.AddTransient<IEmailSender, EmailSender>();
        services.AddTransient<ICalendarActions, CalendarActions>();
        services.AddTransient<IOutlookCommunication, OutlookCommunication>();
        services.Configure<GoogleAuthOptions>(Configuration.GetSection("Google"));
        services.AddTransient<IGoogleCommunication, GoogleCommunication>();
        services.Configure<FormOptions>(x =>
        {
            x.ValueLengthLimit = int.MaxValue;
            x.MultipartBodyLengthLimit = int.MaxValue; // In case of multipart
        });
        services.AddAntiforgery();

    }

    // This method gets called by the runtime. Use this method to configure the HTTP request pipeline.
    public void Configure(IApplicationBuilder app, IHostingEnvironment env)
    {
        if (env.IsDevelopment())
        {
            app.UseDeveloperExceptionPage();
            app.UseBrowserLink();
            app.UseDatabaseErrorPage();                
        }
        else
        {
            app.UseExceptionHandler("/Home/Error");
        }
        app.UseForwardedHeaders(new ForwardedHeadersOptions
        {
            ForwardedHeaders = ForwardedHeaders.XForwardedFor | ForwardedHeaders.XForwardedProto
        });
        app.Use(async (context, next) =>
        {
            context.Request.Scheme = "https";
            await next.Invoke();
        });
        app.UseHttpsRedirection();

        var forceSSL = new RewriteOptions()
            .AddRedirectToHttps();
        app.UseRewriter(forceSSL);

        app.UseStaticFiles();
        app.UseAuthentication();
        app.UseSession();
        app.UseMvc(routes =>
        {
            routes.MapRoute(
                name: "default",
                template: "{controller=Home}/{action=Index}/{id?}");
        });
    }
}

This all runs in GKE as a deployment where the dockerfile is :

FROM microsoft/dotnet:2.1-sdk as builder
COPY . .
RUN dotnet restore && dotnet build
WORKDIR myapp
RUN dotnet publish --output /app --configuration release -r ubuntu.16.04-x64

FROM microsoft/dotnet:2.1-runtime
EXPOSE 5000/tcp
ENV ASPNETCORE_URLS http://*:5000
COPY --from=builder /app .
ENTRYPOINT dotnet myapp.dll  

I suspect the problem is that the google/azure endpoint's callback is not decrypted properly because my application does not realize it is being hosted in a "web farm" in Kubernetes despite my redis cache having "DataProtection-Keys" when I exec into the container.

The application should also understand that its context is https://myapp.com since I set the ForwardHeaders bit.

Help please..?

tl;dr .NET core application behind GLBC gets "System.Exception: Correlation failed." despite using AddDataProtection() and Forwarded headers. I'm livid, please help.

-- David Johannes Christensen
.net-core
authentication
c#
google-kubernetes-engine
openid-connect

1 Answer

9/8/2018

Since this was not answered I made an issue in Aspnet/Security repository on github (https://github.com/aspnet/Security/issues/1844), where some really nice maintainers took their time to help.

The solution to the problem above was as follows:

Application is fully configured to be https (context.Request.Scheme = "https") but the loadbalancing level was letting some http requests through.

When an auth cookie was created through http, it was basically lying to the application that the traffic was https and it caused the correlation error.

-- David Johannes Christensen
Source: StackOverflow