RemoteCertificateNameMismatch error on .NET Core API in Kubernetes with AzureAD Auth

9/3/2021

I am trying to create a web API (ASP.NET Core using Azure AD OAuth for authorization) which runs in Kubernetes (Bare-metal, using NGINX-Ingress). Running the API in IIS Express works without error, but after turning it into a Docker image and deploying it in the cluster, the app randomly throws the following exception on any request:

fail: Microsoft.AspNetCore.Server.Kestrel[13]
      Connection id "0HMBENKCJR3ER", Request id "0HMBENKCJR3ER:00000003": An unhandled exception was thrown by the application.
      System.InvalidOperationException: IDX20803: Unable to obtain configuration from: 'System.String'.
       ---> System.IO.IOException: IDX20804: Unable to retrieve document from: 'System.String'.
       ---> System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception.
       ---> System.Security.Authentication.AuthenticationException: The remote certificate is invalid according to the validation procedure: RemoteCertificateNameMismatch
         at System.Net.Security.SslStream.SendAuthResetSignal(ProtocolToken message, ExceptionDispatchInfo exception)
         at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](TIOAdapter adapter, Boolean receiveFirst, Byte[] reAuthenticationData, Boolean isApm)
         at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Boolean async, Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)
         --- End of inner exception stack trace ---
         at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Boolean async, Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)
         at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
         at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
         at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
         at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
         at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
         at System.Net.Http.DiagnosticsHandler.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
         at System.Net.Http.HttpClient.SendAsyncCore(HttpRequestMessage request, HttpCompletionOption completionOption, Boolean async, Boolean emitTelemetryStartStop, CancellationToken cancellationToken)
         at Microsoft.IdentityModel.Protocols.HttpDocumentRetriever.GetDocumentAsync(String address, CancellationToken cancel)
         --- End of inner exception stack trace ---
         at Microsoft.IdentityModel.Protocols.HttpDocumentRetriever.GetDocumentAsync(String address, CancellationToken cancel)
         at Microsoft.Identity.Web.InstanceDiscovery.IssuerConfigurationRetriever.GetConfigurationAsync(String address, IDocumentRetriever retriever, CancellationToken cancel)
         at Microsoft.IdentityModel.Protocols.ConfigurationManager`1.GetConfigurationAsync(CancellationToken cancel)
         --- End of inner exception stack trace ---
         at Microsoft.IdentityModel.Protocols.ConfigurationManager`1.GetConfigurationAsync(CancellationToken cancel)
         at Microsoft.IdentityModel.Protocols.ConfigurationManager`1.GetConfigurationAsync()
         at Microsoft.Identity.Web.Resource.AadIssuerValidator.GetIssuerValidator(String aadAuthority)
         at Microsoft.Identity.Web.MicrosoftIdentityWebApiAuthenticationBuilderExtensions.<>c__DisplayClass3_0.<AddMicrosoftIdentityWebApiImplementation>b__0(JwtBearerOptions options, IServiceProvider serviceProvider, IOptionsMonitor`1 microsoftIdentityOptionsMonitor)
         at Microsoft.Extensions.Options.ConfigureNamedOptions`3.Configure(String name, TOptions options)
         at Microsoft.Extensions.Options.OptionsFactory`1.Create(String name)
         at Microsoft.Extensions.Options.OptionsMonitor`1.<>c__DisplayClass11_0.<Get>b__0()
         at System.Lazy`1.ViaFactory(LazyThreadSafetyMode mode)
      --- End of stack trace from previous location ---
         at System.Lazy`1.CreateValue()
         at System.Lazy`1.get_Value()
         at Microsoft.Extensions.Options.OptionsCache`1.GetOrAdd(String name, Func`1 createOptions)
         at Microsoft.Extensions.Options.OptionsMonitor`1.Get(String name)
         at Microsoft.AspNetCore.Authentication.AuthenticationHandler`1.InitializeAsync(AuthenticationScheme scheme, HttpContext context)
         at Microsoft.AspNetCore.Authentication.AuthenticationHandlerProvider.GetHandlerAsync(HttpContext context, String authenticationScheme)
         at Microsoft.AspNetCore.Authentication.AuthenticationService.AuthenticateAsync(HttpContext context, String scheme)
         at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
         at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)

Sometimes a pod works without issue and sometimes it will persistently fail with this error on each request, but this changes seemingly randomly each time it's deployed. NGINX-Ingress on the cluster is fully configured with both own certificate and intermediate certificate and can serve a similar API without authorization over HTTPS without error.

Here's the Dockerfile for the image:

FROM mcr.microsoft.com/dotnet/aspnet:5.0-buster-slim AS base
RUN apt-get update \
    && apt-get install -y --no-install-recommends libgdiplus libc6-dev \
    && apt-get clean \
    && rm -rf /var/lib/apt/lists/*
WORKDIR /app
EXPOSE 80
EXPOSE 443

FROM mcr.microsoft.com/dotnet/sdk:5.0-buster-slim AS build
WORKDIR /src
COPY ["AuthTest/AuthTest.csproj", "AuthTest/"]
RUN dotnet restore "AuthTest/AuthTest.csproj"
COPY . .
WORKDIR "/src/AuthTest"
RUN dotnet build "AuthTest.csproj" -c Release -o /app/build

FROM build AS publish
RUN dotnet publish "AuthTest.csproj" -c Release -o /app/publish

FROM base AS final
WORKDIR /app
COPY --from=publish /app/publish .

ENTRYPOINT ["dotnet", "AuthTest.dll"]

And this is the .yaml file for the deployment and ingress:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: authtest-dep
  labels:
    app: authtest
spec:
  selector:
    matchLabels:
      app: authtest-app
  replicas: 4
  template:
    metadata:
      labels:
        app: authtest-app
    spec:
      containers:
        - name: authtest-app
          image: authtest:latest
          imagePullPolicy: Never

---

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: authtest-ingress
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /$1
    nginx.ingress.kubernetes.io/x-forwarded-prefix: /api/auth
spec:
  tls:
  - hosts:
      - valid.hostname.com
    secretName: secret-tls
  rules:
  - host: valid.hostname.com
    http:
      paths:
        - path: /api/auth/(.*)
          pathType: Prefix
          backend:
            service:
              name: authtest-service
              port:
                number: 80

I have tried including our own certificates (which are not self-signed) in the Docker image and also attempted to override the certificate validation to see what certificate fails, with no success. I couldn't find any answers on StackOverflow, because most of them seemed to revolve around the use of self-signed certs or had solutions involving disabling certificate authentication, which seems to be counterproductive. My question is what certificate is the error being thrown for and what can be done to fix it?

-- Breenono
asp.net-core
c#
docker
kubernetes
ssl

1 Answer

10/22/2021

After a lot of searching and diagnosing I found a solution.

In my case the DNS was misbehaving. When the API pod tries to connect to login.microsoftonline.com it first tries to resolve DNS within the cluster, resulting in the following:

[INFO] 10.244.1.60:53217 - 25833 "AAAA IN login.microsoftonline.com.default.svc.cluster.local. udp 69 false 512" NXDOMAIN qr,aa,rd 162 0.000272099s
[INFO] 10.244.1.60:53217 - 30678 "A IN login.microsoftonline.com.default.svc.cluster.local. udp 69 false 512" NXDOMAIN qr,aa,rd 162 0.000654896s
[INFO] 10.244.1.59:42740 - 22396 "AAAA IN login.microsoftonline.com.svc.cluster.local. udp 61 false 512" NXDOMAIN qr,aa,rd 154 0.000201999s
[INFO] 10.244.1.59:42740 - 25712 "A IN login.microsoftonline.com.svc.cluster.local. udp 61 false 512" NXDOMAIN qr,aa,rd 154 0.000690095s
[INFO] 10.244.1.59:44797 - 49225 "A IN login.microsoftonline.com.cluster.local. udp 57 false 512" NXDOMAIN qr,aa,rd 150 0.000318898s
[INFO] 10.244.1.59:44797 - 60243 "AAAA IN login.microsoftonline.com.cluster.local. udp 57 false 512" NXDOMAIN qr,aa,rd 150 0.000847195s
[INFO] 10.244.1.59:53903 - 63962 "AAAA IN login.microsoftonline.com.mydomain.com. udp 57 false 512" NXDOMAIN qr,aa,rd,ra 152 0.001664889s
[INFO] 10.244.1.59:53903 - 58575 "A IN login.microsoftonline.com.mydomain.com. udp 57 false 512" NOERROR qr,aa,rd,ra 112 0.001311591s

DNS was incorrectly giving me a NOERROR result for login.microsoftonline.com.mydomain.com, resulting in a connection to an address with my certificate. Using curl in the pod showed:

$ curl -v login.microsoftonline.com

* Server certificate:
*  subject: CN=*.mydomain.com
*  start date: Mar 11 00:00:00 2021 GMT
*  expire date: Apr 11 23:59:59 2022 GMT
*  subjectAltName does not match login.microsoftonline.com
* SSL: no alternative certificate subject name matches target host name 'login.microsoftonline.com'

This caused the RemoteCertificateNameMismatch error.

I found two ways to work around this:

  1. Use a fully qualified domain name by adding a dot to the end of the URL (example: google.com. instead of google.com). This bypasses the DNS resolution and makes it connect directly to the specified address. Sadly, this didn't work for login.microsoftonline.com, so I used option 2.
  2. Adjust ndots for the DNS config of the pod, by adding the following dnsConfig under spec:
spec:
  containers:
    - name: authtest
      image: authtest:latest
      imagePullPolicy: Never
  dnsConfig:
    options:
      - name: ndots
        value: "2"

By default, ndots is set to 5. This means that any URL with less then five dots is not considered an absolute domain, and DNS will try to resolve it using local search domains first, before trying it finally as an absolute address.

By specifying ndots to be two, login.microsoftonline.com will automatically become an absolute domain and the faulty internal resolution will not happen.

This could be considered a band-aid fix for the problem with the DNS resolving incorrectly, but in my case it solved the issue.

-- Breenono
Source: StackOverflow