How to connect to gRPC server using FQDN within kubernetes cluster?

7/9/2021

I've Docker Desktop Kubernetes cluster setup in my local machine and it is working fine. Now i'm trying to deploy .Net Core gRPC server and .Net core Console load generator to my cluster.

I'm using VisualStudio(2019)'s default template for gRPC application

Server:

proto file

syntax = "proto3";

option csharp_namespace = "KubernetesLoadSample";

package greet;

// The greeting service definition.
service Greeter {
  // Sends a greeting
  rpc SayHello (HelloRequest) returns (HelloReply);
}

// The request message containing the user's name.
message HelloRequest {
  string name = 1;
}

// The response message containing the greetings.
message HelloReply {
  string message = 1;
}

.net core gRPC application

public class GreeterService : Greeter.GreeterBase
{
    private readonly ILogger<GreeterService> _logger;
    public GreeterService(ILogger<GreeterService> logger)
    {
        _logger = logger;
    }

    public override Task<HelloReply> SayHello(HelloRequest request, ServerCallContext context)
    {
        _logger.LogInformation("Compute started");
        double result = 0;
        for (int i = 0; i < 10000; i++)
        {
            for (int j = 0; j < i; j++)
            {
                result += Math.Sqrt(i) + Math.Sqrt(j);
            }
        }
        return Task.FromResult(new HelloReply
        {
            Message = "Completed"
        }); ;
    }
}

and DockerFile for this project as follows,

FROM mcr.microsoft.com/dotnet/aspnet:3.1 AS base
WORKDIR /app
EXPOSE 80
EXPOSE 443

FROM mcr.microsoft.com/dotnet/sdk:3.1 AS build
WORKDIR /src
COPY ["KubernetesLoadSample.csproj", "KubernetesLoadSample/"]
RUN dotnet restore "KubernetesLoadSample/KubernetesLoadSample.csproj"

WORKDIR "/src/KubernetesLoadSample"
COPY . .
RUN dotnet build "KubernetesLoadSample.csproj" -c Release -o /app/build

FROM build AS publish
RUN dotnet publish "KubernetesLoadSample.csproj" -c Release -o /app/publish

FROM base AS final
WORKDIR /app
COPY --from=publish /app/publish .
ENTRYPOINT ["dotnet", "KubernetesLoadSample.dll"]

i was able to check this image working locally using

PS C:\Users\user> docker run -it -p 8000:80 kubernetesloadsample:latest
info: Microsoft.Hosting.Lifetime[0]
      Now listening on: http://[::]:80
info: Microsoft.Hosting.Lifetime[0]
      Application started. Press Ctrl+C to shut down.
info: Microsoft.Hosting.Lifetime[0]
      Hosting environment: Production
info: Microsoft.Hosting.Lifetime[0]
      Content root path: /app
info: KubernetesLoadSample.GreeterService[0]
      Compute started // called from BloomRPC Client

Client

Client is a .net console application, that calls server in a loop

    static async Task Main(string[] args)
    {
        var grpcServer = Environment.GetEnvironmentVariable("GRPC_SERVER");
        Channel channel = new Channel($"{grpcServer}", ChannelCredentials.Insecure);

        Console.WriteLine($"Sending load to port {grpcServer}");
        while(true)
        {
            try
            {
                var client = new Greeter.GreeterClient(channel);
                var reply = await client.SayHelloAsync(
                                  new HelloRequest { Name = "GreeterClient" });

                Console.WriteLine("result: " + reply.Message);
                await Task.Delay(1000);
            }
            catch (Exception ex)
            {
                Console.WriteLine($"{DateTime.UtcNow} : tried to connect : {grpcServer}  Crashed : {ex.Message}");
            }
        }
    }

Docker file for client:

FROM mcr.microsoft.com/dotnet/aspnet:3.1 AS base
WORKDIR /app
EXPOSE 80
EXPOSE 443

FROM mcr.microsoft.com/dotnet/sdk:3.1 AS build
WORKDIR /src
COPY ["GrpcClientConsole.csproj", "GrpcClientConsole/"]
RUN dotnet restore "GrpcClientConsole/GrpcClientConsole.csproj"

WORKDIR "/src/GrpcClientConsole"
COPY . .
RUN dotnet build "GrpcClientConsole.csproj" -c Release -o /app/build

FROM build AS publish
RUN dotnet publish "GrpcClientConsole.csproj" -c Release -o /app/publish

FROM base AS final
WORKDIR /app
COPY --from=publish /app/publish .
ENTRYPOINT ["dotnet", "GrpcClientConsole.dll"]

and deployment file as follows,

---
apiVersion: v1
kind: Namespace
metadata:
  name: core-load
---
apiVersion: apps/v1
kind: Deployment
metadata:
  creationTimestamp: null
  name: compute-server
  namespace: core-load
spec:
  replicas: 4
  selector:
    matchLabels:
      app: compute-server-svc
  strategy: {}
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: compute-server-svc
    spec:
      containers:
      - env:
        image: kubernetesloadsample:latest
        imagePullPolicy: Never
        name: compute-server-svc
        ports:
        - containerPort: 80
          name: grpc
        resources: {}
status: {}
---
apiVersion: v1
kind: Service
metadata:
  name: compute-server-svc
  namespace: core-load
spec:
  clusterIP: None
  
  ports:
  - name: grpc
    port: 5000
    targetPort: 80
    protocol: TCP
  selector:
    app: compute-server-svc
---
apiVersion: apps/v1
kind: Deployment
metadata:
  creationTimestamp: null
  name: compute-client
  namespace: core-load
spec:
  replicas: 1
  selector:
  
    matchLabels:
      app: compute-client
  strategy: {}
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: compute-client
    spec:
      containers:
      - env:
        - name: GRPC_SERVER
          value: compute-server-svc.core-load.svc.cluster.local:5000
        image: grpc-client-console:latest
        imagePullPolicy: Never
        name: compute-client
        resources: {}
status: {}
---

Problem

client is not able to connect gRPC server with this compute-server-svc.core-load.svc.cluster.local:5000 name. I tried compute-server-svc.core-load this as well, but facing below issue

PS E:\study\core\k8sgrpc\KubernetesLoadSample> k get pods -n core-load
NAME                              READY   STATUS    RESTARTS   AGE
compute-client-bff5f666-cjwf5     1/1     Running   0          15s
compute-server-545567f589-5blkv   1/1     Running   0          15s
compute-server-545567f589-bv4r2   1/1     Running   0          15s
compute-server-545567f589-mdp2x   1/1     Running   0          15s
compute-server-545567f589-wdff5   1/1     Running   0          15s
PS E:\study\core\k8sgrpc\KubernetesLoadSample> k logs compute-client-bff5f666-cjwf5 -n  core-load --tail 5
07/09/2021 17:18:35 : tried to connect : compute-server-svc.core-load.svc.cluster.local:5000 Crashed : Status(StatusCode=Unavailable, Detail="failed to connect to all addresses")
07/09/2021 17:18:35 : tried to connect : compute-server-svc.core-load.svc.cluster.local:5000 Crashed : Status(StatusCode=Unavailable, Detail="failed to connect to all addresses")
07/09/2021 17:18:35 : tried to connect : compute-server-svc.core-load.svc.cluster.local:5000 Crashed : Status(StatusCode=Unavailable, Detail="failed to connect to all addresses")
07/09/2021 17:18:35 : tried to connect : compute-server-svc.core-load.svc.cluster.local:5000 Crashed : Status(StatusCode=Unavailable, Detail="failed to connect to all addresses")
07/09/2021 17:18:35 : tried to connect : compute-server-svc.core-load.svc.cluster.local:5000 Crashed : Status(StatusCode=Unavailable, Detail="failed to connect to all addresses")

I didnt get any solution from the stackoverflow questions similar to this, so im creating this.

Can anyone please let me know what i've missed or doing wrong?

TIA

-- WPFUser
.net-core
c#
grpc
kubernetes
kubernetes-service

1 Answer

7/12/2021

You defined your service with the:

clusterIP: None

which is used to create an headless service. This may be the cause of the problem, so removing it could resolve your error.


When you create a ClusterIP type service (which is the default type) Kubernetes automatically assign the service a virtual IP (called also cluster IP, as the type suggests) which is then used to proxy communication towards the Pods selected by the service in question.

This means that there is a "new" IP address (visible only from inside the cluster), different from the various IP assigned to the Pods (or single Pod) behind the service, which then routes the traffic with a sort of load balancing to the Pods standing behind.

If you specify

clusterIP: None

you create an headless service. You are basically telling Kubernetes that you don't want a virtual IP to be assigned to the service. There is no load balancing by the proxy as there is no IP to load balance.

Instead, the DNS configuration will return A records (the IP addresses) for each of the Pods behind (selected) by the service.

This can be useful if your application needs to discover each Pod behind the service and then do whatever they want with the IP address on their own.

Maybe to load balance with an internal implementation, maybe because different Pods (behidn the same service) are used for different things.. or maybe because each one of those Pods wants to discover the other Pods (think about multi-instance primary applications such as Kafka or Zookeeper, for example)


I'm not sure on what exactly could be your problem, it may depends on how the hostname is resolved by that particular app.. but you shouldn't use an headless service, unless you have the necessity to decide which of the Pods selected by the svc you want to contact.

Using DNS round robin to load balance is also (almost always) not a good idea compared to a virtual IP.. as applications could cache the DNS resolution and, if Pods then change IP address (since Pods are ephimeral, they change IP address whenever they restart, for example), there could be network problems in reaching them.. and more.

There's a huge amount of info in the docs: https://kubernetes.io/docs/concepts/services-networking/service/

-- AndD
Source: StackOverflow