Where default kubernetes service gets a list of endpoints?

4/12/2019

I have a cluster with 3 control-planes. As any cluster my cluster also has a default kubernetes service. As any service it has a list of endpoints:

apiVersion: v1
items:
- apiVersion: v1
  kind: Endpoints
  metadata:
    creationTimestamp: 2017-12-12T17:08:34Z
    name: kubernetes
    namespace: default
    resourceVersion: "6242123"
    selfLink: /api/v1/namespaces/default/endpoints/kubernetes
    uid: 161edaa7-df5f-11e7-a311-d09466092927
  subsets:
  - addresses:
    - ip: 10.9.22.25
    - ip: 10.9.22.26
    - ip: 10.9.22.27
    ports:
    - name: https
      port: 8443
      protocol: TCP
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

Everything is ok, but I completely can't understand where do these endpoints come from? It is logical to assume from the Service label selector, but there's no any label selectors:

apiVersion: v1
kind: Service
metadata:
  creationTimestamp: 2017-12-12T17:08:34Z
  labels:
    component: apiserver
    provider: kubernetes
  name: kubernetes
  namespace: default
  resourceVersion: "6"
  selfLink: /api/v1/namespaces/default/services/kubernetes
  uid: 161e4f00-df5f-11e7-a311-d09466092927
spec:
  clusterIP: 10.100.0.1
  ports:
  - name: https
    port: 443
    protocol: TCP
    targetPort: 8443
  sessionAffinity: ClientIP
  sessionAffinityConfig:
    clientIP:
      timeoutSeconds: 10800
  type: ClusterIP
status:
  loadBalancer: {}

So, could anybody explain how k8s services and endpoints work in case of built-in default kubernetes service?

-- Konstantin Vustin
kubernetes

1 Answer

8/22/2019

Its not clear how you created multi-node cluster, but here are some research for you:

Set up High-Availability Kubernetes Masters describe HA k8s creation. They have notes about default kubernetes service.

Instead of trying to keep an up-to-date list of Kubernetes apiserver in the Kubernetes service, the system directs all traffic to the external IP:

in one master cluster the IP points to the single master,

in multi-master cluster the IP points to the load balancer in-front of the masters.

Similarly, the external IP will be used by kubelets to communicate with master

So I would rather expect a LB ip instead of 3 masters.

Service creation: https://github.com/kubernetes/kubernetes/blob/master/pkg/master/controller.go#L46-L83

const kubernetesServiceName = "kubernetes"

// Controller is the controller manager for the core bootstrap Kubernetes
// controller loops, which manage creating the "kubernetes" service, the
// "default", "kube-system" and "kube-public" namespaces, and provide the IP
// repair check on service IPs
type Controller struct {
    ServiceClient   corev1client.ServicesGetter
    NamespaceClient corev1client.NamespacesGetter
    EventClient     corev1client.EventsGetter
    healthClient    rest.Interface

    ServiceClusterIPRegistry rangeallocation.RangeRegistry
    ServiceClusterIPInterval time.Duration
    ServiceClusterIPRange    net.IPNet

    ServiceNodePortRegistry rangeallocation.RangeRegistry
    ServiceNodePortInterval time.Duration
    ServiceNodePortRange    utilnet.PortRange

    EndpointReconciler reconcilers.EndpointReconciler
    EndpointInterval   time.Duration

    SystemNamespaces         []string
    SystemNamespacesInterval time.Duration

    PublicIP net.IP

    // ServiceIP indicates where the kubernetes service will live.  It may not be nil.
    ServiceIP                 net.IP
    ServicePort               int
    ExtraServicePorts         []corev1.ServicePort
    ExtraEndpointPorts        []corev1.EndpointPort
    PublicServicePort         int
    KubernetesServiceNodePort int

    runner *async.Runner
}

Service periodically updates: https://github.com/kubernetes/kubernetes/blob/master/pkg/master/controller.go#L204-L242

// RunKubernetesService periodically updates the kubernetes service
func (c *Controller) RunKubernetesService(ch chan struct{}) {
    // wait until process is ready
    wait.PollImmediateUntil(100*time.Millisecond, func() (bool, error) {
        var code int
        c.healthClient.Get().AbsPath("/healthz").Do().StatusCode(&code)
        return code == http.StatusOK, nil
    }, ch)

    wait.NonSlidingUntil(func() {
        // Service definition is not reconciled after first
        // run, ports and type will be corrected only during
        // start.
        if err := c.UpdateKubernetesService(false); err != nil {
            runtime.HandleError(fmt.Errorf("unable to sync kubernetes service: %v", err))
        }
    }, c.EndpointInterval, ch)
}

// UpdateKubernetesService attempts to update the default Kube service.
func (c *Controller) UpdateKubernetesService(reconcile bool) error {
    // Update service & endpoint records.
    // TODO: when it becomes possible to change this stuff,
    // stop polling and start watching.
    // TODO: add endpoints of all replicas, not just the elected master.
    if err := createNamespaceIfNeeded(c.NamespaceClient, metav1.NamespaceDefault); err != nil {
        return err
    }

    servicePorts, serviceType := createPortAndServiceSpec(c.ServicePort, c.PublicServicePort, c.KubernetesServiceNodePort, "https", c.ExtraServicePorts)
    if err := c.CreateOrUpdateMasterServiceIfNeeded(kubernetesServiceName, c.ServiceIP, servicePorts, serviceType, reconcile); err != nil {
        return err
    }
    endpointPorts := createEndpointPortSpec(c.PublicServicePort, "https", c.ExtraEndpointPorts)
    if err := c.EndpointReconciler.ReconcileEndpoints(kubernetesServiceName, c.PublicIP, endpointPorts, reconcile); err != nil {
        return err
    }
    return nil
}

Endpoint update place: https://github.com/kubernetes/kubernetes/blob/72f69546142a84590550e37d70260639f8fa3e88/pkg/master/reconcilers/lease.go#L163

Also endpoint could be created manually. Visit Services without selectors for more info.

-- VKR
Source: StackOverflow