I am using Kafka Helm charts from here. I was trying Horizontal Pod Autoscaler for the same.

I added a hpa.yaml file as given below inside the templates folder.

apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
  name: kafka-hpa
spec:
  scaleTargetRef:
    apiVersion: extensions/v1beta1
    kind: Deployment
    name: {{ include "kafka.fullname" . }}
minReplicas: {{ .Values.replicas }}
maxReplicas: 5
metrics:
- type: Resource
  resource:
    name: cpu
    targetAverageUtilization: 50
- type: Resource
  resource:
    name: memory
    targetAverageValue: 8000Mi

I have also tried the above YAML with kind: StatefulSet but the same issue persists.

My intention is to have 3 Kafka pods initially and scale it up to 5 based on CPU and memory targetValues as mentioned above.

However, the hpa gets deployed but it is unable to read the metrics as per my understanding as the current usage shows unknown as mentioned below.

NAME        REFERENCE                          TARGETS                          MINPODS   MAXPODS   REPLICAS   AGE
kafka-hpa   Deployment/whopping-walrus-kafka   <unknown>/8000Mi, <unknown>/50%   3         5         0          1h . 

I am new to helm and Kubernetes, so I am assuming there might be some issue with my understanding.

I have also deployed metrics-server.

$ kubectl get deployments
NAME                             DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
metrics-server                   1         1         1            1           1d
whopping-walrus-kafka-exporter   1         1         1            1           1h

Pods output

$ kubectl get pods
NAME                                              READY     STATUS    RESTARTS   AGE
metrics-server-55cbf87bbb-vm2v5                   1/1       Running   0          15m
whopping-walrus-kafka-0                           1/1       Running   1          1h
whopping-walrus-kafka-1                           1/1       Running   0          1h
whopping-walrus-kafka-2                           1/1       Running   0          1h
whopping-walrus-kafka-exporter-5c66b5b4f9-mv5kv   1/1       Running   1          1h
whopping-walrus-zookeeper-0                       1/1       Running   0          1h

I want the whopping-walrus-kafka pod to scale up to 5 on load, however, there's no deployment corresponding to it.

StatefulSet Output

$ kubectl get statefulset
NAME                        DESIRED   CURRENT   AGE
original-bobcat-kafka       3         2         2m
original-bobcat-zookeeper   1         1         2m

Output of describe hpa when kind in hpa.yaml is StatefulSet.

$ kubectl describe hpa
Name:                                                  kafka-hpa
Namespace:                                             default
Labels:                                                <none>
Annotations:                                           <none>
CreationTimestamp:                                     Fri, 18 Jan 2019 12:13:59 +0530
Reference:                                             StatefulSet/original-bobcat-kafka
Metrics:                                               ( current / target )
  resource memory on pods:                             <unknown> / 8000Mi
  resource cpu on pods  (as a percentage of request):  <unknown> / 5%
Min replicas:                                          3
Max replicas:                                          5
Conditions:
  Type         Status  Reason          Message
  ----         ------  ------          -------
  AbleToScale  False   FailedGetScale  the HPA controller was unable to get the target's current scale: no matches for kind "StatefulSet" in group "extensions"
Events:
  Type     Reason          Age                From                       Message
  ----     ------          ----               ----                       -------
  Warning  FailedGetScale  15s (x17 over 8m)  horizontal-pod-autoscaler  no matches for kind "StatefulSet" in group "extensions"

Output of describe hpa when kind in hpa.yaml is Deployment.

$ kubectl describe hpa
Name:                                                  kafka-hpa
Namespace:                                             default
Labels:                                                <none>
Annotations:                                           <none>
CreationTimestamp:                                     Fri, 18 Jan 2019 12:30:07 +0530
Reference:                                             Deployment/good-elephant-kafka
Metrics:                                               ( current / target )
  resource memory on pods:                             <unknown> / 8000Mi
  resource cpu on pods  (as a percentage of request):  <unknown> / 5%
Min replicas:                                          3
Max replicas:                                          5
Conditions:
  Type         Status  Reason          Message
  ----         ------  ------          -------
  AbleToScale  False   FailedGetScale  the HPA controller was unable to get the target's current scale: could not fetch the scale for deployments.extensions good-elephant-kafka: deployments/scale.extensions "good-elephant-kafka" not found
Events:
  Type     Reason          Age   From                       Message
  ----     ------          ----  ----                       -------
  Warning  FailedGetScale  9s    horizontal-pod-autoscaler  could not fetch the scale for deployments.extensions good-elephant-kafka: deployments/scale.extensions "good-elephant-kafka" not found

Output from metrics server pod

$ kubectl describe pods metrics-server-55cbf87bbb-vm2v5
Name:           metrics-server-55cbf87bbb-vm2v5
Namespace:      default
Node:           docker-for-desktop/192.168.65.3
Start Time:     Fri, 18 Jan 2019 11:26:33 +0530
Labels:         app=metrics-server
            pod-template-hash=1176943666
            release=metrics-server
Annotations:    <none>
Status:         Running
IP:             10.1.0.119
Controlled By:  ReplicaSet/metrics-server-55cbf87bbb
Containers:
  metrics-server:
    Container ID:  docker://ee4b3d9ed1b15c2c8783345b0ffbbc565ad25f1493dec0148f245c9581443631
    Image:         gcr.io/google_containers/metrics-server-amd64:v0.3.1
    Image ID:      docker-pullable://gcr.io/google_containers/metrics-server-amd64@sha256:78938f933822856f443e6827fe5b37d6cc2f74ae888ac8b33d06fdbe5f8c658b
    Port:          <none>
    Host Port:     <none>
    Command:
      /metrics-server
      --kubelet-insecure-tls
      --kubelet-preferred-address-types=InternalIP
      --logtostderr
    State:          Running
  Started:      Fri, 18 Jan 2019 11:26:35 +0530
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
  /var/run/secrets/kubernetes.io/serviceaccount from metrics-server-token-d2g7b (ro)
Conditions:
  Type           Status
  Initialized    True 
  Ready          True 
  PodScheduled   True 
Volumes:
  metrics-server-token-d2g7b:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  metrics-server-token-d2g7b
    Optional:    false
    QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
             node.kubernetes.io/unreachable:NoExecute for 300s
Events:          <none>

Guys please feel free to clear my understanding as well if I am going wrong somewhere.

Help on this would be highly appreciated.