K
Q

HPA with windows worker nodes - EKS 1.11

7/31/2019

HPA doesn't work and keeps showing /10% for targets.

Metrics-server is installed and registers fine

PS C:\k> kubectl get apiservice v1beta1.metrics.k8s.io -o yaml

status:
  conditions:
  - lastTransitionTime: 2019-07-31T08:20:04Z
    message: all checks passed
    reason: Passed
    status: "True"
    type: Available

checking the logs for the metrics-server pod

$ kubectl logs metrics-server-686978657d-8rvzs -n kube-system

kubectl
E0731 20:30:09.062734       1 manager.go:102] unable to fully collect metrics: [unable to fully scrape metrics from source kubelet_summary:ip-xxx-xxx-xxx-xxx.us-west.computer.internal: [unable t
o get CPU for node "ip-xxx-xxx-xxx-xxx.us-west.computer.internal": missing cpu usage metric, unable to get CPU for container "windows-server-iis" in pod default/windows-server-iis-846f465947-n9t
tg on node "ip-xxx-xxx-xxx-xxx.us-west.computer.internal": missing cpu usage metric], unable to fully scrape metrics from source kubelet_summary:ip-xxx-xxx-xxx-xxx.us-west.computer.internal: [un
able to get CPU for node "ip-xxx-xxx-xxx-xxx.us-west.computer.internal": missing cpu usage metric, unable to get CPU for container "mymicroservice-eks" in pod default/mymicroservice-eks-5f47bc8
9bb-4nmkc on node "ip-xxx-xxx-xxx-xxx.us-west.computer.internal": missing cpu usage metric], unable to fully scrape metrics from source kubelet_summary:ip-xxx-xxx-xxx-xxx.us-west.computer.inte
rnal: [unable to get CPU for node "ip-xxx-xxx-xxx-xxx.us-west.computer.internal": missing cpu usage metric, unable to get CPU for container "mymicroservice-eks" in pod default/mymicroservice-e
ks-5f47bc89bb-dv9gx on node "ip-xxx-xxx-xxx-xxx.us-west.computer.internal": missing cpu usage metric, unable to get CPU for container "windows-iis" in pod default/windows-iis-64ddbbd57-929hv o
n node "ip-xxx-xxx-xxx-xxx.us-west.computer.internal": missing cpu usage metric]]
E0731 20:30:17.560396       1 reststorage.go:98] unable to fetch pod metrics for pod default/windows-iis-64ddbbd57-929hv: no metrics known for pod "default/windows-iis-64ddbbd57-929hv"
E0731 20:30:47.565251       1 reststorage.go:98] unable to fetch pod metrics for pod default/windows-iis-64ddbbd57-929hv: no metrics known for pod "default/windows-iis-64ddbbd57-929hv"

Deployment YAML

apiVersion: apps/v1
kind: Deployment
metadata:
  name: windows-server-iis
spec:
  selector:
    matchLabels:
      app: windows-server-iis
      tier: backend
      track: stable
  replicas: 1
  template:
    metadata:
      labels:
        app: windows-server-iis
        tier: backend
        track: stable
    spec:
      containers:
      - name: windows-server-iis
        image: mcr.microsoft.com/windows/servercore:1809 
        ports:
        - name: http
          containerPort: 80
        imagePullPolicy: IfNotPresent
        command:
        - powershell.exe
        - -command
        - "Add-WindowsFeature Web-Server; Invoke-WebRequest -UseBasicParsing -Uri 'https://dotnetbinaries.blob.core.windows.net/servicemonitor/2.0.1.6/ServiceMonitor.exe ' -OutFile 'C:\\ServiceMonitor.exe'; echo '<html><body><br/><br/><marquee><H1>Hello EKS!!!<H1><marquee></body><html>' > C:\\inetpub\\wwwroot\\default.html; C:\\ServiceMonitor.exe 'w3svc'; "
        resources:
          requests:
            cpu: 500m
      nodeSelector:
        beta.kubernetes.io/os:  windows

Details of the hpa that I have configured are provided below

PS C:\k> kubectl get hpa
NAME          REFERENCE                TARGETS         MINPODS   MAXPODS   REPLICAS   AGE
windows-iis   Deployment/windows-server-iis   <unknown>/10%   1         10        1          16m

What am I missing, is HPA not supported for windows containers ?

-- Aashish Jolly
containers
kubernetes
windows

Similar Questions

Is it possible to dynamically add a new pod to an existing deployment?
Istio throwing 404 for URL mapped in Virtual Service
How do I set the minio domain for pre-signed URLs?
How to use the kubernetes go-client to get the same Pod status info that kubectl gives

1 Answer

8/5/2019

Error you have provided: no metrics known for pod "default/windows-iis-64ddbbd57-929hv means that metrics-server pod did not received any updated. Due to lack of any CPU usage or wrong spec.scaleTargetRef: declaration. Here you can find Kubernetes example. Without `kubectl describe hpa is hard to say.

During searching solution for this I found that many people on EKS have similar problem. For example on Github or Stackoverflow. Please try to add this to your deployment.

command:
    - /metrics-server 
    - --kubelet-preferred-address-types=InternalIP
    - --kubelet-insecure-tls

However you mentioned that you are using Windows Containers. According to documentation Kubernetes is supporting HPA on Windows Containers since version 1.15. If you are using pervious version HPA will not work.

Hope it helps.

-- PjoterS
Source: StackOverflow