I installed Prometheus in my K3S cluster, the image from here, the image is included Prometheus and Grafana, the problem comes when I forward the port.

#kubectl port-forward -n prometheus prometheus-prometheus-kube-prometheus-prometheus-0 9090

this is the error

orwarding from 127.0.0.1:9090 -> 9090

Forwarding from [::1]:9090 -> 9090

Handling connection for 9090

E1220 15:17:44.164690  373403 portforward.go:400] an error occurred forwarding 9090 -> 9090: error forwarding port 9090 to pod 044703990a9e4d5562dd26ca78084993038401acf3647ede3323991c512923c1, uid : failed to execute portforward in network namespace "/var/run/netns/cni-7bc4f557-7fa0-d886-3e10-47f587c4633b": failed to dial 9090: dial tcp4: lookup localhost: Try again

Handling connection for 9090

E1220 15:17:54.162609  373403 portforward.go:400] an error occurred forwarding 9090 -> 9090: error forwarding port 9090 to pod 044703990a9e4d5562dd26ca78084993038401acf3647ede3323991c512923c1, uid : failed to execute portforward in network namespace "/var/run/netns/cni-7bc4f557-7fa0-d886-3e10-47f587c4633b": failed to dial 9090: dial tcp4: lookup localhost: Try again

Handling connection for 9090

E1220 15:24:49.689777  373403 portforward.go:340] error creating error stream for port 9090 -> 9090: Timeout occurred

Handling connection for 9090

E1220 15:24:59.710498  373403 portforward.go:362] error creating forwarding stream for port 9090 -> 9090: Timeout occurred

Handling connection for 9090

E1220 15:25:09.742766  373403 portforward.go:362] error creating forwarding stream for port 9090 -> 9090: Timeout occurred

Handling connection for 9090

I do the same thing to Grafana using port 3000, but it is totally fine.

#kubectl port-forward -n prometheus prometheus-grafana-8664fc46fd-j3hns 3000

this is pod description for prometheus-prometheus-kube-prometheus-prometheus-0

Name:         prometheus-prometheus-kube-prometheus-prometheus-0

Namespace:    prometheus

Priority:     0

Node:         k3s-agent1/192.168.20.20
Start Time:   Mon, 20 Dec 2021 11:20:15 +0800

Labels:       app=prometheus

              controller-revision-hash=prometheus-prometheus-kube-prometheus-prometheus-5647b8f994

              operator.prometheus.io/name=prometheus-kube-prometheus-prometheus

              operator.prometheus.io/shard=0

              prometheus=prometheus-kube-prometheus-prometheus

              statefulset.kubernetes.io/pod-name=prometheus-prometheus-kube-prometheus-prometheus-0

Annotations:  <none>

Status:       Running

IP:           10.42.1.84

IPs:

  IP:           10.42.1.84

Controlled By:  StatefulSet/prometheus-prometheus-kube-prometheus-prometheus

Containers:

  prometheus:

    Container ID:  containerd://3165a8c3d2cac548f91235330be1b2880db19f68d409205a8469da39ad0a50c9

    Image:         quay.io/prometheus/prometheus:v2.22.1

    Image ID:      quay.io/prometheus/prometheus@sha256:b899dbd1b9017b9a379f76ce5b40eead01a62762c4f2057eacef945c3c22d210

    Port:          9090/TCP

    Host Port:     0/TCP

    Args:

      --web.console.templates=/etc/prometheus/consoles

      --web.console.libraries=/etc/prometheus/console_libraries

      --config.file=/etc/prometheus/config_out/prometheus.env.yaml

      --storage.tsdb.path=/prometheus

      --storage.tsdb.retention.time=10d

      --web.enable-lifecycle

      --storage.tsdb.no-lockfile

      --web.external-url=http://prometheus-kube-prometheus-prometheus.prometheus:9090

      --web.route-prefix=/

    State:          Running

      Started:      Mon, 20 Dec 2021 11:20:17 +0800

    Ready:          True

    Restart Count:  0

    Readiness:      http-get http://:web/-/ready delay=0s timeout=3s period=5s #success=1 #failure=120

    Environment:    <none>

    Mounts:

      /etc/prometheus/certs from tls-assets (ro)

      /etc/prometheus/config_out from config-out (ro)

      /etc/prometheus/rules/prometheus-prometheus-kube-prometheus-prometheus-rulefiles-0 from prometheus-prometheus-kube-prometheus-prometheus-rulefiles-0 (rw)

      /prometheus from prometheus-prometheus-kube-prometheus-prometheus-db (rw)

      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-slt5r (ro)

  config-reloader:

    Container ID:  containerd://c526856553816a463bb7d9562277b61d201a6bed6562f2ae97f0dd1a7794c72b

    Image:         quay.io/prometheus-operator/prometheus-config-reloader:v0.44.0

    Image ID:      quay.io/prometheus-operator/prometheus-config-reloader@sha256:6659cb3d97b5f846b5e1651ca1ab95189509f899713c727143f1bc50cc6e57cf

    Port:          8080/TCP

    Host Port:     0/TCP

    Command:

      /bin/prometheus-config-reloader

    Args:

      --listen-address=:8080

      --reload-url=http://127.0.0.1:9090/-/reload

      --config-file=/etc/prometheus/config/prometheus.yaml.gz

      --config-envsubst-file=/etc/prometheus/config_out/prometheus.env.yaml

      --watched-dir=/etc/prometheus/rules/prometheus-prometheus-kube-prometheus-prometheus-rulefiles-0

    State:          Running

      Started:      Mon, 20 Dec 2021 11:20:20 +0800

    Ready:          True

    Restart Count:  0

    Limits:

      cpu:     100m

      memory:  25Mi

    Requests:

      cpu:     100m

      memory:  25Mi

    Environment:

      POD_NAME:  prometheus-prometheus-kube-prometheus-prometheus-0 (v1:metadata.name)

      SHARD:     0

    Mounts:

      /etc/prometheus/config from config (rw)

      /etc/prometheus/config_out from config-out (rw)

      /etc/prometheus/rules/prometheus-prometheus-kube-prometheus-prometheus-rulefiles-0 from prometheus-prometheus-kube-prometheus-prometheus-rulefiles-0 (rw)

      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-slt5r (ro)

Conditions:

  Type              Status

  Initialized       True 

  Ready             True 

  ContainersReady   True 

  PodScheduled      True 

Volumes:

  config:

    Type:        Secret (a volume populated by a Secret)

    SecretName:  prometheus-prometheus-kube-prometheus-prometheus

    Optional:    false

  tls-assets:

    Type:        Secret (a volume populated by a Secret)

    SecretName:  prometheus-prometheus-kube-prometheus-prometheus-tls-assets

    Optional:    false

  config-out:

    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)

    Medium:     

    SizeLimit:  <unset>

  prometheus-prometheus-kube-prometheus-prometheus-rulefiles-0:

    Type:      ConfigMap (a volume populated by a ConfigMap)

    Name:      prometheus-prometheus-kube-prometheus-prometheus-rulefiles-0

    Optional:  false

  prometheus-prometheus-kube-prometheus-prometheus-db:

    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)

    Medium:     

    SizeLimit:  <unset>

  kube-api-access-slt5r:

    Type:                    Projected (a volume that contains injected data from multiple sources)

    TokenExpirationSeconds:  3607

    ConfigMapName:           kube-root-ca.crt

    ConfigMapOptional:       <nil>

    DownwardAPI:             true

QoS Class:                   Burstable

Node-Selectors:              <none>

Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s

                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s

Events:                      <none>