I have a Kubernetes cluster and I am trying to deploy and configure heartbeat to monitor services.

When heartbeat starts, I see this in the logs:

2019-08-28T15:31:19.116Z        ERROR   kubernetes/util.go:90   kubernetes: Querying for pod failed with error: kubernetes api: Failure 404 pods "ip-172-20-64-197" not found

This is my autodiscover config:

heartbeat.autodiscover:
  providers:
    - type: kubernetes
      templates:
        - condition:
            equals:
              kubernetes.namespace: myappnamespace
          config:
            - type: http
              enabled: true
              urls: ["${data.host}:${data.port}"]
              schedule: "@every 1s"

I also see this in the logs which also suggests that the autodiscover isn't working properly.

2019-08-28T15:31:19.117Z        DEBUG   [kubernetes]    kubernetes/watcher.go:189       Got 0 items from the resource sync
2019-08-28T15:31:19.117Z        DEBUG   [kubernetes]    kubernetes/watcher.go:194       Done syncing 0 items from the resource sync

I've tried all sorts of combinations of the above configuration but to no avail. Can anyone point me in the right direction? Thanks.