I am using this query to calculate memory utilization of a node
(1 - node_memory_MemAvailable_bytes{cluster="$cluster"}/node_memory_MemTotal_bytes{cluster="$cluster"}) * 100I see values around 25%
but when I run kubectl top node for the same node I see a value around 16%
The difference in values between kubectl top node and Prometheus's node exporter values comes from the way these data is being collected and calculated.
When you execute kubectl top node Kubernetes reads values from root cgroup. Specifically from /sys/fs/cgroup/memory/memory.usage_in_bytes and
/sys/fs/cgroup/memory/memory.stat. The total memory usage is being calculated as: memory.usage_in_bytes - total_inactive_file.
However, Prometheus's node exporter reads values from /proc/meminfo and than calculates it according to your query which seems correct.
This difference is also being discussed in this (still open) issue.
One way of dealing with it would be to simply stick to one method of measurement and reporting.