Following is my kubernetes version details:

Client Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.1", GitCommit:"82450d03cb057bab0950214ef122b67c83fb11df", GitTreeState:"clean", BuildDate:"2016-12-14T00:57:05Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.1", GitCommit:"82450d03cb057bab0950214ef122b67c83fb11df", GitTreeState:"clean", BuildDate:"2016-12-14T00:52:01Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"}

kubectl cluster-info gives the following output

Kubernetes master is running at http://localhost:8080

kubectl --server=localhost:8080 get pods works fine:

NAME                     READY     STATUS    RESTARTS   AGE
nginx-3449338310-sc54z   1/1       Running   0          3d
nginx-3449338310-sfql2   1/1       Running   0          3d
nginx-controller-0hq0n   1/1       Running   0          2h
nginx-controller-sc0nc   1/1       Running   0          2h

But this doesn't work when accessing via IP address: kubectl --server=10.10.65.151:8080

The connection to the server 10.10.65.151:8080 was refused - did you specify the right host or port?

This is my kube-config:

apiVersion: v1
kind: Config
clusters:
- name: kubernetes
  cluster:
    certificate-authority: /etc/kubernetes/pki/ca.crt
    server: http://10.10.65.151:8080
users:
- name: kubeuser
  user:
    client-certificate: /etc/kubernetes/pki/kube-master.crt
    client-key: /etc/kubernetes/pki/kube-master.key
contexts:
- context:
    cluster: kubernetes
    user: kubeuser
  name: kubeuser-context
current-context: kubeuser-context

I get the same error while trying to access the api-server from a remote machine. May I know what's missing?