The easiest way to skip the gcloud CLI is to probably use the --token option. You can get a token with RBAC by creating a service account and tying it to a ClusterRole or Role with either a ClusterRoleBinding or RoleBinding.

Then from the command line:

$ kubectl --token <token-from-your-service-account> get pods

You still will need a context in your ~/.kube/config:

- context:
    cluster: kubernetes
  name: kubernetes-token

Otherwise, you will have to use:

$ kubectl --insecure-skip-tls-verify --token <token-from-your-service-account> -s https://<address-of-your-kube-api-server>:6443 get pods

Note that if you don't want the token to show up on the logs you can do something like this:

$ kubectl --token $(cat /path/to/a/file/where/the/token/is/stored) get pods

Also, note that this doesn't prevent you from someone running ps -Af on your box and grabbing the token from there, for the lifetime of the kubectl process (It's a good idea to rotate the tokens)

Edit:

You can use the --token-auth-file=/path/to/a/file/where/the/token/is/stored with kubectl to avoid passing it through $(cat /path/to/a/file/where/the/token/is/stored)