Google finally added custom routes export to VPC peering with master subnet. So the problem is now gone, you can access private master from different VPC or through VPN.

I have multiple gcp projects, kube clusters are in separate project.

This dramatically changes the context of your question as VPC from other projects aren't routeable by simply adding project-level network rules.

For cross-project VPC peering, you need to set up a VPC Network Peering.

I want my CI (which is in different project) to be able to access private kube master.

For this, each GKE private cluster has Master Authorized Networks, which are basically IP addresses/CIDRs that are allowed to authenticate with the master endpoint for administration.

If your CI has a unified address or if the administrators have fixed IPs, you can add them to these networks so that they can authenticate to the master.

If there are not unified addresses for these clients, then depending on your specific scenario, you might need some sort of SNATing to "unify" the source of your requests to match the authorized addresses.

Additionally, you can make a private cluster without a public address. This will allow access to the master endpoint to the nodes allocated in the cluster VPC. However:

There is still an external IP address used by Google for cluster management purposes, but the IP address is not accessible to anyone.