You can use the RBAC for managing the Role-based access in K8s

For example, let's say I have a Kubernetes Secret. I want this Secret to be bound to a specific ClusterRole, so only a ServiceAccount bound to this specific ClusterRole could read it. Is there a way to set up something like that?

No, you can not use the ClusterRole for granular level access, however, you can create some Role to restrict secret.

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: default
  name: secret-read-role
rules:
- apiGroups: ["*"] 
  resources: ["secret"]
  verbs: ["get", "watch", "list"]  
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: secret-read-sa
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: secret-read-rolebinding
  namespace: default
subjects:
- kind: ServiceAccount
  name: secret-read-sa
  apiGroup: ""
roleRef:
  kind: Role
  name: secret-read-role
  apiGroup: "" 

Checkout about the resourceNames you can also give a name or pattern in name so this way it might be helpful to attach a specific secret to Role.

- apiGroups: [""]
  resources: ["namespaces"]
  verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
  resourceNames: ["userA-*"]

If you planning to Go with RBAC you can use the RBAC manager for better management : https://github.com/FairwindsOps/rbac-manager

Extra :

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
namespace: data-engineering
name: umbrella:data-engineering-app
rules:
– apiGroups: [“”]
resources: [“configmaps”]
resourceNames:  [“data-engineering-app-configmap”]    <<<<<<<<<
verbs: [“get”]
—
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
name: umbrella:data-engineering-app
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: umbrella:data-engineering-app
subjects:
– kind: ServiceAccount
name: data-engineering-app
namespace: data-engineering

You can also refer to resources by name for certain requests through the resourceNames list. When specified, requests can be restricted to individual instances of a resource. Here is an example that restricts its subject to only get or update a ConfigMap named my-configmap

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: default
  name: configmap-updater
rules:
- apiGroups: [""]
  #
  # at the HTTP level, the name of the resource for accessing ConfigMap
  # objects is "configmaps"
  resources: ["configmaps"]
  resourceNames: ["my-configmap"]
  verbs: ["update", "get"]

https://kubernetes.io/docs/reference/access-authn-authz/rbac/#referring-to-resources

Good Example : https://thenewstack.io/three-realistic-approaches-to-kubernetes-rbac/