Realizing that the apiserver is running inside a docker container, I connected to one of my controller nodes using SSH. Then I started a container using the following command to get a shell prompt using the same docker image that apiserver is using.

docker run \
  -it \
  --rm \
  --entrypoint /bin/sh \
  --volume /etc/kubernetes:/etc/kubernetes:ro \
  --volume /etc/ssl/certs:/etc/ssl/certs:ro \
  --volume /etc/pki:/etc/pki:ro \
  --volume /etc/pki/ca-trust:/etc/pki/ca-trust:ro \
  --volume /etc/pki/tls:/etc/pki/tls:ro \
  --volume /etc/ssl/etcd/ssl:/etc/ssl/etcd/ssl:ro \
  --volume /etc/kubernetes/ssl:/etc/kubernetes/ssl:ro \
  --volume /var/run/kmsplugin:/var/run/kmsplugin \
  k8s.gcr.io/kube-apiserver:v1.18.5

Once inside that container, I could run the same command that is setup in kube-apiserver.yaml. This command was:

kube-apiserver \
    --encryption-provider-config=/etc/kubernetes/aws-encryption-provider-config.yaml \
    --advertise-address=10.250.203.201 \
    ...
    --service-node-port-range=30000-32767 \
    --storage-backend=etcd3 \

I elided the bulk of the command since you'll need to get specific values from your own kube-apiserver.yaml file.

Using this technique showed me the error message:

Error: error while parsing encryption provider configuration file 
"/etc/kubernetes/aws-encryption-provider-config.yaml": error while parsing 
file: resources[0].providers[0]: Invalid value: 
config.ProviderConfiguration{AESGCM:(*config.AESConfiguration)(nil), 
AESCBC:(*config.AESConfiguration)(nil), Secretbox:(*config.SecretboxConfiguration)
(nil), Identity:(*config.IdentityConfiguration)(nil), KMS:(*config.KMSConfiguration)
(nil)}: provider does not contain any of the expected providers: KMS, AESGCM, 
AESCBC, Secretbox, Identity