If you're trying to pass something really complicated like that, it might be easier to package it up in a YAML file and pass it with the helm upgrade -f option.

mongodb-replicaset.podAnnotations.ad.datadoghq.com/mongodb-replicaset.instances: >-
  [
    {
      "server": "mongodb://datadog:passw0rd@%%host%%:%%port%%",
      "replica_check": true,
      "additional_metrics": ["metrics.commands","tcmalloc","top","collection"]
    }
  ]

(The >- marker is YAML syntax causing the following indented block to be processed as a string, folding newlines into spaces, and trimming leading and trailing newlines.)

Since valid JSON is valid YAML, if you're trying to launch this from a script, you could use your language-native JSON serializer.

import json, os
db_password = os.environ.get('DB_PASSWORD')
instances = [{
  "server": "mongodb://datadog:" + db_password + "@%%host:%%port",
  "replica_check": True,
  "additional_metrics": ["metrics.commands", "tcmalloc", "top", "collection"]
}]
annotations = {
  "mongodb-replicaset.podAnnotations.ad.datadoghq.com/mongodb-replicaset.instances": json.dumps(instances)
}
print(json.dumps(annotations))

Once you've done this you can pass the generated file to the helm upgrade -f option. (You don't need to include the chart's own values.yaml file, Helm includes this by default and your values override it.)

helm upgrade myservice helm/charts/myservice \
  -f mongo-dd.yaml \
  --wait --install

Yet another option is to generate this inside the chart itself. Helm includes an undocumented toJson template function; you can construct objects using the Sprig dict function.

{{- $server := printf "mongodb://datadog:%s@%%%%host%%%%:%%%%port%%%%" .Values.mongodbPassword -}}
{{- $metrics := list "metrics.commands" "tcmalloc" "top" "collection" -}}
{{- $instance := dict "server" $server "replica_check" true "additional_metrics" $metrics -}}
{{- $instances := list $instance -}}
mongodb-replicaset...instances: {{ toJson $instances }}

Your error message suggests your problem isn't the spaces per se, but that helm --set treats commas as separators between multiple key/value pairs. You can backslash-escape the commas

... --set 'mongodb-replicaset..."replica_check": true\, "additional_metrics": ["metrics.commands\,"tcmalloc"\,...]}]'