Thanks to @Brian-Silverman help I was able to triage all my issues and landed with this solution.

  go_image(
      name = "go_auth_image",
      embed = [":go-auth_lib"],
      visibility = ["//visibility:public"],
  )

  GRPC_HEALTH_PROBE_VERSION = "v0.4.5"


  container_run_and_commit_layer(
      name = "health_probe",
      image = "@grpc_health_image//image",
      commands = [
          "apk add --no-cache git",
          "wget -qO/bin/grpc_health_probe https://github.com/grpc-ecosystem/grpc-health-probe/releases/download/%s/grpc_health_probe-linux-amd64" % GRPC_HEALTH_PROBE_VERSION,
          "chmod +x /bin/grpc_health_probe",
      ],
  )

  container_image(
      name = "go_auth_api",
      base = ":go_auth_image",
      layers = [":health_probe",],
      visibility = ["//visibility:public"],
      ports = ["5001"],
  )

I've responded to your github issue, but posting the answer here as well for visibility: I would accomplish this in Bazel by making a TAR which contains the grpc_health_probe binary in the location I would like and add that in the tars section of a container_image rule. I try to avoid the RUN-like rules, because they introduce a dependency on the docker toolchain and they are not guaranteed to be hermetic/reproducible. There's an open issue about this here: https://github.com/bazelbuild/rules_docker/issues/1961.

You can declare the binary in question as a dependency in your WORKSPACE (or a macro invoked by your WORKSPACE) like so:

load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_file")
GRPC_HEALTH_PROBE_VERSION = "v0.4.5"
GRPC_HEALTH_PROBE_SHA256 = "8699c46352d752d8f533cae72728b0e65663f399fc28fb9cd854b14ad5f85f44"
http_file(
    name = "grpc_health_probe_linux_amd64",
    executable = True,  # Here is your `chmod +x` from the `container_run_and_commit`
    sha256 = GRPC_HEALTH_PROBE_SHA256,
    downloaded_file_path = "grpc_health_probe",  # This is necessary to remove `-linux-amd64` from the end of the file name.
    urls = ["https://github.com/grpc-ecosystem/grpc-health-probe/releases/download/{}/grpc_health_probe-linux-amd64".format(GRPC_HEALTH_PROBE_VERSION)],
)

Having done that, and included the rules_docker toolchain configuration, you can then add this binary to a container image with a build file that looks like the following:

load("@rules_pkg//:pkg.bzl", "pkg_tar")
load("@io_bazel_rules_docker//container:container.bzl", "container_image")

# This rule creates a TAR file containing the executable file
# /bin/grpc_health_probe.
pkg_tar(
    name = "health_probe_linux_amd64",
    srcs = ["@grpc_health_probe_linux_amd64//file"],
    package_dir = "/bin",  # specifying the path of the file
)

# Here, we add that health probe to our base image. In this case
# I've used the ubuntu image for a minimal example.
container_image(
    name = "health_probe_image",
    base = "YOUR_BASE_IMAGE_HERE",
    tars = [":health_probe_linux_amd64"],
)

Github reply: https://github.com/bazelbuild/rules_docker/issues/1943#issuecomment-998400122

container_run_and_commit is the closest equivalent to RUN. Something like this is the direct equivalent:

load("@io_bazel_rules_docker//docker/util:run.bzl", "container_run_and_commit")

GRPC_HEALTH_PROBE_VERSION = "v0.3.1"

container_run_and_commit(
    name = "install_stuff",
    image = ":go_auth_image.tar",
    commands = [
        "wget -qO/bin/grpc_health_probe https://github.com/grpc-ecosystem/grpc-health-probe/releases/download/%s/grpc_health_probe-linux-amd64" % GRPC_HEALTH_PROBE_VERSION,
        "chmod +x /bin/grpc_health_probe",
    ],
)


container_image(
    name = "go_auth_api",
    visibility = ["//visibility:public"],
    base = ":install_stuff",
    ... # everything else you're doing with container_image
)

It runs the commands a builds a new image, and then uses container_image to add things to the result.

However, doing more of the build with bazel will make better use of bazel's cache and be more reproducible. I think that's what you're doing with the grpc_health_probe-linux-amd64 source file. That approach looks something like this:

load("@io_bazel_rules_docker//docker/util:run.bzl", "container_run_and_commit")

container_image(
    name = "add_stuff",
    base = ":go_auth_image",
    ports = ["5001", "5002"],
    files = ["grpc_health_probe-linux-amd64"],
    symlinks = {
        "grpc_health_prob-linux-amd64": "/bin/grpc_health_probe",
    },
)

container_run_and_commit(
    name = "go_auth_api",
    visibility = ["//visibility:public"],
    image = ":add_stuff.tar",
    commands = [
        "apk add --no-cache git",
        "chmod +x /bin/grpc_health_probe",
    ],
)

That uses container_image to add things first, and then runs the commands afterwards.

Also, instead of running chmod +x, you can use pkg_tar to package the file+symlink (it has a symlinks attribute just like container_image), and then set mode = 0755. container_image.tars will take the tar file and add it to the image. In general, pkg_tar gives a lot of flexibility for building up files, and container_image takes a subset of its functionality directly for simple use cases.

container_image.cmd is the equivalent of CMD in a Dockerfile. It's just setting the information when the container is used, not doing anything while building it. I don't think you want to use it at all for this.