I'm trying to launch docker-in-docker so that I can connect to it in a different container in the same Kubernetes pod. I am using the following yaml:

apiVersion: batch/v1
kind: Job
metadata:
  name: {{job_name}}
  labels:
    taskType: store_v510
    taskName: {{task_name}}
spec:
  template:
    spec:
      activeDeadlineSeconds: 86400
      volumes:
        - name: hdfs-volume
          configMap:
            name: hdfs-config
            items:
              - key: core-site.xml
                path: core-site.xml
              - key: hadoop-env.sh
                path: hadoop-env.sh
              - key: hdfs-site.xml
                path: hdfs-site.xml
              - key: log4j.properties
                path: log4j.properties
        - name: download-dir
          hostPath:
              path: /tmp/store_pipeline
        - name: docker-graph-storage 
          emptyDir: {}
      containers:
        - name: {{job_name}}
          envFrom:
          - configMapRef:
              name: job-configs
          env: 
          - name: DOCKER_HOST 
            value: tcp://localhost:2375
          volumeMounts: 
          - mountPath: /tmp/store_pipeline
            name: download-dir
          image: registry.com.cn/com/store_pipeline:0.1
          imagePullPolicy: Always
          resources:
            requests:
              cpu: 2
              memory: 32Gi
              nvidia.com/gpu: 0
            limits:
              cpu: 2
              memory: 40Gi
              nvidia.com/gpu: 0
          command: ["bash", "run_store_pipeline.sh", "--data_dir", "{{data_dir}}", "--config_dir", "{{config_dir}}", "--output_dir", "{{output_dir}}"]
        - name: dind-daemon 
          image: registry.com.cn/com/1.12.6-dind
          resources: 
              requests: 
                  cpu: 20m 
                  memory: 512Mi 
          securityContext: 
              privileged: true 
          volumeMounts: 
            - name: docker-graph-storage 
              mountPath: /var/lib/docker 
          command: []

Based on the documentation I see, I should be able to connect to the docker-in-docker docker daemon by setting DOCKER_HOST = tcp://localhost:2375. However, I'm not able to do so.

I notice that if I test it locally and run these two containers in the same docker network, I'm able to connect them by specifying the actual container name (i.e. tcp://dind-daemon:2375). Thanks!