I'm running a number of python apps as Replica Sets inside of kubernetes on Google Container Engine (gke). Along side them I've created the Datadog DaemonSet which launches a dd-agent on each node in my cluster.

Now I would like to use that agents dogstatsd for metrics logging from python apps as well as try out the new Datadog APM. If I just install the ddtrace python package and use it like documented it fills up my logs with

[2017-02-24 14:09:15,199] [5] [ddtrace.writer] [ERROR] cannot send spans: [Errno 110] Connection timed out
[2017-02-24 14:11:23,660] [5] [ddtrace.writer] [ERROR] cannot send spans: [Errno 110] Connection timed out

Clearly it don't have magical way to guess how to access port 8126/7777 of the ddagent pods.

Ive tried creating a Service which expose the ports:

---
apiVersion: v1
kind: Service
metadata:
  annotations:
    prometheus.io/scrape: 'true'
  labels:
    app: datadog-statsd
  name: datadog-statsd
spec:
  ports:
  - name: dogstatsd
    port: 8125
    targetPort: dogstatsdport
    protocol: UDP
  - name: ddapm
    port: 8126
    targetPort: ddtraceport
    protocol: TCP
  selector:
    app: dd-agent

but my python pods still don't seem to be able access for example os.environ['DATADOG_STATSD_PORT_8126_TCP_ADDR'] and .._PORT. They are defined and all, I just still get the connection timed out. If I connect to the dd-agent pods and enable tcpdump I also don't see any trafic on ports 8126 etc.

The dd-agent DaemonSet is defined like this:

apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: dd-agent
spec:
  template:
    metadata:
      labels:
        app: dd-agent
      name: dd-agent
    spec:
      containers:
      - image: datadog/docker-dd-agent:latest
        imagePullPolicy: Always
        name: dd-agent
        ports:
          - containerPort: 8125
            name: dogstatsdport
            protocol: UDP
          - containerPort: 8126
            name: ddtraceport
            protocol: TCP
        env:
          - name: API_KEY
            value: .....
          - name: KUBERNETES
            value: "yes"
          - name: SD_BACKEND
            value: docker
          - name: DD_APM_ENABLED
            value: "true"
        volumeMounts:
          - name: dockersocket
            mountPath: /var/run/docker.sock
          - name: procdir
            mountPath: /host/proc
            readOnly: true
          - name: cgroups
            mountPath: /host/sys/fs/cgroup
            readOnly: true
      volumes:
        - hostPath:
            path: /var/run/docker.sock
          name: dockersocket
        - hostPath:
            path: /proc
          name: procdir
        - hostPath:
            path: /sys/fs/cgroup
          name: cgroups