Issue in placing DataDog log annotation on deployment via ansible

2/21/2019

I am using ansible version 2.7 for kubernetes deployment. For sending logs to datadog on kubernetes one of the way is to configure annotations like below,

template:
    metadata:
      annotations:
        ad.datadoghq.com/nginx.logs: '[{"source":"nginx","service":"webapp"}]'

this works fine and I could see logs in DataDog.

However I would like to achieve above configuration via ansible deployment on kubernetes for which I have used below code

 template:
        metadata:
           annotations:
             ad.datadoghq.com/xxx.logs: "{{ lookup('template', './datadog.json.j2')}}"

and datadog.json.j2 looks like below

'[{{ '{' }}"source":"{{ sourcea }}"{{ ',' }} "service":"{{ serviceb }}"{{ '}' }}]'  **--> sourcea and serviceb are defined as vars**

However the resulting config on deployment is below

template:
    metadata:
      annotations:
        ad.datadoghq.com/yps.logs: |
          '[{"source":"test", "service":"test"}]'

and this config does not allow datadog agent to parse logs failing with below error

[ AGENT ] 2019-xx-xx xx10:50 UTC | ERROR | (kubelet.go:97 in parseKubeletPodlist) | Can't parse template for pod xxx-5645f7c66c-s9zj4: could not extract logs config: in logs: invalid character '\'' looking for beginning of value

if I use ansible code as below (using replace)

template:
        metadata:
           annotations:
             ad.datadoghq.com/xxx.logs: "{{ lookup('template', './datadog.json.j2', convert_data=False) | string | replace('\n','')}}"

it generates deployment config as below

template:
    metadata:
      annotations:
        ad.datadoghq.com/yps.logs: '''[{"source":"test", "service":"test"}]'''
      creationTimestamp: null
      labels:

Which also fails,

to configure the working config with ansible, I have to either remove leading pipe (|) or three quotes coming when using replace).

I would like to have jinja variables substitution in place so that I could configure deployment with desired source and service at deployment time.

kindly suggest

-- Ruchir Bharadwaj
ansible
datadog
jinja2
kubernetes

1 Answer

2/21/2019

By introducing space in datadog.json.j2 template definition .i.e.

 [{"source":"{{ sourcea }}"{{ ',' }} "service":"{{ serviceb }}"}] (space at start)

and running deployment again I got the working config as below

template:
    metadata:
      annotations:
        ad.datadoghq.com/yps.logs: ' [{"source":"test", "service":"test"}]'

However I am not able to understand the behaviour if anyone could help me understand it

-- Ruchir Bharadwaj
Source: StackOverflow