K
Q

Debugging an unnecessary newline character in a Kubernetes secret

3/7/2018

I have an environment variable called GOOGLE_MAPS_DIRECTIONS_API_KEY, populated by a Kubernetes secret YAML:

apiVersion: v1
kind: Secret
metadata:
  name: google-maps-directions-api-secret
type: Opaque
data:
  GOOGLE_MAPS_DIRECTIONS_API_KEY: QUl...QbUpqTHNJ

The secret was created by copy-pasting the result of running echo -n "AIz..." | base64 on my API key. I've provided the beginning and the end of the key in this code snippet, to show that there is no newline in the key included in the secret file.

Here is what I see when I run cat google-maps-directions-api-key-secret.yaml | hexdump -C:

00000000  61 70 69 56 65 72 73 69  6f 6e 3a 20 76 31 0a 6b  |apiVersion: v1.k|
00000010  69 6e 64 3a 20 53 65 63  72 65 74 0a 6d 65 74 61  |ind: Secret.meta|
00000020  64 61 74 61 3a 0a 20 20  6e 61 6d 65 3a 20 67 6f  |data:.  name: go|
00000030  6f 67 6c 65 2d 6d 61 70  73 2d 64 69 72 65 63 74  |ogle-maps-direct|
00000040  69 6f 6e 73 2d 61 70 69  2d 73 65 63 72 65 74 0a  |ions-api-secret.|
00000050  74 79 70 65 3a 20 4f 70  61 71 75 65 0a 64 61 74  |type: Opaque.dat|
00000060  61 3a 0a 20 20 47 4f 4f  47 4c 45 5f 4d 41 50 53  |a:.  GOOGLE_MAPS|
00000070  5f 44 49 52 45 43 54 49  4f 4e 53 5f 41 50 49 5f  |_DIRECTIONS_API_|
00000080  4b 45 59 3a 20 51 55 6c  36 59 56 4e 35 51 7a 68  |KEY: QUl6YVN5Qzh|
...
000000b0  51 62 55 70 71 54 48 4e  4a                       |QbUpqTHNJ|
000000b9

But! When I step into a Node.JS interpreter inside of the pod, I see the following:

> process.env.GOOGLE_MAPS_DIRECTIONS_API_KEY
'AIz...jLsI\n'

There is an auxiliary newline character appended to the end of the string!

This is, frankly, extremely frustrating. I have several questions on this subject.

  • Can you spot my error? E.g. at what point in the secret propagation pipeline am I accidentally inserting that newline?
  • What Unix command should I use to print a newline character to console in such a way that it is interpreted literally (as a \n), so that I can actually see it?
  • Is it considered bad practice to inject code removing trailing newlines from environment variables into my container image? I know this is not technically correct, but this hurts like hell.
-- Aleksey Bilogur
base64
kubernetes

Similar Questions

Live Update of Log Rotation Policy in a Kubernetes Cluster
Kubernetes reports ImagePullBackOff for pod on minikube
Kubernetes 1.9 can't initialize SparkContext
using web API from Heapster in kubernetes

2 Answers

3/7/2018

If you previously created the secret without the -n option to echo, verify the Secret persisted in the API (kubectl get secret/google-maps-directions-api-secret -o yaml) matches the secret in your yaml file, and also verify the consuming app has been redeployed since the secret was updated with the correct value

-- Jordan Liggitt
Source: StackOverflow
3/7/2018

I don't see anything odd with how your secret looks. As you alluded to, the first thing I would do is exec into the pod, drop into bash, and echo out the environment variable to confirm it's propagated incorrectly. After doing a quick test, the newline should show up fine with a printf:

printf '%s' $GOOGLE_MAPS_DIRECTIONS_API_KEY

If it looks fine when printing it from bash, then the issue is with how node is interpreting it. If it looks messed up, then you need to take another look at how you're generating it.

FYI if the result of process.env is actually your API key, you should revoke it ASAP as you just published it in your question.

As for whether it's bad practice to strip newlines, yes. This can cause unexpected issues down the line if an actual piece of secret information contains a newline.

-- Grant David Bachman
Source: StackOverflow