Promtail basic auth using kubernetes secret in helm values.yaml

3/3/2021

I am using the promtail helm chart to connect to a Loki server running on a different stack. I have Loki behind an Nginx ingress secured with basic auth.

I can't find any documentation on this, and it's very possible it's just my admittedly limited understanding of helm.

I'm simply trying to use basic auth to connect to the Loki instance while using a Kubernetes secret instead of plaintext credentials in the helm values

This works perfect:

  snippets:
    extraClientConfigs: |
      basic_auth:
        username: myusername
        password: mypassword

I created a secret like this:

kubectl create secret generic loki-credentials -n monitoring --from-literal=password="mypassword" --from-literal=username="myusername"

and now I want to use that in the values.yaml file.

This is what I got so far:

extraEnv:
  - name: LOKI_USERNAME
    valueFrom:
      secretKeyRef:
        name: loki-credentials
        key: username
  - name: LOKI_PASSWORD
    valueFrom:
      secretKeyRef:
        name: loki-credentials
        key: password

extraArgs:
  - -client.external-labels=stack=development
  - -config.expand-env

config:
  serverPort: 3101
  lokiAddress: myurl
  snippets:
    extraClientConfigs: |
      basic_auth:
        username: ${LOKI_USERNAME}
        password: ${LOKI_PASSWORD}

I just get a 401 response.

 Chart version: 3.1.0
 Promtail version: 2.1.0

EDIT

Here is my ingress yaml:

controller:
  replicaCount: 1

  config:
    force-ssl-redirect: "true"
    use-forwarded-headers: "true"

  service:
    targetPorts:
      http: http
      https: http
    annotations:
      service.beta.kubernetes.io/aws-load-balancer-ssl-cert: "arn:aws:acm:us-west-2:123456:certificate/123456"
      service.beta.kubernetes.io/aws-load-balancer-backend-protocol: "http"
      service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "https"

    enableHttp: false
    enableHttps: true
    type: LoadBalancer
    loadBalancerSourceRanges:
      - "0.0.0.0/0"

   ## Name of the ingress class to route through this controller
  ingressClass: nginx-external

...and my loki values.yaml

ingress:
  enabled: true
  annotations:
    kubernetes.io/ingress.class: "nginx-external"
    nginx.ingress.kubernetes.io/auth-type: basic
    nginx.ingress.kubernetes.io/auth-secret: loki-ingress-auth
    nginx.ingress.kubernetes.io/auth-realm: "Authentication Required"
  hosts:
    - host: loki.mydomain.com
      paths: 
        - "/"
  tls: []

config:
  auth_enabled: false
  ingester:
    chunk_idle_period: 3m
    chunk_block_size: 262144
    chunk_retain_period: 1m
    max_transfer_retries: 3
    lifecycler:
      ring:
        kvstore:
          store: inmemory
        replication_factor: 1
  
  schema_config:
    configs:
    - from: 2021-03-06
      store: boltdb-shipper
      object_store: aws
      schema: v11
      index:
        prefix: loki_index_
        period: 24h
        
  server:
    http_listen_port: 3100

  storage_config:
    aws:
      bucketnames: my-bucket-name
      region: us-west-2
      s3forcepathstyle: true

    boltdb_shipper:
      active_index_directory: /data/loki/boltdb-shipper-active
      cache_location: /data/loki/boltdb-shipper-cache
      shared_store: s3

  chunk_store_config:
    max_look_back_period: 0s
  
  table_manager:
    retention_deletes_enabled: false
    retention_period: 0s

  compactor:
    working_directory: /data/loki/boltdb-shipper-compactor
    shared_store: aws

replicas: 1

podAnnotations:
  iam.amazonaws.com/role: "arn:aws:iam::123456:role/my-loki-role"

resources:
  limits:
    cpu: 500m
    memory: 2G
  requests:
    cpu: 250m
    memory: 1G

# The values to set in the PodDisruptionBudget spec
# If not set then a PodDisruptionBudget will not be created
podDisruptionBudget:
  minAvailable: 1

My logs from the nginx pod that Loki is sitting behind:

2021/03/09 04:23:44 [error] 37#37: *925 user "myusername": password mismatch, client: xxx.xx.xxx.xxx, server: loki.mydomain.com, request: "POST /loki/api/v1/push HTTP/1.1", host: "loki.mydomain.com"
2021/03/09 04:23:44 [error] 37#37: *921 user "myusername": password mismatch, client: xxx.xx.xxx.xxx, server: loki.mydomain.com, request: "POST /loki/api/v1/push HTTP/1.1", host: "loki.mydomain.com"
xx.xxx.xxx.xx - myusername [09/Mar/2021:04:23:44 +0000] "POST /loki/api/v1/push HTTP/1.1" 401 172 "-" "promtail/2.1.0" 326 0.000 [monitoring-loki-3100] [] - - - - 63294b16fe010a8c9ec1d4684f0472f5
xxx.xx.xxx.xxx: - myusername [09/Mar/2021:04:23:44 +0000] "POST /loki/api/v1/push HTTP/1.1" 204 0 "-" "promtail/2.1.0" 2744 0.003 [monitoring-loki-3100] [] xxx.xx.xxx.xxx:3100 0 0.004 204 029e0a9d1ee6242cad8b9a6d2ee50940
2021/03/09 04:23:44 [error] 37#37: *925 user "myusername": password mismatch, client: xx.xxx.xxx.xx, server: loki.mydomain.com, request: "POST /loki/api/v1/push HTTP/1.1", host: "loki.mydomain.com"
xxx.xx.xxx.xxx - myusername [09/Mar/2021:04:23:44 +0000] "POST /loki/api/v1/push HTTP/1.1" 401 172 "-" "promtail/2.1.0" 325 0.000 [monitoring-loki-3100] [] - - - - b75a2cfcf6c62b81953dd4fb26f1a844
xxx.xx.xxx.xxx - myusername [09/Mar/2021:04:23:44 +0000] "POST /loki/api/v1/push HTTP/1.1" 204 0 "-" "promtail/2.1.0" 1513 0.014 [monitoring-loki-3100] [] xxx.xx.xxx.xxx:3100 0 0.016 204 0049965a49877cb5d336ac6ec869feb4
2021/03/09 04:23:45 [error] 36#36: *941 user "myusername": password mismatch, client: xxx.xx.xxx.xxx, server: loki.mydomain.com, request: "POST /loki/api/v1/push HTTP/1.1", host: "loki.c3dops.com"
xxx.xx.xxx.xxx - myusername [09/Mar/2021:04:23:45 +0000] "POST /loki/api/v1/push HTTP/1.1" 401 172 "-" "promtail/2.1.0" 326 0.000 [monitoring-loki-3100] [] - - - - e5954bd055db5b3e9bd3227f57651847
-- hhaslam11
kubernetes
kubernetes-helm
logging
prometheus

1 Answer

3/9/2021

To give a bit of background to anyone new to Loki, as stated in the documentation: Loki does not come with any included authentication layer. Operators are expected to run an authenticating reverse proxy in front of your services, such as NGINX using basic auth or an OAuth2 proxy.

This basically means that you'll have to place something in between the client(s) and Loki to enforce e.g. basic authentication. In this case there's a Ingress (Nginx) acting as a reverse proxy with basic authentication.

To troubleshoot problems with authentication using Nginx there's a number of things to check:

  • Logs of the Nginx Ingress Pod, check for authentication errors.
  • That the added Kubernetes Secret contains what you expect.
  • That you have configured the Ingress object with the needed annotations.

When it comes to using Nginx as Ingress and adding basic authentication this resource from the official docs is really helpful.

In general when creating Kubernetes secrets, especially from the command line using kubectl you'll have to single quote the password if it contains special characters. This is so that the special characters wont be interpreted by your shell. More info here.

-- mikejoh
Source: StackOverflow