I am using the promtail helm chart to connect to a Loki server running on a different stack. I have Loki behind an Nginx ingress secured with basic auth.
I can't find any documentation on this, and it's very possible it's just my admittedly limited understanding of helm.
I'm simply trying to use basic auth to connect to the Loki instance while using a Kubernetes secret instead of plaintext credentials in the helm values
This works perfect:
snippets:
extraClientConfigs: |
basic_auth:
username: myusername
password: mypassword
I created a secret like this:
kubectl create secret generic loki-credentials -n monitoring --from-literal=password="mypassword" --from-literal=username="myusername"
and now I want to use that in the values.yaml
file.
This is what I got so far:
extraEnv:
- name: LOKI_USERNAME
valueFrom:
secretKeyRef:
name: loki-credentials
key: username
- name: LOKI_PASSWORD
valueFrom:
secretKeyRef:
name: loki-credentials
key: password
extraArgs:
- -client.external-labels=stack=development
- -config.expand-env
config:
serverPort: 3101
lokiAddress: myurl
snippets:
extraClientConfigs: |
basic_auth:
username: ${LOKI_USERNAME}
password: ${LOKI_PASSWORD}
I just get a 401 response.
Chart version: 3.1.0
Promtail version: 2.1.0
EDIT
Here is my ingress yaml:
controller:
replicaCount: 1
config:
force-ssl-redirect: "true"
use-forwarded-headers: "true"
service:
targetPorts:
http: http
https: http
annotations:
service.beta.kubernetes.io/aws-load-balancer-ssl-cert: "arn:aws:acm:us-west-2:123456:certificate/123456"
service.beta.kubernetes.io/aws-load-balancer-backend-protocol: "http"
service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "https"
enableHttp: false
enableHttps: true
type: LoadBalancer
loadBalancerSourceRanges:
- "0.0.0.0/0"
## Name of the ingress class to route through this controller
ingressClass: nginx-external
...and my loki values.yaml
ingress:
enabled: true
annotations:
kubernetes.io/ingress.class: "nginx-external"
nginx.ingress.kubernetes.io/auth-type: basic
nginx.ingress.kubernetes.io/auth-secret: loki-ingress-auth
nginx.ingress.kubernetes.io/auth-realm: "Authentication Required"
hosts:
- host: loki.mydomain.com
paths:
- "/"
tls: []
config:
auth_enabled: false
ingester:
chunk_idle_period: 3m
chunk_block_size: 262144
chunk_retain_period: 1m
max_transfer_retries: 3
lifecycler:
ring:
kvstore:
store: inmemory
replication_factor: 1
schema_config:
configs:
- from: 2021-03-06
store: boltdb-shipper
object_store: aws
schema: v11
index:
prefix: loki_index_
period: 24h
server:
http_listen_port: 3100
storage_config:
aws:
bucketnames: my-bucket-name
region: us-west-2
s3forcepathstyle: true
boltdb_shipper:
active_index_directory: /data/loki/boltdb-shipper-active
cache_location: /data/loki/boltdb-shipper-cache
shared_store: s3
chunk_store_config:
max_look_back_period: 0s
table_manager:
retention_deletes_enabled: false
retention_period: 0s
compactor:
working_directory: /data/loki/boltdb-shipper-compactor
shared_store: aws
replicas: 1
podAnnotations:
iam.amazonaws.com/role: "arn:aws:iam::123456:role/my-loki-role"
resources:
limits:
cpu: 500m
memory: 2G
requests:
cpu: 250m
memory: 1G
# The values to set in the PodDisruptionBudget spec
# If not set then a PodDisruptionBudget will not be created
podDisruptionBudget:
minAvailable: 1
My logs from the nginx pod that Loki is sitting behind:
2021/03/09 04:23:44 [error] 37#37: *925 user "myusername": password mismatch, client: xxx.xx.xxx.xxx, server: loki.mydomain.com, request: "POST /loki/api/v1/push HTTP/1.1", host: "loki.mydomain.com"
2021/03/09 04:23:44 [error] 37#37: *921 user "myusername": password mismatch, client: xxx.xx.xxx.xxx, server: loki.mydomain.com, request: "POST /loki/api/v1/push HTTP/1.1", host: "loki.mydomain.com"
xx.xxx.xxx.xx - myusername [09/Mar/2021:04:23:44 +0000] "POST /loki/api/v1/push HTTP/1.1" 401 172 "-" "promtail/2.1.0" 326 0.000 [monitoring-loki-3100] [] - - - - 63294b16fe010a8c9ec1d4684f0472f5
xxx.xx.xxx.xxx: - myusername [09/Mar/2021:04:23:44 +0000] "POST /loki/api/v1/push HTTP/1.1" 204 0 "-" "promtail/2.1.0" 2744 0.003 [monitoring-loki-3100] [] xxx.xx.xxx.xxx:3100 0 0.004 204 029e0a9d1ee6242cad8b9a6d2ee50940
2021/03/09 04:23:44 [error] 37#37: *925 user "myusername": password mismatch, client: xx.xxx.xxx.xx, server: loki.mydomain.com, request: "POST /loki/api/v1/push HTTP/1.1", host: "loki.mydomain.com"
xxx.xx.xxx.xxx - myusername [09/Mar/2021:04:23:44 +0000] "POST /loki/api/v1/push HTTP/1.1" 401 172 "-" "promtail/2.1.0" 325 0.000 [monitoring-loki-3100] [] - - - - b75a2cfcf6c62b81953dd4fb26f1a844
xxx.xx.xxx.xxx - myusername [09/Mar/2021:04:23:44 +0000] "POST /loki/api/v1/push HTTP/1.1" 204 0 "-" "promtail/2.1.0" 1513 0.014 [monitoring-loki-3100] [] xxx.xx.xxx.xxx:3100 0 0.016 204 0049965a49877cb5d336ac6ec869feb4
2021/03/09 04:23:45 [error] 36#36: *941 user "myusername": password mismatch, client: xxx.xx.xxx.xxx, server: loki.mydomain.com, request: "POST /loki/api/v1/push HTTP/1.1", host: "loki.c3dops.com"
xxx.xx.xxx.xxx - myusername [09/Mar/2021:04:23:45 +0000] "POST /loki/api/v1/push HTTP/1.1" 401 172 "-" "promtail/2.1.0" 326 0.000 [monitoring-loki-3100] [] - - - - e5954bd055db5b3e9bd3227f57651847
To give a bit of background to anyone new to Loki, as stated in the documentation: Loki does not come with any included authentication layer. Operators are expected to run an authenticating reverse proxy in front of your services, such as NGINX using basic auth or an OAuth2 proxy.
This basically means that you'll have to place something in between the client(s) and Loki to enforce e.g. basic authentication. In this case there's a Ingress (Nginx) acting as a reverse proxy with basic authentication.
To troubleshoot problems with authentication using Nginx there's a number of things to check:
annotations
.When it comes to using Nginx as Ingress and adding basic authentication this resource from the official docs is really helpful.
In general when creating Kubernetes secrets, especially from the command line using kubectl
you'll have to single quote the password if it contains special characters. This is so that the special characters wont be interpreted by your shell. More info here.