K
Q

What host and port to use for http plugin for a Logstash container running in K8s pod?

7/11/2019

I am receiving this error Error: Cannot assign requested address in logstash when attempting to set up a pipeline that uses http input plugin.

I am trying to send data from a Python process to logstash using the Python requests http library. I am not sure which host and port to use in my logstash http input configs. Should I be using the defaults, the logstash pod ClusterIP service IP, the logstash pod IP, or something else?

input {
    http {
        host => "0.0.0.0"
        port => 80
    }
  }

The defaults url is 0.0.0.0:80 but I get a connection error on the Python side. I've also tried the url of the Logstash K8s pod that the logstash container is running in and get Error: Cannot assign requested address in the logstash container.

Edit: included logstash service details

Name:              central-logstash
Namespace:         default
Labels:            app=logstash
                   chart=logstash-1.10.0
                   heritage=Tiller
                   release=central-logstash
Annotations:       <none>
Selector:          app=logstash,release=central-logstash
Type:              ClusterIP
IP:                10.110.133.189
Port:              beats  5044/TCP
TargetPort:        beats/TCP
Endpoints:         192.168.0.79:5044
Session Affinity:  None
Events:            <none>
-- bbmhmmad
kubernetes
logstash
logstash-configuration
python
python-requests

Similar Questions

LoadBalancer service not reachable
Variable substitution in Prometheus yaml file
Istio ingress-controller via host network (bare-metal)
Kubernetes - Best way to initialize persistent volume for a database deployment

2 Answers

7/12/2019

As per your information, you are using helm to deploy and the version of the docker image from elastic on the stable chart is the 7.1.1

If you check the Dockerfile for that version

They create a user logstash here

And during the Dockerfile definition switch to root but returns at the end to that user, because of that you cannot use port 80

Try to use a different port like 8080 and adjust all other services that need to connect to logstash to use the new port or map that port on the service to use on the services level the 80 port but with the target pointing to the 8080

-- wolmi
Source: StackOverflow
7/18/2019

I ended up switching to TCP to avoid HTTP headers in my messages.

In my Logstash Helm configuration (https://github.com/helm/charts/tree/master/stable/logstash), I set up the service as such:

service:
  type: ClusterIP
  annotations: {}
  ports:
     tcp-data:
       port: 1514
       targetPort: tcp-data
       protocol: TCP
     tcp-event:
       port: 1515
       targetPort: tcp-event
       protocol: TCP

ports:
  - name: tcp-data
    containerPort: 1514
    protocol: TCP
  - name: tcp-event
    containerPort: 1515
    protocol: TCP

And the tcp plugin:

inputs:
 data: |-
    input {
        tcp {
            port => 1514
            type => json
        }
      }
 event: |-
    input {
        tcp {
            port => 1515
            type => json
        }
      }

And then on the Python side, I was able to use the socket library to send messages to Logstash using the Logstash clusterIP service's IP and port 1514 or 1515.

-- bbmhmmad
Source: StackOverflow