K
Q

Installing repository-se plugin on Elasticsearch

9/17/2018

We got EKS and worker nodes running of course on AWS. We are using gcr.io/google_containers/elasticsearch:v6.3.0. However now we want to add curator to take snapshots of ES indexes and store them in S3 bucket. For this ES needs a repository-s3 plugin.

So we decided to extend this image and create our own with the plugin installed. Dockerfile is:

FROM gcr.io/google_containers/elasticsearch:v6.3.0
RUN /usr/share/elasticsearch/bin/elasticsearch-plugin install --batch repository-s3

This returns:

Step 2/2 : RUN /usr/share/elasticsearch/bin/elasticsearch-plugin install --batch repository-s3
 ---> Running in 8d96a792a7e3
Exception in thread "main" java.lang.IllegalArgumentException: Could not resolve placeholder 'MINIMUM_MASTER_NODES'
    at org.elasticsearch.common.settings.PropertyPlaceholder.parseStringValue(PropertyPlaceholder.java:116)
    at org.elasticsearch.common.settings.PropertyPlaceholder.replacePlaceholders(PropertyPlaceholder.java:69)
    at org.elasticsearch.common.settings.Settings$Builder.replacePropertyPlaceholders(Settings.java:1263)
    at org.elasticsearch.common.settings.Settings$Builder.replacePropertyPlaceholders(Settings.java:1213)
    at org.elasticsearch.node.InternalSettingsPreparer.initializeSettings(InternalSettingsPreparer.java:128)
    at org.elasticsearch.node.InternalSettingsPreparer.prepareEnvironment(InternalSettingsPreparer.java:107)
    at org.elasticsearch.cli.EnvironmentAwareCommand.createEnv(EnvironmentAwareCommand.java:95)
    at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86)
    at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124)
    at org.elasticsearch.cli.MultiCommand.execute(MultiCommand.java:79)
    at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124)
    at org.elasticsearch.cli.Command.main(Command.java:90)
    at org.elasticsearch.plugins.PluginCli.main(PluginCli.java:48)
The command '/bin/sh -c /usr/share/elasticsearch/bin/elasticsearch-plugin install --batch repository-s3' returned a non-zero code: 1

The same thing happens when I try to install the plugin on the running es instances.

And I can sudo to elasticsearch user and run the command again and it still gives ends up with the same issue:

It appears environment is missing the following variables as when I add them in Dockerfile with ARG it appears to complete the image (certainly those values are hardcoded but just to prove the point):

MINIMUM_MASTER_NODES
HTTP_PORT
NODE_DATA
NODE_MASTER
NODE_NAME
TRANSPORT_PORT

Not sure what am I missing in this whole story.

-- Greg Hill
amazon-web-services
dockerfile
elasticsearch
kubernetes

Similar Questions

systemctl start kubelet deleting docker images from the master node
how to do the ibmmq replication in docker swarm or kubernetes?
Let's Encrypt on Traefik with Helm
Traefik Ingress (Kubernetes) not receiving letsencrypt certificates

2 Answers

9/17/2018

The plugin installer expects those variables to be set or read from the config. Those are parameters in your Elasticsearch configs. Which one is the directory where your elasticsearch.yml, jvm.options, and log4j2.properties files are? You can try:

  1. ES_PATH_CONF=/path/to/my/config /usr/share/elasticsearch/bin/elasticsearch-plugin install --batch repository-s3

  2. Set environment vars before running the script:

    export MINIMUM_MASTER_NODES=<value>
export HTTP_PORT=<value>
export NODE_DATA=<value>
export NODE_MASTER=<value>
export NODE_NAME=<value>
export TRANSPORT_PORT=<value>
/usr/share/elasticsearch/bin/elasticsearch-plugin install --batch repository-s3

  3. Add those parameters using the ENV keyword in your Dockerfile when you build the image. Dockerfile:

    FROM gcr.io/google_containers/elasticsearch:v6.3.0
ENV MINIMUM_MASTER_NODES=<value>
ENV HTTP_PORT=<value>
ENV NODE_DATA=<value>
ENV NODE_MASTER=<value>
ENV NODE_NAME=<value>
ENV TRANSPORT_PORT=<value>
RUN /usr/share/elasticsearch/bin/elasticsearch-plugin install --batch repository-s3
-- Rico
Source: StackOverflow
9/18/2018

Configuring the Dockerfile as below does the trick. It is important to have ARG rather than ENV as ARG is only used at build time. At run time it is elasticsearch provides the environment variables so we don't have to worry about that.

FROM gcr.io/google_containers/elasticsearch:v6.3.0
ARG MINIMUM_MASTER_NODES=${MINIMUM_MASTER_NODES}
ARG HTTP_PORT=${HTTP_PORT}
ARG NODE_DATA=${NODE_DATA}
ARG NODE_MASTER=${NODE_MASTER}
ARG NODE_NAME=${NODE_NAME}
ARG TRANSPORT_PORT=${TRANSPORT_PORT}
RUN su elasticsearch -c "/usr/share/elasticsearch/bin/elasticsearch-plugin install --batch repository-s3"
-- Greg Hill
Source: StackOverflow