I'm running an Elasticsearch V6.3.2 cluster on Kubernetes, and I've installed the repository-s3 plugin. I've added my AWS Access Key and Secret Key to the elasticsearch.keystore.

I'm trying to register my snapshot repository with:

curl -X PUT -H "Content-Type: application/json" -d '{ "type": "s3", "settings": { "bucket": "efk-snapshots-k8s" } }' "localhost:9200/_snapshot/snap123"

The above request fails with:

    {
      "error":{
        "root_cause":[
          {
            "type":"amazon_s3_exception",
            "reason":"amazon_s3_exception: Forbidden (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden; Request ID: 962AE79FCC57FA8E; S3 Extended Request ID: kyx9TnYewW8NUk35CsJRLy7SF7iCsg515zhXtOMjz4EdsxGUBJYaqHgs/e2SZCatEWQHwLFd39k=)"
          }
        ],
        "type":"blob_store_exception",
        "reason":"Failed to check if blob [master.dat-temp] exists",
        "caused_by":{
          "type":"amazon_s3_exception",
          "reason":"amazon_s3_exception: Forbidden (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden; Request ID: 962AE79FCC57FA8E; S3 Extended Request ID: kyx9TnYewW8NUk35CsJRLy7SF7iCsg515zhXtOMjz4EdsxGUBJYaqHgs/e2SZCatEWQHwLFd39k=)"
        }
      },
      "status":500
    }

I have granted appropriate permissions with the IAM policy, even so far that I tried giving it complete S3 access, but still no luck. What's going wrong here?

-EDIT:

Here's my elasticsearch.yml

     thread_pool.bulk.queue_size: 2000
    thread_pool.index.size: 2
    thread_pool.index.queue_size: 2000
    network.host: 0.0.0.0
    cluster:
      name: ${CLUSTER_NAME}
    node:
      master: ${NODE_MASTER}
      data: ${NODE_DATA}
      name: ${NODE_NAME}
      ingest: ${NODE_INGEST}
      max_local_storage_nodes: ${MAX_LOCAL_STORAGE_NODES}
    processors: ${PROCESSORS:1}
    path:
      data: /data/data
      logs: /data/log
      repo: ${REPO_LOCATIONS}
    bootstrap:
      memory_lock: ${MEMORY_LOCK}
    http:
      enabled: ${HTTP_ENABLE}
      compression: true
      cors:
        enabled: ${HTTP_CORS_ENABLE}
        allow-origin: ${HTTP_CORS_ALLOW_ORIGIN}
    discovery:
      zen:
        ping.unicast.hosts: ${DISCOVERY_SERVICE}
        minimum_master_nodes: ${NUMBER_OF_MASTERS}