I am deploying HA-HDFS in Kubernetes Cluster. My K8S-cluster architecture is One Master Node and Two Worker Nodes. My HDFS has two namenodes, One active node, and one standBy Node. 3 datanodes, 3 zookeepers, 3 JounralNodes. My DataNodes goes to running state and then to CrashLoopBackOff. Here are logs for dataNode.
Configuring core
- Setting hadoop.proxyuser.hue.hosts=*
- Setting fs.defaultFS=hdfs://namenode:8020
- Setting ha.zookeeper.quorum=zookepper-0:2181,zookepper-1:2181,zookepper-2:2181
- Setting hadoop.http.staticuser.user=root
- Setting io.compression.codecs=org.apache.hadoop.io.compress.SnappyCodec
- Setting hadoop.proxyuser.zeppelin.hosts=*
- Setting hadoop.proxyuser.zeppelin.groups=*
- Setting hadoop.proxyuser.hue.groups=*
Configuring hdfs
- Setting dfs.datanode.data.dir=file:///hadoop/dfs/data
- Setting dfs.namenode.datanode.registration.ip-hostname-check=false
- Setting dfs.webhdfs.enabled=true
/entrypoint.sh: line 28: HDFS_CONF_dfs_ha_automatic-failover_enabled: bad substitution
Configuring yarn
- Setting yarn.timeline-service.enabled=true
- Setting yarn.scheduler.capacity.root.default.maximum-allocation-vcores=4
- Setting yarn.resourcemanager.system-metrics-publisher.enabled=true
- Setting yarn.resourcemanager.store.class=org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore
- Setting yarn.nodemanager.disk-health-checker.max-disk-utilization-per-disk-percentage=98.5
- Setting yarn.log.server.url=http://historyserver:8188/applicationhistory/logs/
- Setting yarn.resourcemanager.fs.state-store.uri=/rmstate
- Setting yarn.timeline-service.generic-application-history.enabled=true
- Setting yarn.log-aggregation-enable=true
- Setting yarn.resourcemanager.hostname=resourcemanager
- Setting yarn.scheduler.capacity.root.default.maximum-allocation-mb=8192
- Setting yarn.nodemanager.aux-services=mapreduce_shuffle
- Setting yarn.resourcemanager.resource_tracker.address=resourcemanager:8031
- Setting yarn.timeline-service.hostname=historyserver
- Setting yarn.resourcemanager.scheduler.address=resourcemanager:8030
- Setting yarn.resourcemanager.address=resourcemanager:8032
- Setting mapred.map.output.compress.codec=org.apache.hadoop.io.compress.SnappyCodec
- Setting yarn.nodemanager.remote-app-log-dir=/app-logs
- Setting yarn.resourcemanager.scheduler.class=org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler
- Setting mapreduce.map.output.compress=true
- Setting yarn.nodemanager.resource.memory-mb=16384
- Setting yarn.resourcemanager.recovery.enabled=true
- Setting yarn.nodemanager.resource.cpu-vcores=8
Configuring httpfs
Configuring kms
Configuring mapred
- Setting mapreduce.map.java.opts=-Xmx3072m
- Setting mapreduce.reduce.java.opts=-Xmx6144m
- Setting mapreduce.reduce.memory.mb=8192
- Setting yarn.app.mapreduce.am.env=HADOOP_MAPRED_HOME=/opt/hadoop-3.2.1/
- Setting mapreduce.map.memory.mb=4096
- Setting mapred.child.java.opts=-Xmx4096m
- Setting mapreduce.reduce.env=HADOOP_MAPRED_HOME=/opt/hadoop-3.2.1/
- Setting mapreduce.framework.name=yarn
- Setting mapreduce.map.env=HADOOP_MAPRED_HOME=/opt/hadoop-3.2.1/
Configuring for multihomed network
2021-08-02 12:17:25,734 INFO datanode.DataNode: registered UNIX signal handlers for [TERM, HUP, INT]
2021-08-02 12:17:26,724 INFO checker.ThrottledAsyncChecker: Scheduling a check for [DISK]file:/hadoop/dfs/data
2021-08-02 12:17:27,020 INFO impl.MetricsConfig: Loaded properties from hadoop-metrics2.properties
2021-08-02 12:17:27,170 INFO impl.MetricsSystemImpl: Scheduled Metric snapshot period at 10 second(s).
2021-08-02 12:17:27,170 INFO impl.MetricsSystemImpl: DataNode metrics system started
2021-08-02 12:17:27,565 INFO common.Util: dfs.datanode.fileio.profiling.sampling.percentage set to 0. Disabling file IO profiling
2021-08-02 12:17:27,572 INFO datanode.BlockScanner: Initialized block scanner with targetBytesPerSec 1048576
2021-08-02 12:17:27,583 INFO datanode.DataNode: Configured hostname is datanode-1.datanode.hdfs.svc.cluster.local
2021-08-02 12:17:27,583 INFO common.Util: dfs.datanode.fileio.profiling.sampling.percentage set to 0. Disabling file IO profiling
2021-08-02 12:17:27,590 INFO datanode.DataNode: Starting DataNode with maxLockedMemory = 0
2021-08-02 12:17:27,631 INFO datanode.DataNode: Opened streaming server at /0.0.0.0:9866
2021-08-02 12:17:27,636 INFO datanode.DataNode: Balancing bandwidth is 10485760 bytes/s
2021-08-02 12:17:27,636 INFO datanode.DataNode: Number threads for balancing is 50
2021-08-02 12:17:27,734 INFO util.log: Logging initialized @3185ms
2021-08-02 12:17:27,929 INFO server.AuthenticationFilter: Unable to initialize FileSignerSecretProvider, falling back to use random secrets.
2021-08-02 12:17:27,934 INFO http.HttpRequestLog: Http request log for http.requests.datanode is not defined
2021-08-02 12:17:27,947 INFO http.HttpServer2: Added global filter 'safety' (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter)
2021-08-02 12:17:27,951 INFO http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context datanode
2021-08-02 12:17:27,951 INFO http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context static
2021-08-02 12:17:27,951 INFO http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context logs
2021-08-02 12:17:27,990 INFO http.HttpServer2: Jetty bound to port 42767
2021-08-02 12:17:27,992 INFO server.Server: jetty-9.3.24.v20180605, build timestamp: 2018-06-05T17:11:56Z, git hash: 84205aa28f11a4f31f2a3b86d1bba2cc8ab69827
2021-08-02 12:17:28,064 INFO handler.ContextHandler: Started o.e.j.s.ServletContextHandler@5038d0b5{/logs,file:///opt/hadoop-3.2.1/logs/,AVAILABLE}
2021-08-02 12:17:28,065 INFO handler.ContextHandler: Started o.e.j.s.ServletContextHandler@2ad48653{/static,file:///opt/hadoop-3.2.1/share/hadoop/hdfs/webapps/static/,AVAILABLE}
2021-08-02 12:17:28,224 INFO handler.ContextHandler: Started o.e.j.w.WebAppContext@68034211{/,file:///opt/hadoop-3.2.1/share/hadoop/hdfs/webapps/datanode/,AVAILABLE}{/datanode}
2021-08-02 12:17:28,236 INFO server.AbstractConnector: Started ServerConnector@275bf9b3{HTTP/1.1,[http/1.1]}{localhost:42767}
2021-08-02 12:17:28,237 INFO server.Server: Started @3688ms
2021-08-02 12:17:28,596 INFO web.DatanodeHttpServer: Listening HTTP traffic on /0.0.0.0:9864
2021-08-02 12:17:28,606 INFO util.JvmPauseMonitor: Starting JVM pause monitor
2021-08-02 12:17:28,617 INFO datanode.DataNode: dnUserName = root
2021-08-02 12:17:28,618 INFO datanode.DataNode: supergroup = supergroup
2021-08-02 12:17:28,734 INFO ipc.CallQueueManager: Using callQueue: class java.util.concurrent.LinkedBlockingQueue, queueCapacity: 1000, scheduler: class org.apache.hadoop.ipc.DefaultRpcScheduler, ipcBackoff: false.
2021-08-02 12:17:28,759 INFO ipc.Server: Starting Socket Reader #1 for port 9867
2021-08-02 12:17:29,162 INFO datanode.DataNode: Opened IPC server at /0.0.0.0:9867
2021-08-02 12:17:29,192 INFO datanode.DataNode: Refresh request received for nameservices: null
2021-08-02 12:17:29,210 INFO datanode.DataNode: Starting BPOfferServices for nameservices: <default>
2021-08-02 12:17:29,230 INFO datanode.DataNode: Block pool <registering> (Datanode Uuid unassigned) service to namenode/10.108.44.52:8020 starting to offer service
2021-08-02 12:17:29,240 INFO ipc.Server: IPC Server Responder: starting
2021-08-02 12:17:29,241 INFO ipc.Server: IPC Server listener on 9867: starting
2021-08-02 12:17:29,538 INFO datanode.DataNode: Acknowledging ACTIVE Namenode during handshakeBlock pool <registering> (Datanode Uuid unassigned) service to namenode/10.108.44.52:8020
2021-08-02 12:17:29,544 INFO common.Storage: Using 1 threads to upgrade data directories (dfs.datanode.parallel.volumes.load.threads.num=1, dataDirs=1)
2021-08-02 12:17:30,343 INFO common.Storage: Lock on /hadoop/dfs/data/in_use.lock acquired by nodename 378@datanode-1.datanode.hdfs.svc.cluster.local
2021-08-02 12:17:30,978 INFO common.Storage: Analyzing storage directories for bpid BP-1854773598-10.32.0.21-1627895491824
2021-08-02 12:17:30,978 INFO common.Storage: Locking is disabled for /hadoop/dfs/data/current/BP-1854773598-10.32.0.21-1627895491824
2021-08-02 12:17:31,445 INFO datanode.DataNode: Setting up storage: nsid=982693042;bpid=BP-1854773598-10.32.0.21-1627895491824;lv=-57;nsInfo=lv=-65;cid=CID-080a4fec-11de-48d9-be9e-645d2655fa7e;nsid=982693042;c=1627895491824;bpid=BP-1854773598-10.32.0.21-1627895491824;dnuuid=6def0bcc-f4e8-45a6-99c3-6710be17351b
2021-08-02 12:17:31,691 INFO impl.FsDatasetImpl: Added new volume: DS-a4e899e1-656e-4115-919b-fae25c51846b
2021-08-02 12:17:31,691 INFO impl.FsDatasetImpl: Added volume - [DISK]file:/hadoop/dfs/data, StorageType: DISK
2021-08-02 12:17:31,701 INFO impl.FsDatasetImpl: Registered FSDatasetState MBean
2021-08-02 12:17:31,724 INFO checker.ThrottledAsyncChecker: Scheduling a check for /hadoop/dfs/data
2021-08-02 12:17:31,745 INFO checker.DatasetVolumeChecker: Scheduled health check for volume /hadoop/dfs/data
2021-08-02 12:17:31,748 INFO impl.FsDatasetImpl: Adding block pool BP-1854773598-10.32.0.21-1627895491824
2021-08-02 12:17:31,750 INFO impl.FsDatasetImpl: Scanning block pool BP-1854773598-10.32.0.21-1627895491824 on volume /hadoop/dfs/data...
2021-08-02 12:17:31,802 INFO impl.FsDatasetImpl: Cached dfsUsed found for /hadoop/dfs/data/current/BP-1854773598-10.32.0.21-1627895491824/current: 2048
2021-08-02 12:17:31,837 INFO impl.FsDatasetImpl: Time taken to scan block pool BP-1854773598-10.32.0.21-1627895491824 on /hadoop/dfs/data: 83ms
2021-08-02 12:17:31,838 INFO impl.FsDatasetImpl: Total time to scan all replicas for block pool BP-1854773598-10.32.0.21-1627895491824: 88ms
2021-08-02 12:17:31,843 INFO impl.FsDatasetImpl: Adding replicas to map for block pool BP-1854773598-10.32.0.21-1627895491824 on volume /hadoop/dfs/data...
2021-08-02 12:17:31,846 INFO impl.BlockPoolSlice: Replica Cache file: /hadoop/dfs/data/current/BP-1854773598-10.32.0.21-1627895491824/current/replicas doesn't exist
2021-08-02 12:17:31,850 INFO impl.FsDatasetImpl: Time to add replicas to map for block pool BP-1854773598-10.32.0.21-1627895491824 on volume /hadoop/dfs/data: 8ms
2021-08-02 12:17:31,851 INFO impl.FsDatasetImpl: Total time to add all replicas to map for block pool BP-1854773598-10.32.0.21-1627895491824: 10ms
2021-08-02 12:17:31,908 INFO datanode.VolumeScanner: VolumeScanner(/hadoop/dfs/data, DS-a4e899e1-656e-4115-919b-fae25c51846b): no suitable block pools found to scan. Waiting 1804233458 ms.
2021-08-02 12:17:31,949 INFO datanode.DirectoryScanner: Periodic Directory Tree Verification scan starting at 8/2/21 1:56 PM with interval of 21600000ms
2021-08-02 12:17:31,967 INFO datanode.DataNode: Block pool BP-1854773598-10.32.0.21-1627895491824 (Datanode Uuid 6def0bcc-f4e8-45a6-99c3-6710be17351b) service to namenode/10.108.44.52:8020 beginning handshake with NN
2021-08-02 12:17:32,058 INFO datanode.DataNode: Block pool Block pool BP-1854773598-10.32.0.21-1627895491824 (Datanode Uuid 6def0bcc-f4e8-45a6-99c3-6710be17351b) service to namenode/10.108.44.52:8020 successfully registered with NN
2021-08-02 12:17:32,058 INFO datanode.DataNode: For namenode namenode/10.108.44.52:8020 using BLOCKREPORT_INTERVAL of 21600000msec CACHEREPORT_INTERVAL of 10000msec Initial delay: 0msec; heartBeatInterval=3000
2021-08-02 12:17:32,298 INFO datanode.DataNode: Successfully sent block report 0xe102947467610cac, containing 1 storage report(s), of which we sent 1. The reports had 0 total blocks and used 1 RPC(s). This took 11 msec to generate and 62 msecs for RPC and NN processing. Got back one command: FinalizeCommand/5.
2021-08-02 12:17:32,298 INFO datanode.DataNode: Got finalize command for block pool BP-1854773598-10.32.0.21-1627895491824
2021-08-02 12:21:56,012 WARN datanode.DataNode: Block pool BP-1854773598-10.32.0.21-1627895491824 (Datanode Uuid 6def0bcc-f4e8-45a6-99c3-6710be17351b) service to namenode/10.108.44.52:8020 is shutting down
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.protocol.UnregisteredNodeException): Unregistered server: DatanodeRegistration(10.47.0.0:9866, datanodeUuid=6def0bcc-f4e8-45a6-99c3-6710be17351b, infoPort=9864, infoSecurePort=0, ipcPort=9867, storageInfo=lv=-57;cid=CID-080a4fec-11de-48d9-be9e-645d2655fa7e;nsid=982693042;c=1627895491824)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.verifyRequest(NameNodeRpcServer.java:1704)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.sendHeartbeat(NameNodeRpcServer.java:1552)
at org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolServerSideTranslatorPB.sendHeartbeat(DatanodeProtocolServerSideTranslatorPB.java:119)
at org.apache.hadoop.hdfs.protocol.proto.DatanodeProtocolProtos$DatanodeProtocolService$2.callBlockingMethod(DatanodeProtocolProtos.java:31662)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:528)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1070)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:999)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:927)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2915)
at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1545)
at org.apache.hadoop.ipc.Client.call(Client.java:1491)
at org.apache.hadoop.ipc.Client.call(Client.java:1388)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118)
at com.sun.proxy.$Proxy16.sendHeartbeat(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolClientSideTranslatorPB.sendHeartbeat(DatanodeProtocolClientSideTranslatorPB.java:168)
at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.sendHeartBeat(BPServiceActor.java:517)
at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:648)
at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:849)
at java.lang.Thread.run(Thread.java:748)
2021-08-02 12:21:56,015 WARN datanode.DataNode: Ending block pool service for: Block pool BP-1854773598-10.32.0.21-1627895491824 (Datanode Uuid 6def0bcc-f4e8-45a6-99c3-6710be17351b) service to namenode/10.108.44.52:8020
2021-08-02 12:21:56,120 INFO datanode.DataNode: Removed Block pool BP-1854773598-10.32.0.21-1627895491824 (Datanode Uuid 6def0bcc-f4e8-45a6-99c3-6710be17351b)
2021-08-02 12:21:56,121 INFO impl.FsDatasetImpl: Removing block pool BP-1854773598-10.32.0.21-1627895491824
2021-08-02 12:21:59,466 WARN impl.FsDatasetImpl: Lock held time above threshold: lock identifier: org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl lockHeldTimeMs=3344 ms. Suppressed 0 lock warnings. The stack trace is: java.lang.Thread.getStackTrace(Thread.java:1559)
org.apache.hadoop.util.StringUtils.getStackTrace(StringUtils.java:1032)
org.apache.hadoop.util.InstrumentedLock.logWarning(InstrumentedLock.java:148)
org.apache.hadoop.util.InstrumentedLock.check(InstrumentedLock.java:186)
org.apache.hadoop.util.InstrumentedLock.unlock(InstrumentedLock.java:133)
org.apache.hadoop.util.AutoCloseableLock.release(AutoCloseableLock.java:84)
org.apache.hadoop.util.AutoCloseableLock.close(AutoCloseableLock.java:96)
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.shutdownBlockPool(FsDatasetImpl.java:2819)
org.apache.hadoop.hdfs.server.datanode.DataNode.shutdownBlockPool(DataNode.java:1644)
org.apache.hadoop.hdfs.server.datanode.BPOfferService.shutdownActor(BPOfferService.java:473)
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.cleanUp(BPServiceActor.java:594)
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:861)
java.lang.Thread.run(Thread.java:748)
2021-08-02 12:21:59,467 WARN fs.CachingGetSpaceUsed: Thread Interrupted waiting to refresh disk information: sleep interrupted
2021-08-02 12:22:01,468 WARN datanode.DataNode: Exiting Datanode
2021-08-02 12:22:01,492 INFO datanode.DataNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down DataNode at datanode-1.datanode.hdfs.svc.cluster.local/10.36.0.0
************************************************************/
And the yaml file for my DataNode is:
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: {{.Values.datanode.name}}
namespace: {{.Values.namespace}}
spec:
serviceName: "{{.Values.datanode.service.name}}"
replicas: {{.Values.datanode.replicas}}
selector:
matchLabels:
app: {{.Values.datanode.name}}
template:
metadata:
labels:
app: {{.Values.datanode.name}}
spec:
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: "app"
operator: In
values:
- {{.Values.datanode.name}}
topologyKey: "kubernetes.io/hostname"
containers:
- env:
- name: CORE_CONF_fs_defaultFS
value: hdfs://namenode:8020
- name: CORE_CONF_hadoop_http_staticuser_user
value: root
- name: CORE_CONF_hadoop_proxyuser_hue_groups
value: '*'
- name: CORE_CONF_hadoop_proxyuser_hue_hosts
value: '*'
- name: CORE_CONF_hadoop_proxyuser_zeppelin_groups
value: '*'
- name: CORE_CONF_hadoop_proxyuser_zeppelin_hosts
value: '*'
- name: CORE_CONF_io_compression_codecs
value: org.apache.hadoop.io.compress.SnappyCodec
- name: CORE_CONF_ha_zookeeper_quorum
value: zookepper-0:2181,zookepper-1:2181,zookepper-2:2181
- name: HDFS_CONF_dfs_namenode_acls_enabled
value: ' true'
- name: HDFS_CONF_dfs_namenode_datanode_registration_ip___hostname___check
value: "false"
- name: HDFS_CONF_dfs_permissions_enabled
value: "false"
- name: HDFS_CONF_dfs_replication
value: "6"
- name: HDFS_CONF_dfs_webhdfs_enabled
value: "true"
- name: HDFS_CONF_dfs_ha_automatic-failover_enabled
value: "true"
- name: HIVE_SITE_CONF_datanucleus_autoCreateSchema
value: "false"
- name: HIVE_SITE_CONF_hive_metastore_uris
value: thrift://hive-metastore:9083
- name: HIVE_SITE_CONF_javax_jdo_option_ConnectionDriverName
value: org.postgresql.Driver
- name: HIVE_SITE_CONF_javax_jdo_option_ConnectionPassword
value: hive
- name: HIVE_SITE_CONF_javax_jdo_option_ConnectionURL
value: jdbc:postgresql://hive-metastore-postgresql/metastore
- name: HIVE_SITE_CONF_javax_jdo_option_ConnectionUserName
value: hive
- name: MAPRED_CONF_mapred_child_java_opts
value: -Xmx4096m
- name: MAPRED_CONF_mapreduce_framework_name
value: yarn
- name: MAPRED_CONF_mapreduce_map_env
value: HADOOP_MAPRED_HOME=/opt/hadoop-3.2.1/
- name: MAPRED_CONF_mapreduce_map_java_opts
value: -Xmx3072m
- name: MAPRED_CONF_mapreduce_map_memory_mb
value: "4096"
- name: MAPRED_CONF_mapreduce_reduce_env
value: HADOOP_MAPRED_HOME=/opt/hadoop-3.2.1/
- name: MAPRED_CONF_mapreduce_reduce_java_opts
value: -Xmx6144m
- name: MAPRED_CONF_mapreduce_reduce_memory_mb
value: "8192"
- name: MAPRED_CONF_yarn_app_mapreduce_am_env
value: HADOOP_MAPRED_HOME=/opt/hadoop-3.2.1/
- name: SERVICE_PRECONDITION
value: namenode:9870
- name: YARN_CONF_mapred_map_output_compress_codec
value: org.apache.hadoop.io.compress.SnappyCodec
- name: YARN_CONF_mapreduce_map_output_compress
value: "true"
- name: YARN_CONF_yarn_log___aggregation___enable
value: "true"
- name: YARN_CONF_yarn_log_server_url
value: http://historyserver:8188/applicationhistory/logs/
- name: YARN_CONF_yarn_nodemanager_aux___services
value: mapreduce_shuffle
- name: YARN_CONF_yarn_nodemanager_disk___health___checker_max___disk___utilization___per___disk___percentage
value: "98.5"
- name: YARN_CONF_yarn_nodemanager_remote___app___log___dir
value: /app-logs
- name: YARN_CONF_yarn_nodemanager_resource_cpu___vcores
value: "8"
- name: YARN_CONF_yarn_nodemanager_resource_memory___mb
value: "16384"
- name: YARN_CONF_yarn_resourcemanager_address
value: resourcemanager:8032
- name: YARN_CONF_yarn_resourcemanager_fs_state___store_uri
value: /rmstate
- name: YARN_CONF_yarn_resourcemanager_hostname
value: resourcemanager
- name: YARN_CONF_yarn_resourcemanager_recovery_enabled
value: "true"
- name: YARN_CONF_yarn_resourcemanager_resource__tracker_address
value: resourcemanager:8031
- name: YARN_CONF_yarn_resourcemanager_scheduler_address
value: resourcemanager:8030
- name: YARN_CONF_yarn_resourcemanager_scheduler_class
value: org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler
- name: YARN_CONF_yarn_resourcemanager_store_class
value: org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore
- name: YARN_CONF_yarn_resourcemanager_system___metrics___publisher_enabled
value: "true"
- name: YARN_CONF_yarn_scheduler_capacity_root_default_maximum___allocation___mb
value: "8192"
- name: YARN_CONF_yarn_scheduler_capacity_root_default_maximum___allocation___vcores
value: "4"
- name: YARN_CONF_yarn_timeline___service_enabled
value: "true"
- name: YARN_CONF_yarn_timeline___service_generic___application___history_enabled
value: "true"
- name: YARN_CONF_yarn_timeline___service_hostname
value: historyserver
image: {{.Values.datanode.image}}
name: {{.Values.datanode.name}}
ports:
- containerPort: 9864
- containerPort: 50075
resources: {}
volumeMounts:
- mountPath: /hadoop/dfs/data
name: datanode
restartPolicy: Always
volumeClaimTemplates:
- metadata:
name: datanode
spec:
accessModes: [ "ReadWriteMany"]
storageClassName: {{.Values.storageClassName}}
resources:
requests:
storage: 1Gi
status:
replicas: {{.Values.namenode.replicas}}