how to limit Apache Flink task manager log size

4/18/2020

I am using duc to check the disk usage and finally find out Apache Flink takes more than 2GB,so how to limit the log size less than 100MB? My Apache Flink deploy in Kubernetes(v1.15.2) cluster.

[root@uat-k8s-01 opt]# duc ls -Fg /var/lib/docker/overlay2/d1f441865d83867a21dd1dc0b11da2c75ffe1efe39209770cbd5e12e386df065/diff/opt/flink/log/
  1.9G flink--taskexecutor-0-flink-taskmanager-7f9df8fbf6-79746.log  [+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++]
  4.0K flink--taskexecutor-0-flink-taskmanager-7f9df8fbf6-79746.out  [                                                                                                                           ]

I am already searching from internet and tweak my config like this,but still not work:

flink-conf.yaml:
jobmanager.rpc.address: flink-jobmanager
taskmanager.numberOfTaskSlots: 6
blob.server.port: 6124
jobmanager.rpc.port: 6123
taskmanager.rpc.port: 6122
jobmanager.heap.size: 1024m
taskmanager.memory.process.size: 1024m
log4j.properties:
log4j.rootLogger=INFO, file
log4j.logger.akka=INFO
log4j.logger.org.apache.kafka=INFO
log4j.logger.org.apache.hadoop=INFO
log4j.logger.org.apache.zookeeper=INFO
log4j.appender.file=org.apache.log4j.RollingFileAppender
log4j.appender.file.file=${log.file}
log4j.appender.file.MaxFileSize=50MB
log4j.appender.file.MaxBackupIndex=1
log4j.appender.file.layout=org.apache.log4j.PatternLayout
log4j.appender.file.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss,SSS} %-5p %-60c %x - %m%n
log4j.logger.org.apache.flink.shaded.akka.org.jboss.netty.channel.DefaultChannelPipeline=ERROR, file
-- Dolphin
kubernetes

0 Answers