hadoop job 未跑满资源_mapreduce任务占满整个集群资源
贊同來自:
哦，我把yarn.nodemanager.pmem-check-enabled設置為true后，再次運行任務就報錯了。
您說的使用量超過limit 兩倍時，才啟動kill。應該指得是虛擬內存的使用超出虛擬內存的兩倍，因為當物理內存不夠用時，任務就會使用虛擬內內存，而一旦虛擬內存超過設置的兩倍時，isOverLimit?就是true了，這樣就會觸發kill?container的動作了。
下面是我測試的報錯信息：
16/03/15 18:51:58 INFO mapreduce.Job: Task Id : attempt_1458038021988_0002_m_000000_0, Status : FAILED
Container [pid=9781,containerID=container_1458038021988_0002_01_000002] is running beyond physical memory limits. Current usage: 1.1 GB of 1 GB physical memory used; 4.9 GB of 2 GB virtual memory used. Killing container.
Dump of the process-tree for container_1458038021988_0002_01_000002 :
|- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
|- 9781 9779 9781 9781 (bash) 0 0 108646400 301 /bin/bash -c /opt/beh-6.5/core/jdk/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN? -Xmx4g -Djava.io.tmpdir=/opt/beh-6.5/metadata/yarn/usercache/user1/appcache/application_1458038021988_0002/container_1458038021988_0002_01_000002/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/opt/beh-6.5/logs/yarn/userlogs/application_1458038021988_0002/container_1458038021988_0002_01_000002 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA org.apache.hadoop.mapred.YarnChild 192.168.6.153 59177 attempt_1458038021988_0002_m_000000_0 2 1>/opt/beh-6.5/logs/yarn/userlogs/application_1458038021988_0002/container_1458038021988_0002_01_000002/stdout 2>/opt/beh-6.5/logs/yarn/userlogs/application_1458038021988_0002/container_1458038021988_0002_01_000002/stderr
|- 9785 9781 9781 9781 (java) 570 45 5128753152 286023 /opt/beh-6.5/core/jdk/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx4g -Djava.io.tmpdir=/opt/beh-6.5/metadata/yarn/usercache/user1/appcache/application_1458038021988_0002/container_1458038021988_0002_01_000002/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/opt/beh-6.5/logs/yarn/userlogs/application_1458038021988_0002/container_1458038021988_0002_01_000002 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA org.apache.hadoop.mapred.YarnChild 192.168.6.153 59177 attempt_1458038021988_0002_m_000000_0 2
Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
16/03/15 18:51:58 INFO mapreduce.Job: Task Id : attempt_1458038021988_0002_m_000000_1, Status : FAILED
Container [pid=9914,containerID=container_1458038021988_0002_01_000003] is running beyond physical memory limits. Current usage: 1.6 GB of 1 GB physical memory used; 4.9 GB of 2 GB virtual memory used. Killing container.
Dump of the process-tree for container_1458038021988_0002_01_000003 :
|- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
|- 9914 9912 9914 9914 (bash) 0 0 108646400 302 /bin/bash -c /opt/beh-6.5/core/jdk/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN? -Xmx4g -Djava.io.tmpdir=/opt/beh-6.5/metadata/yarn/usercache/user1/appcache/application_1458038021988_0002/container_1458038021988_0002_01_000003/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/opt/beh-6.5/logs/yarn/userlogs/application_1458038021988_0002/container_1458038021988_0002_01_000003 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA org.apache.hadoop.mapred.YarnChild 192.168.6.153 59177 attempt_1458038021988_0002_m_000000_1 3 1>/opt/beh-6.5/logs/yarn/userlogs/application_1458038021988_0002/container_1458038021988_0002_01_000003/stdout 2>/opt/beh-6.5/logs/yarn/userlogs/application_1458038021988_0002/container_1458038021988_0002_01_000003/stderr
|- 9918 9914 9914 9914 (java) 884 65 5106462720 416443 /opt/beh-6.5/core/jdk/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx4g -Djava.io.tmpdir=/opt/beh-6.5/metadata/yarn/usercache/user1/appcache/application_1458038021988_0002/container_1458038021988_0002_01_000003/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/opt/beh-6.5/logs/yarn/userlogs/application_1458038021988_0002/container_1458038021988_0002_01_000003 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA org.apache.hadoop.mapred.YarnChild 192.168.6.153 59177 attempt_1458038021988_0002_m_000000_1 3
Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
16/03/15 18:51:58 INFO mapreduce.Job: Task Id : attempt_1458038021988_0002_m_000000_2, Status : FAILED
Container [pid=10150,containerID=container_1458038021988_0002_01_000004] is running beyond physical memory limits. Current usage: 1.6 GB of 1 GB physical memory used; 4.9 GB of 2 GB virtual memory used. Killing container.
Dump of the process-tree for container_1458038021988_0002_01_000004 :
|- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
|- 10150 10148 10150 10150 (bash) 0 0 108646400 302 /bin/bash -c /opt/beh-6.5/core/jdk/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN? -Xmx4g -Djava.io.tmpdir=/opt/beh-6.5/metadata/yarn/usercache/user1/appcache/application_1458038021988_0002/container_1458038021988_0002_01_000004/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/opt/beh-6.5/logs/yarn/userlogs/application_1458038021988_0002/container_1458038021988_0002_01_000004 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA org.apache.hadoop.mapred.YarnChild 192.168.6.153 59177 attempt_1458038021988_0002_m_000000_2 4 1>/opt/beh-6.5/logs/yarn/userlogs/application_1458038021988_0002/container_1458038021988_0002_01_000004/stdout 2>/opt/beh-6.5/logs/yarn/userlogs/application_1458038021988_0002/container_1458038021988_0002_01_000004/stderr
|- 10154 10150 10150 10150 (java) 802 64 5106597888 418707 /opt/beh-6.5/core/jdk/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx4g -Djava.io.tmpdir=/opt/beh-6.5/metadata/yarn/usercache/user1/appcache/application_1458038021988_0002/container_1458038021988_0002_01_000004/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/opt/beh-6.5/logs/yarn/userlogs/application_1458038021988_0002/container_1458038021988_0002_01_000004 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA org.apache.hadoop.mapred.YarnChild 192.168.6.153 59177 attempt_1458038021988_0002_m_000000_2 4
Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
16/03/15 18:51:58 INFO mapreduce.Job: Task Id : attempt_1458038021988_0002_m_000000_3, Status : FAILED
Container [pid=4373,containerID=container_1458038021988_0002_01_000005] is running beyond physical memory limits. Current usage: 1.6 GB of 1 GB physical memory used; 4.9 GB of 2 GB virtual memory used. Killing container.
Dump of the process-tree for container_1458038021988_0002_01_000005 :
|- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
|- 4377 4373 4373 4373 (java) 752 73 5141807104 419857 /opt/beh-6.5/core/jdk/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx4g -Djava.io.tmpdir=/opt/beh-6.5/metadata/yarn/usercache/user1/appcache/application_1458038021988_0002/container_1458038021988_0002_01_000005/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/opt/beh-6.5/logs/yarn/userlogs/application_1458038021988_0002/container_1458038021988_0002_01_000005 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA org.apache.hadoop.mapred.YarnChild 192.168.6.153 59177 attempt_1458038021988_0002_m_000000_3 5
|- 4373 4371 4373 4373 (bash) 0 0 108650496 302 /bin/bash -c /opt/beh-6.5/core/jdk/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN? -Xmx4g -Djava.io.tmpdir=/opt/beh-6.5/metadata/yarn/usercache/user1/appcache/application_1458038021988_0002/container_1458038021988_0002_01_000005/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/opt/beh-6.5/logs/yarn/userlogs/application_1458038021988_0002/container_1458038021988_0002_01_000005 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA org.apache.hadoop.mapred.YarnChild 192.168.6.153 59177 attempt_1458038021988_0002_m_000000_3 5 1>/opt/beh-6.5/logs/yarn/userlogs/application_1458038021988_0002/container_1458038021988_0002_01_000005/stdout 2>/opt/beh-6.5/logs/yarn/userlogs/application_1458038021988_0002/container_1458038021988_0002_01_000005/stderr
Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
16/03/15 18:51:58 INFO mapreduce.Job: Job job_1458038021988_0002 failed with state FAILED due to:
總結
以上是生活随笔為你收集整理的hadoop job 未跑满资源_mapreduce任务占满整个集群资源的全部內容，希望文章能夠幫你解決所遇到的問題。
- 上一篇： 树莓派模拟电路_9.树莓派3B+ PWM
- 下一篇： 学科实践活动感悟50字_中学生学科实践活