首页 新闻 会员 周边

Hadoop3.1.1运行wordcount出错 该怎么解决

0
悬赏园豆:50 [已解决问题] 解决于 2018-12-11 09:54

[root@localhost ~]# jps
10049 Jps
7938 ResourceManager
8083 NodeManager
7447 DataNode
7309 NameNode
7678 SecondaryNameNode
[root@localhost ~]# hadoop fs -rmr /output
rmr: DEPRECATED: Please use '-rm -r' instead.
^[[ADeleted /output
[root@localhost ~]# hadoop jar /usr/local/hadoop/hadoop-3.1.1/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.1.1.jar wordcount /test /output
2018-12-09 22:01:31,884 INFO client.RMProxy: Connecting to ResourceManager at localhost/127.0.0.1:8032
2018-12-09 22:01:32,752 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/root/.staging/job_1544410258618_0005
2018-12-09 22:01:33,056 INFO input.FileInputFormat: Total input files to process : 1
2018-12-09 22:01:33,621 INFO mapreduce.JobSubmitter: number of splits:1
2018-12-09 22:01:33,664 INFO Configuration.deprecation: yarn.resourcemanager.system-metrics-publisher.enabled is deprecated. Instead, use yarn.system-metrics-publisher.enabled
2018-12-09 22:01:34,306 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1544410258618_0005
2018-12-09 22:01:34,307 INFO mapreduce.JobSubmitter: Executing with tokens: []
2018-12-09 22:01:34,595 INFO conf.Configuration: resource-types.xml not found
2018-12-09 22:01:34,595 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.
2018-12-09 22:01:34,715 INFO impl.YarnClientImpl: Submitted application application_1544410258618_0005
2018-12-09 22:01:34,772 INFO mapreduce.Job: The url to track the job: http://localhost:8088/proxy/application_1544410258618_0005/
2018-12-09 22:01:34,773 INFO mapreduce.Job: Running job: job_1544410258618_0005
2018-12-09 22:01:43,122 INFO mapreduce.Job: Job job_1544410258618_0005 running in uber mode : false
2018-12-09 22:01:43,123 INFO mapreduce.Job: map 0% reduce 0%
2018-12-09 22:01:57,528 INFO mapreduce.Job: Task Id : attempt_1544410258618_0005_m_000000_0, Status : FAILED
[2018-12-09 22:01:54.252]Container [pid=10353,containerID=container_1544410258618_0005_01_000002] is running 523131392B beyond the 'VIRTUAL' memory limit. Current usage: 257.3 MB of 1 GB physical memory used; 2.6 GB of 2.1 GB virtual memory used. Killing container.
Dump of the process-tree for container_1544410258618_0005_01_000002 :
|- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
|- 10353 10352 10353 10353 (bash) 0 0 115896320 173 /bin/bash -c /usr/local/java/jdk1.8.0_191/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx820m -Djava.io.tmpdir=/usr/local/hadoop/tmp/nm-local-dir/usercache/root/appcache/application_1544410258618_0005/container_1544410258618_0005_01_000002/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/usr/local/hadoop/hadoop-3.1.1/logs/userlogs/application_1544410258618_0005/container_1544410258618_0005_01_000002 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog org.apache.hadoop.mapred.YarnChild 127.0.0.1 38435 attempt_1544410258618_0005_m_000000_0 2 1>/usr/local/hadoop/hadoop-3.1.1/logs/userlogs/application_1544410258618_0005/container_1544410258618_0005_01_000002/stdout 2>/usr/local/hadoop/hadoop-3.1.1/logs/userlogs/application_1544410258618_0005/container_1544410258618_0005_01_000002/stderr
|- 10364 10353 10353 10353 (java) 624 174 2662092800 65693 /usr/local/java/jdk1.8.0_191/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx820m -Djava.io.tmpdir=/usr/local/hadoop/tmp/nm-local-dir/usercache/root/appcache/application_1544410258618_0005/container_1544410258618_0005_01_000002/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/usr/local/hadoop/hadoop-3.1.1/logs/userlogs/application_1544410258618_0005/container_1544410258618_0005_01_000002 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog org.apache.hadoop.mapred.YarnChild 127.0.0.1 38435 attempt_1544410258618_0005_m_000000_0 2

[2018-12-09 22:01:55.961]Container killed on request. Exit code is 143
[2018-12-09 22:01:55.961]Container exited with a non-zero exit code 143.

2018-12-09 22:02:07,613 INFO mapreduce.Job: Task Id : attempt_1544410258618_0005_m_000000_1, Status : FAILED
[2018-12-09 22:02:06.339]Container [pid=10421,containerID=container_1544410258618_0005_01_000003] is running 521972224B beyond the 'VIRTUAL' memory limit. Current usage: 136.2 MB of 1 GB physical memory used; 2.6 GB of 2.1 GB virtual memory used. Killing container.
Dump of the process-tree for container_1544410258618_0005_01_000003 :
|- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
|- 10430 10421 10421 10421 (java) 346 100 2660933632 34552 /usr/local/java/jdk1.8.0_191/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx820m -Djava.io.tmpdir=/usr/local/hadoop/tmp/nm-local-dir/usercache/root/appcache/application_1544410258618_0005/container_1544410258618_0005_01_000003/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/usr/local/hadoop/hadoop-3.1.1/logs/userlogs/application_1544410258618_0005/container_1544410258618_0005_01_000003 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog org.apache.hadoop.mapred.YarnChild 127.0.0.1 38435 attempt_1544410258618_0005_m_000000_1 3
|- 10421 10419 10421 10421 (bash) 0 0 115896320 306 /bin/bash -c /usr/local/java/jdk1.8.0_191/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx820m -Djava.io.tmpdir=/usr/local/hadoop/tmp/nm-local-dir/usercache/root/appcache/application_1544410258618_0005/container_1544410258618_0005_01_000003/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/usr/local/hadoop/hadoop-3.1.1/logs/userlogs/application_1544410258618_0005/container_1544410258618_0005_01_000003 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog org.apache.hadoop.mapred.YarnChild 127.0.0.1 38435 attempt_1544410258618_0005_m_000000_1 3 1>/usr/local/hadoop/hadoop-3.1.1/logs/userlogs/application_1544410258618_0005/container_1544410258618_0005_01_000003/stdout 2>/usr/local/hadoop/hadoop-3.1.1/logs/userlogs/application_1544410258618_0005/container_1544410258618_0005_01_000003/stderr

[2018-12-09 22:02:06.409]Container killed on request. Exit code is 143
[2018-12-09 22:02:06.420]Container exited with a non-zero exit code 143.

2018-12-09 22:02:19,533 INFO mapreduce.Job: map 100% reduce 0%
2018-12-09 22:02:19,534 INFO mapreduce.Job: Task Id : attempt_1544410258618_0005_m_000000_2, Status : FAILED
[2018-12-09 22:02:16.854]Container [pid=10477,containerID=container_1544410258618_0005_01_000004] is running 551004672B beyond the 'VIRTUAL' memory limit. Current usage: 272.7 MB of 1 GB physical memory used; 2.6 GB of 2.1 GB virtual memory used. Killing container.
Dump of the process-tree for container_1544410258618_0005_01_000004 :
|- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
|- 10486 10477 10477 10477 (java) 412 63 2689966080 69764 /usr/local/java/jdk1.8.0_191/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx820m -Djava.io.tmpdir=/usr/local/hadoop/tmp/nm-local-dir/usercache/root/appcache/application_1544410258618_0005/container_1544410258618_0005_01_000004/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/usr/local/hadoop/hadoop-3.1.1/logs/userlogs/application_1544410258618_0005/container_1544410258618_0005_01_000004 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog org.apache.hadoop.mapred.YarnChild 127.0.0.1 38435 attempt_1544410258618_0005_m_000000_2 4
|- 10477 10475 10477 10477 (bash) 0 0 115896320 50 /bin/bash -c /usr/local/java/jdk1.8.0_191/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx820m -Djava.io.tmpdir=/usr/local/hadoop/tmp/nm-local-dir/usercache/root/appcache/application_1544410258618_0005/container_1544410258618_0005_01_000004/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/usr/local/hadoop/hadoop-3.1.1/logs/userlogs/application_1544410258618_0005/container_1544410258618_0005_01_000004 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog org.apache.hadoop.mapred.YarnChild 127.0.0.1 38435 attempt_1544410258618_0005_m_000000_2 4 1>/usr/local/hadoop/hadoop-3.1.1/logs/userlogs/application_1544410258618_0005/container_1544410258618_0005_01_000004/stdout 2>/usr/local/hadoop/hadoop-3.1.1/logs/userlogs/application_1544410258618_0005/container_1544410258618_0005_01_000004/stderr

[2018-12-09 22:02:18.409]Container killed on request. Exit code is 143
[2018-12-09 22:02:18.410]Container exited with a non-zero exit code 143.

2018-12-09 22:02:20,541 INFO mapreduce.Job: map 0% reduce 0%
2018-12-09 22:02:33,619 INFO mapreduce.Job: map 100% reduce 100%
2018-12-09 22:02:35,639 INFO mapreduce.Job: Job job_1544410258618_0005 failed with state FAILED due to: Task failed task_1544410258618_0005_m_000000
Job failed as tasks failed. failedMaps:1 failedReduces:0 killedMaps:0 killedReduces: 0

2018-12-09 22:02:35,794 INFO mapreduce.Job: Counters: 13
Job Counters
Failed map tasks=4
Killed reduce tasks=1
Launched map tasks=4
Other local map tasks=3
Data-local map tasks=1
Total time spent by all maps in occupied slots (ms)=40350
Total time spent by all reduces in occupied slots (ms)=0
Total time spent by all map tasks (ms)=40350
Total vcore-milliseconds taken by all map tasks=40350
Total megabyte-milliseconds taken by all map tasks=41318400
Map-Reduce Framework
CPU time spent (ms)=0
Physical memory (bytes) snapshot=0
Virtual memory (bytes) snapshot=0

小二君i的主页 小二君i | 初学一级 | 园豆:157
提问于:2018-12-10 11:04
< >
分享
最佳答案
0

略看像是内存不够导致的。

收获园豆:50
会长 | 专家六级 |园豆:12401 | 2018-12-10 11:23
清除回答草稿
   您需要登录以后才能回答,未注册用户请先注册