进程:
[itsels@hadoop spark--bin-2.5.0-cdh5.3.6]$ jps
2390 HistoryServer
2521 ResourceManager
2578 NodeManager
2192 NameNode
2247 DataNode
4736 Jps
运行案例:
[itsels@hadoop spark--bin-2.5.0-cdh5.3.6]$ bin/spark-submit \
> --master yarn-cluster \
> jars/SparkApp.jar
日志信息
17/04/20 23:57:06 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
17/04/20 23:57:06 INFO client.RMProxy: Connecting to ResourceManager at hadoop.senior04/192.168.204.103:8032
17/04/20 23:57:07 INFO yarn.Client: Requesting a new application from cluster with 1 NodeManagers
17/04/20 23:57:07 INFO yarn.Client: Verifying our application has not requested more than the maximum memory capability of the cluster (8192 MB per container)
17/04/20 23:57:07 INFO yarn.Client: Will allocate AM container, with 896 MB memory including 384 MB overhead
17/04/20 23:57:07 INFO yarn.Client: Setting up container launch context for our AM
17/04/20 23:57:07 INFO yarn.Client: Preparing resources for our AM container
17/04/20 23:57:08 INFO yarn.Client: Uploading resource file:/opt/cdh-5.3.6/spark--bin-2.5.0-cdh5.3.6/lib/spark-assembly-1.3.1-hadoop2.5.0-cdh5.3.6.jar -> hdfs://hadoop.senior04:8020/user/ibeifeng/.sparkStaging/application_1492700824445_0003/spark-assembly-1.3.1-hadoop2.5.0-cdh5.3.6.jar
17/04/20 23:57:22 INFO yarn.Client: Uploading resource file:/opt/cdh-5.3.6/spark--bin-2.5.0-cdh5.3.6/jars/SparkApp.jar -> hdfs://hadoop.senior04:8020/user/ibeifeng/.sparkStaging/application_1492700824445_0003/SparkApp.jar
17/04/20 23:57:22 INFO yarn.Client: Setting up the launch environment for our AM container
17/04/20 23:57:22 INFO spark.SecurityManager: Changing view acls to: ibeifeng
17/04/20 23:57:22 INFO spark.SecurityManager: Changing modify acls to: ibeifeng
17/04/20 23:57:22 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(ibeifeng); users with modify permissions: Set(ibeifeng)
17/04/20 23:57:22 INFO yarn.Client: Submitting application 3 to ResourceManager
17/04/20 23:57:23 INFO impl.YarnClientImpl: Submitted application application_1492700824445_0003
17/04/20 23:57:24 INFO yarn.Client: Application report for application_1492700824445_0003 (state: ACCEPTED)
17/04/20 23:57:24 INFO yarn.Client:
client token: N/A
diagnostics: N/A
ApplicationMaster host: N/A
ApplicationMaster RPC port: -1
queue: root.ibeifeng
start time: 1492703842981
final status: UNDEFINED
tracking URL: http://hadoop.senior04:8088/proxy/application_1492700824445_0003/
user: ibeifeng
17/04/20 23:57:25 INFO yarn.Client: Application report for application_1492700824445_0003 (state: ACCEPTED)
17/04/20 23:57:27 INFO yarn.Client: Application report for application_1492700824445_0003 (state: ACCEPTED)
17/04/20 23:57:28 INFO yarn.Client: Application report for application_1492700824445_0003 (state: ACCEPTED)
17/04/20 23:57:30 INFO yarn.Client: Application report for application_1492700824445_0003 (state: ACCEPTED)
17/04/20 23:57:31 INFO yarn.Client: Application report for application_1492700824445_0003 (state: ACCEPTED)
17/04/20 23:57:33 INFO yarn.Client: Application report for application_1492700824445_0003 (state: ACCEPTED)
17/04/20 23:57:34 INFO yarn.Client: Application report for application_1492700824445_0003 (state: ACCEPTED)
17/04/20 23:57:35 INFO yarn.Client: Application report for application_1492700824445_0003 (state: ACCEPTED)
17/04/20 23:57:36 INFO yarn.Client: Application report for application_1492700824445_0003 (state: ACCEPTED)
17/04/20 23:57:37 INFO yarn.Client: Application report for application_1492700824445_0003 (state: ACCEPTED)
17/04/20 23:57:38 INFO yarn.Client: Application report for application_1492700824445_0003 (state: ACCEPTED)
17/04/20 23:57:39 INFO yarn.Client: Application report for application_1492700824445_0003 (state: ACCEPTED)
17/04/20 23:57:40 INFO yarn.Client: Application report for application_1492700824445_0003 (state: ACCEPTED)
17/04/20 23:57:41 INFO yarn.Client: Application report for application_1492700824445_0003 (state: ACCEPTED)
17/04/20 23:57:42 INFO yarn.Client: Application report for application_1492700824445_0003 (state: ACCEPTED)
17/04/20 23:57:43 INFO yarn.Client: Application report for application_1492700824445_0003 (state: ACCEPTED)
17/04/20 23:57:44 INFO yarn.Client: Application report for application_1492700824445_0003 (state: ACCEPTED)
17/04/20 23:57:45 INFO yarn.Client: Application report for application_1492700824445_0003 (state: ACCEPTED)
17/04/20 23:57:46 INFO yarn.Client: Application report for application_1492700824445_0003 (state: ACCEPTED)
17/04/20 23:57:47 INFO yarn.Client: Application report for application_1492700824445_0003 (state: ACCEPTED)
17/04/20 23:57:48 INFO yarn.Client: Application report for application_1492700824445_0003 (state: ACCEPTED)
17/04/20 23:57:49 INFO yarn.Client: Application report for application_1492700824445_0003 (state: ACCEPTED)
17/04/20 23:57:50 INFO yarn.Client: Application report for application_1492700824445_0003 (state: ACCEPTED)
17/04/20 23:57:51 INFO yarn.Client: Application report for application_1492700824445_0003 (state: ACCEPTED)
17/04/20 23:57:52 INFO yarn.Client: Application report for application_1492700824445_0003 (state: ACCEPTED)
17/04/20 23:57:53 INFO yarn.Client: Application report for application_1492700824445_0003 (state: ACCEPTED)
17/04/20 23:57:54 INFO yarn.Client: Application report for application_1492700824445_0003 (state: ACCEPTED)
17/04/20 23:57:55 INFO yarn.Client: Application report for application_1492700824445_0003 (state: ACCEPTED)
17/04/20 23:57:56 INFO yarn.Client: Application report for application_1492700824445_0003 (state: ACCEPTED)
17/04/20 23:57:57 INFO yarn.Client: Application report for application_1492700824445_0003 (state: ACCEPTED)
17/04/20 23:57:58 INFO yarn.Client: Application report for application_1492700824445_0003 (state: ACCEPTED)
17/04/20 23:57:59 INFO yarn.Client: Application report for application_1492700824445_0003 (state: ACCEPTED)
17/04/20 23:58:00 INFO yarn.Client: Application report for application_1492700824445_0003 (state: ACCEPTED)
17/04/20 23:58:01 INFO yarn.Client: Application report for application_1492700824445_0003 (state: ACCEPTED)
17/04/20 23:58:02 INFO yarn.Client: Application report for application_1492700824445_0003 (state: ACCEPTED)
17/04/20 23:58:03 INFO yarn.Client: Application report for application_1492700824445_0003 (state: ACCEPTED)
17/04/20 23:58:04 INFO yarn.Client: Application report for application_1492700824445_0003 (state: ACCEPTED)
17/04/20 23:58:06 INFO yarn.Client: Application report for application_1492700824445_0003 (state: ACCEPTED)
17/04/20 23:58:07 INFO yarn.Client: Application report for application_1492700824445_0003 (state: ACCEPTED)
17/04/20 23:58:08 INFO yarn.Client: Application report for application_1492700824445_0003 (state: ACCEPTED)
17/04/20 23:58:09 INFO yarn.Client: Application report for application_1492700824445_0003 (state: ACCEPTED)
17/04/20 23:58:10 INFO yarn.Client: Application report for application_1492700824445_0003 (state: ACCEPTED)
17/04/20 23:58:11 INFO yarn.Client: Application report for application_1492700824445_0003 (state: ACCEPTED)
17/04/20 23:58:12 INFO yarn.Client: Application report for application_1492700824445_0003 (state: ACCEPTED)
17/04/20 23:58:13 INFO yarn.Client: Application report for application_1492700824445_0003 (state: ACCEPTED)
17/04/20 23:58:14 INFO yarn.Client: Application report for application_1492700824445_0003 (state: ACCEPTED)
17/04/20 23:58:15 INFO yarn.Client: Application report for application_1492700824445_0003 (state: ACCEPTED)
17/04/20 23:58:16 INFO yarn.Client: Application report for application_1492700824445_0003 (state: ACCEPTED)
17/04/20 23:58:17 INFO yarn.Client: Application report for application_1492700824445_0003 (state: ACCEPTED)
17/04/20 23:58:18 INFO yarn.Client: Application report for application_1492700824445_0003 (state: ACCEPTED)
17/04/20 23:58:19 INFO yarn.Client: Application report for application_1492700824445_0003 (state: ACCEPTED)
17/04/20 23:58:20 INFO yarn.Client: Application report for application_1492700824445_0003 (state: ACCEPTED)
17/04/20 23:58:21 INFO yarn.Client: Application report for application_1492700824445_0003 (state: ACCEPTED)
17/04/20 23:58:22 INFO yarn.Client: Application report for application_1492700824445_0003 (state: ACCEPTED)
17/04/20 23:58:23 INFO yarn.Client: Application report for application_1492700824445_0003 (state: ACCEPTED)
17/04/20 23:58:24 INFO yarn.Client: Application report for application_1492700824445_0003 (state: ACCEPTED)
17/04/20 23:58:25 INFO yarn.Client: Application report for application_1492700824445_0003 (state: ACCEPTED)
17/04/20 23:58:26 INFO yarn.Client: Application report for application_1492700824445_0003 (state: ACCEPTED)
17/04/20 23:58:27 INFO yarn.Client: Application report for application_1492700824445_0003 (state: ACCEPTED)
17/04/20 23:58:28 INFO yarn.Client: Application report for application_1492700824445_0003 (state: ACCEPTED)
17/04/20 23:58:29 INFO yarn.Client: Application report for application_1492700824445_0003 (state: ACCEPTED)
17/04/20 23:58:30 INFO yarn.Client: Application report for application_1492700824445_0003 (state: ACCEPTED)
17/04/20 23:58:32 INFO yarn.Client: Application report for application_1492700824445_0003 (state: ACCEPTED)
17/04/20 23:58:44 INFO yarn.Client: Application report for application_1492700824445_0003 (state: FAILED)
17/04/20 23:58:44 INFO yarn.Client:
client token: N/A
diagnostics: Application application_1492700824445_0003 failed 2 times due to AM Container for appattempt_1492700824445_0003_000002 exited with exitCode: 15 due to: Exception from container-launch.
Container id: container_1492700824445_0003_02_000001
Exit code: 15
Stack trace: ExitCodeException exitCode=15:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
at org.apache.hadoop.util.Shell.run(Shell.java:455)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:197)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:299)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Container exited with a non-zero exit code 15
.Failing this attempt.. Failing the application.
ApplicationMaster host: N/A
ApplicationMaster RPC port: -1
queue: root.ibeifeng
start time: 1492703842981
final status: FAILED
tracking URL: http://hadoop.senior04:8088/cluster/app/application_1492700824445_0003
user: ibeifeng
Exception in thread "main" org.apache.spark.SparkException: Application finished with failed status
at org.apache.spark.deploy.yarn.Client.run(Client.scala:622)
at org.apache.spark.deploy.yarn.Client$.main(Client.scala:647)
at org.apache.spark.deploy.yarn.Client.main(Client.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:569)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:166)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:189)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:110)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
1. 下载源码
2. 在集群的某台机器上编译
3. 替换之前的$HADOOP_HOME/lib/native为新编译的native