首页 新闻 会员 周边 捐助

spark 运行在yarn上

-1
悬赏园豆:50 [待解决问题]

进程:

  [itsels@hadoop spark--bin-2.5.0-cdh5.3.6]$ jps
      2390 HistoryServer
      2521 ResourceManager
      2578 NodeManager
      2192 NameNode
      2247 DataNode
      4736 Jps

运行案例:

[itsels@hadoop spark--bin-2.5.0-cdh5.3.6]$ bin/spark-submit \
> --master yarn-cluster \
> jars/SparkApp.jar

 

日志信息
17/04/20 23:57:06 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
17/04/20 23:57:06 INFO client.RMProxy: Connecting to ResourceManager at hadoop.senior04/192.168.204.103:8032
17/04/20 23:57:07 INFO yarn.Client: Requesting a new application from cluster with 1 NodeManagers
17/04/20 23:57:07 INFO yarn.Client: Verifying our application has not requested more than the maximum memory capability of the cluster (8192 MB per container)
17/04/20 23:57:07 INFO yarn.Client: Will allocate AM container, with 896 MB memory including 384 MB overhead
17/04/20 23:57:07 INFO yarn.Client: Setting up container launch context for our AM
17/04/20 23:57:07 INFO yarn.Client: Preparing resources for our AM container
17/04/20 23:57:08 INFO yarn.Client: Uploading resource file:/opt/cdh-5.3.6/spark--bin-2.5.0-cdh5.3.6/lib/spark-assembly-1.3.1-hadoop2.5.0-cdh5.3.6.jar -> hdfs://hadoop.senior04:8020/user/ibeifeng/.sparkStaging/application_1492700824445_0003/spark-assembly-1.3.1-hadoop2.5.0-cdh5.3.6.jar
17/04/20 23:57:22 INFO yarn.Client: Uploading resource file:/opt/cdh-5.3.6/spark--bin-2.5.0-cdh5.3.6/jars/SparkApp.jar -> hdfs://hadoop.senior04:8020/user/ibeifeng/.sparkStaging/application_1492700824445_0003/SparkApp.jar
17/04/20 23:57:22 INFO yarn.Client: Setting up the launch environment for our AM container
17/04/20 23:57:22 INFO spark.SecurityManager: Changing view acls to: ibeifeng
17/04/20 23:57:22 INFO spark.SecurityManager: Changing modify acls to: ibeifeng
17/04/20 23:57:22 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(ibeifeng); users with modify permissions: Set(ibeifeng)
17/04/20 23:57:22 INFO yarn.Client: Submitting application 3 to ResourceManager
17/04/20 23:57:23 INFO impl.YarnClientImpl: Submitted application application_1492700824445_0003
17/04/20 23:57:24 INFO yarn.Client: Application report for application_1492700824445_0003 (state: ACCEPTED)
17/04/20 23:57:24 INFO yarn.Client:
         client token: N/A
         diagnostics: N/A
         ApplicationMaster host: N/A
         ApplicationMaster RPC port: -1
         queue: root.ibeifeng
         start time: 1492703842981
         final status: UNDEFINED
         tracking URL: http://hadoop.senior04:8088/proxy/application_1492700824445_0003/
         user: ibeifeng
17/04/20 23:57:25 INFO yarn.Client: Application report for application_1492700824445_0003 (state: ACCEPTED)
17/04/20 23:57:27 INFO yarn.Client: Application report for application_1492700824445_0003 (state: ACCEPTED)
17/04/20 23:57:28 INFO yarn.Client: Application report for application_1492700824445_0003 (state: ACCEPTED)
17/04/20 23:57:30 INFO yarn.Client: Application report for application_1492700824445_0003 (state: ACCEPTED)
17/04/20 23:57:31 INFO yarn.Client: Application report for application_1492700824445_0003 (state: ACCEPTED)
17/04/20 23:57:33 INFO yarn.Client: Application report for application_1492700824445_0003 (state: ACCEPTED)
17/04/20 23:57:34 INFO yarn.Client: Application report for application_1492700824445_0003 (state: ACCEPTED)
17/04/20 23:57:35 INFO yarn.Client: Application report for application_1492700824445_0003 (state: ACCEPTED)
17/04/20 23:57:36 INFO yarn.Client: Application report for application_1492700824445_0003 (state: ACCEPTED)
17/04/20 23:57:37 INFO yarn.Client: Application report for application_1492700824445_0003 (state: ACCEPTED)
17/04/20 23:57:38 INFO yarn.Client: Application report for application_1492700824445_0003 (state: ACCEPTED)
17/04/20 23:57:39 INFO yarn.Client: Application report for application_1492700824445_0003 (state: ACCEPTED)
17/04/20 23:57:40 INFO yarn.Client: Application report for application_1492700824445_0003 (state: ACCEPTED)
17/04/20 23:57:41 INFO yarn.Client: Application report for application_1492700824445_0003 (state: ACCEPTED)
17/04/20 23:57:42 INFO yarn.Client: Application report for application_1492700824445_0003 (state: ACCEPTED)
17/04/20 23:57:43 INFO yarn.Client: Application report for application_1492700824445_0003 (state: ACCEPTED)
17/04/20 23:57:44 INFO yarn.Client: Application report for application_1492700824445_0003 (state: ACCEPTED)
17/04/20 23:57:45 INFO yarn.Client: Application report for application_1492700824445_0003 (state: ACCEPTED)
17/04/20 23:57:46 INFO yarn.Client: Application report for application_1492700824445_0003 (state: ACCEPTED)
17/04/20 23:57:47 INFO yarn.Client: Application report for application_1492700824445_0003 (state: ACCEPTED)
17/04/20 23:57:48 INFO yarn.Client: Application report for application_1492700824445_0003 (state: ACCEPTED)
17/04/20 23:57:49 INFO yarn.Client: Application report for application_1492700824445_0003 (state: ACCEPTED)
17/04/20 23:57:50 INFO yarn.Client: Application report for application_1492700824445_0003 (state: ACCEPTED)
17/04/20 23:57:51 INFO yarn.Client: Application report for application_1492700824445_0003 (state: ACCEPTED)
17/04/20 23:57:52 INFO yarn.Client: Application report for application_1492700824445_0003 (state: ACCEPTED)
17/04/20 23:57:53 INFO yarn.Client: Application report for application_1492700824445_0003 (state: ACCEPTED)
17/04/20 23:57:54 INFO yarn.Client: Application report for application_1492700824445_0003 (state: ACCEPTED)
17/04/20 23:57:55 INFO yarn.Client: Application report for application_1492700824445_0003 (state: ACCEPTED)
17/04/20 23:57:56 INFO yarn.Client: Application report for application_1492700824445_0003 (state: ACCEPTED)
17/04/20 23:57:57 INFO yarn.Client: Application report for application_1492700824445_0003 (state: ACCEPTED)
17/04/20 23:57:58 INFO yarn.Client: Application report for application_1492700824445_0003 (state: ACCEPTED)
17/04/20 23:57:59 INFO yarn.Client: Application report for application_1492700824445_0003 (state: ACCEPTED)
17/04/20 23:58:00 INFO yarn.Client: Application report for application_1492700824445_0003 (state: ACCEPTED)
17/04/20 23:58:01 INFO yarn.Client: Application report for application_1492700824445_0003 (state: ACCEPTED)
17/04/20 23:58:02 INFO yarn.Client: Application report for application_1492700824445_0003 (state: ACCEPTED)
17/04/20 23:58:03 INFO yarn.Client: Application report for application_1492700824445_0003 (state: ACCEPTED)
17/04/20 23:58:04 INFO yarn.Client: Application report for application_1492700824445_0003 (state: ACCEPTED)
17/04/20 23:58:06 INFO yarn.Client: Application report for application_1492700824445_0003 (state: ACCEPTED)
17/04/20 23:58:07 INFO yarn.Client: Application report for application_1492700824445_0003 (state: ACCEPTED)
17/04/20 23:58:08 INFO yarn.Client: Application report for application_1492700824445_0003 (state: ACCEPTED)
17/04/20 23:58:09 INFO yarn.Client: Application report for application_1492700824445_0003 (state: ACCEPTED)
17/04/20 23:58:10 INFO yarn.Client: Application report for application_1492700824445_0003 (state: ACCEPTED)
17/04/20 23:58:11 INFO yarn.Client: Application report for application_1492700824445_0003 (state: ACCEPTED)
17/04/20 23:58:12 INFO yarn.Client: Application report for application_1492700824445_0003 (state: ACCEPTED)
17/04/20 23:58:13 INFO yarn.Client: Application report for application_1492700824445_0003 (state: ACCEPTED)
17/04/20 23:58:14 INFO yarn.Client: Application report for application_1492700824445_0003 (state: ACCEPTED)
17/04/20 23:58:15 INFO yarn.Client: Application report for application_1492700824445_0003 (state: ACCEPTED)
17/04/20 23:58:16 INFO yarn.Client: Application report for application_1492700824445_0003 (state: ACCEPTED)
17/04/20 23:58:17 INFO yarn.Client: Application report for application_1492700824445_0003 (state: ACCEPTED)
17/04/20 23:58:18 INFO yarn.Client: Application report for application_1492700824445_0003 (state: ACCEPTED)
17/04/20 23:58:19 INFO yarn.Client: Application report for application_1492700824445_0003 (state: ACCEPTED)
17/04/20 23:58:20 INFO yarn.Client: Application report for application_1492700824445_0003 (state: ACCEPTED)
17/04/20 23:58:21 INFO yarn.Client: Application report for application_1492700824445_0003 (state: ACCEPTED)
17/04/20 23:58:22 INFO yarn.Client: Application report for application_1492700824445_0003 (state: ACCEPTED)
17/04/20 23:58:23 INFO yarn.Client: Application report for application_1492700824445_0003 (state: ACCEPTED)
17/04/20 23:58:24 INFO yarn.Client: Application report for application_1492700824445_0003 (state: ACCEPTED)
17/04/20 23:58:25 INFO yarn.Client: Application report for application_1492700824445_0003 (state: ACCEPTED)
17/04/20 23:58:26 INFO yarn.Client: Application report for application_1492700824445_0003 (state: ACCEPTED)
17/04/20 23:58:27 INFO yarn.Client: Application report for application_1492700824445_0003 (state: ACCEPTED)
17/04/20 23:58:28 INFO yarn.Client: Application report for application_1492700824445_0003 (state: ACCEPTED)
17/04/20 23:58:29 INFO yarn.Client: Application report for application_1492700824445_0003 (state: ACCEPTED)
17/04/20 23:58:30 INFO yarn.Client: Application report for application_1492700824445_0003 (state: ACCEPTED)
17/04/20 23:58:32 INFO yarn.Client: Application report for application_1492700824445_0003 (state: ACCEPTED)
17/04/20 23:58:44 INFO yarn.Client: Application report for application_1492700824445_0003 (state: FAILED)
17/04/20 23:58:44 INFO yarn.Client:
         client token: N/A
         diagnostics: Application application_1492700824445_0003 failed 2 times due to AM Container for appattempt_1492700824445_0003_000002 exited with  exitCode: 15 due to: Exception from container-launch.
Container id: container_1492700824445_0003_02_000001
Exit code: 15
Stack trace: ExitCodeException exitCode=15:
        at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
        at org.apache.hadoop.util.Shell.run(Shell.java:455)
        at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702)
        at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:197)
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:299)
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)


Container exited with a non-zero exit code 15
.Failing this attempt.. Failing the application.
         ApplicationMaster host: N/A
         ApplicationMaster RPC port: -1
         queue: root.ibeifeng
         start time: 1492703842981
         final status: FAILED
         tracking URL: http://hadoop.senior04:8088/cluster/app/application_1492700824445_0003
         user: ibeifeng
Exception in thread "main" org.apache.spark.SparkException: Application finished with failed status
        at org.apache.spark.deploy.yarn.Client.run(Client.scala:622)
        at org.apache.spark.deploy.yarn.Client$.main(Client.scala:647)
        at org.apache.spark.deploy.yarn.Client.main(Client.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:569)
        at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:166)
        at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:189)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:110)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

 

 

 

 

偶看旋律的主页 偶看旋律 | 初学一级 | 园豆:200
提问于:2017-04-21 11:12
< >
分享
所有回答(1)
0

1. 下载源码

2. 在集群的某台机器上编译

3. 替换之前的$HADOOP_HOME/lib/native为新编译的native

狼爷 | 园豆:1204 (小虾三级) | 2017-04-22 09:52
清除回答草稿
   您需要登录以后才能回答,未注册用户请先注册