首页 新闻 会员 周边 捐助

spark-sql启动的时候需要制定master吗?

0
悬赏园豆:50 [已关闭问题] 关闭于 2017-03-14 22:36

spark-sql 启动时制定master

spark-sql   --master  spark://mater:7077 指定maser 相当于standalone模式

如果直接spark-sql 启动,spark-env.sh中配置了spark master的ip,此时不指定master也相当于standalone模式

spark-sql  --master yarn-client 是以yarn客户端的模式去跑sql

但是如何让sql以yarn-cluster模式运行呢?

spark-sql  --master yarn-cluster 

spark-sql  --master yarn --deploy-mode cluster

 这两种方式启动是一样的,但是为什么报错呢?

17/01/04 13:03:59 INFO yarn.Client:
client token: N/A
diagnostics: Application application_1483257147443_0300 failed 2 times due to AM Container for appattempt_1483257147443_0300_000002 exited with exitCode: 1 due to: Exception from container-launch.
Container id: container_1483257147443_0300_02_000001
Exit code: 1
Stack trace: ExitCodeException exitCode=1:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
at org.apache.hadoop.util.Shell.run(Shell.java:455)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:197)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:299)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)


Container exited with a non-zero exit code 1
.Failing this attempt.. Failing the application.
ApplicationMaster host: N/A
ApplicationMaster RPC port: -1
queue: root.hadoop
start time: 1483506229770
final status: FAILED
tracking URL: http://cdhm1:8088/cluster/app/application_1483257147443_0300
user: hadoop
Exception in thread "main" org.apache.spark.SparkException: Application finished with failed status
at org.apache.spark.deploy.yarn.ClientBase$class.run(ClientBase.scala:509)
at org.apache.spark.deploy.yarn.Client.run(Client.scala:35)
at org.apache.spark.deploy.yarn.Client$.main(Client.scala:139)
at org.apache.spark.deploy.yarn.Client.main(Client.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:358)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:75)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

求大神指导

 

记忆残留的主页 记忆残留 | 初学一级 | 园豆:168
提问于:2017-01-04 13:05
< >
分享
清除回答草稿
   您需要登录以后才能回答,未注册用户请先注册