spark-submit 怎么设置才能避免这些警告
0
悬赏园豆：30
[已关闭问题]
关闭于 2017-11-02 09:10
admin@spark1:~/work$ spark-submit import_other.py 17/10/27 17:57:28 WARN HiveConf: HiveConf of name hive.metastore.hbase.aggregate.stats.false.positive.probability does not exist 17/10/27 17:57:28 WARN HiveConf: HiveConf of name hive.llap.io.orc.time.counters does not exist 17/10/27 17:57:28 WARN HiveConf: HiveConf of name hive.orc.splits.ms.footer.cache.ppd.enabled does not exist 17/10/27 17:57:28 WARN HiveConf: HiveConf of name hive.server2.metrics.enabled does not exist 17/10/27 17:57:28 WARN HiveConf: HiveConf of name hive.llap.am.liveness.connection.timeout.ms does not exist 17/10/27 17:57:28 WARN HiveConf: HiveConf of name hive.server2.thrift.client.connect.retry.limit does not exist 17/10/27 17:57:28 WARN HiveConf: HiveConf of name hive.llap.io.allocator.direct does not exist 17/10/27 17:57:28 WARN HiveConf: HiveConf of name hive.llap.auto.enforce.stats does not exist 17/10/27 17:57:28 WARN HiveConf: HiveConf of name hive.llap.client.consistent.splits does not exist 17/10/27 17:57:28 WARN HiveConf: HiveConf of name hive.server2.tez.session.lifetime does not exist 17/10/27 17:57:28 WARN HiveConf: HiveConf of name hive.timedout.txn.reaper.start does not exist 17/10/27 17:57:28 WARN HiveConf: HiveConf of name hive.metastore.hbase.cache.ttl does not exist 17/10/27 17:57:28 WARN HiveConf: HiveConf of name hive.llap.management.acl does not exist 17/10/27 17:57:28 WARN HiveConf: HiveConf of name hive.llap.daemon.delegation.token.lifetime does not exist
每次submit都会打印一长串这种警告，请问是哪里没有配置好？
所有回答(1)
0
最初以为打印这些警告是由于没有配置这些项导致的。然后某天突然醒悟了，从另一方面来讲可能是spark的hiveConf并不接受这些配置，相当于多扔给它一些参数。于是把spark配置目录下的hive-site.xml中对应WARN配置项都删除了(因为是直接拷贝的hive配置，配置项比较多。可以先vim处理下输出日志截出配置项，然后写个python脚本解析下xml剔掉WARN项)，再次提交任务果然没有这些WARN了。
