首页 新闻 会员 周边

spark-submit提交带有kerberos访问认证的HBase的任务时,读取不到krb5.conf文件

0
悬赏园豆:5 [待解决问题]

寻求各位大佬的帮忙!!!

在进行编写spark程序时,需要从HBase中读写数据,最近HBase集群进行了kerberos的认证,我在代码中进行了HBase的kerberos认证配置之后,kerberos的相关配置文件放在resource目录下,本地(IDEA上)测试没问题。但是当提交任务到spark集群上跑时,一直报下面的问题,寻求各位大佬的帮忙。

这是出现的异常:

这是提交任务的命令:

bin/spark-submit \
--master yarn \
--deploy-mode client \
--keytab ../conf/hadoop.keytab \
--principal hadoop/hdfs01@BDSOFT.COM \
--files ../conf/hbase.keytab,../conf/krb5.conf \
envelope-0.8.0.jar  83f8876b-3b94-4f96-addc-eaca5339f50a_20210310164014016.conf

这一块是关于获取配置文件,以及进行HBase的kerberos认证的代码

public static Configuration kerberosLogin(Configuration hbConf){

    LOG.info("开启kerberso 认证 ...... ");

    String krb5ConfPath = KerberosCheckUtil.getKrb5Conf();
    String keytabPath = KerberosCheckUtil.getKeyTabFile();
    //kerbose
    hbConf.set("hadoop.security.authentication", "Kerberos");
    hbConf.set("hbase.security.authentication", "kerberos");
    hbConf.set("hbase.master.kerberos.principal", KerberosCheckUtil.principal);
    hbConf.set("hbase.regionserver.kerberos.principal", KerberosCheckUtil.principal);
    System.setProperty("javax.security.auth.useSubjectCredOnly", "false");
    System.setProperty("sun.security.krb5.conf", krb5ConfPath);

    try{
      UserGroupInformation.setConfiguration(hbConf);

      if (UserGroupInformation.isLoginKeytabBased() && UserGroupInformation.getLoginUser().getUserName().equals(KerberosCheckUtil.principal)) {
        LOG.info("hbase:" + hbConf.get("hbase.master")+ ",user [{}] is login already!",KerberosCheckUtil.principal);
      }else {
        UserGroupInformation.loginUserFromKeytab(KerberosCheckUtil.principal, keytabPath);
        LOG.info("hbase:" + hbConf.get("hbase.master") + ",user [{}] login successed!",KerberosCheckUtil.principal);
      }
      LOG.info("Hbase Kerberos 认证成功 ...... ");
      return hbConf;
    }catch (Exception e){
      e.printStackTrace();
      LOG.error("Hbase kerberos 登录报错," + KerberosCheckUtil.getKeyTabFile());
      System.exit(1);
      return hbConf;
    }
  }
public class KerberosCheckUtil {

    private static final Logger LOG = LoggerFactory.getLogger(KerberosCheckUtil.class);

    //主体
    public static String principal="hadoop/hdfs01@BDSOFT.COM";
    //秘钥文件
    public static String keyTabFileName="hadoop.keytab";
    //默认配置文件
    public static String krb5FileName= "krb5.conf";

    public static String getKeyTabFile() {
        String keyTabPath = SparkFiles.get(keyTabFileName);
        LOG.info("keyTabPath ========= " +  keyTabPath);
        return  keyTabPath;
    }

    public static String getKrb5Conf() {
        String krb5Path = SparkFiles.get(krb5FileName);
        LOG.info("krb5Path ========= " + krb5Path);
        return  krb5Path;
    }
}
Lucier_k的主页 Lucier_k | 初学一级 | 园豆:197
提问于:2021-03-16 17:10

解决了吗,我这边ok了,详情可私聊

木子民 2年前
< >
分享
所有回答(1)
0

解决了吗,遇到点问题

云淡风轻Go | 园豆:202 (菜鸟二级) | 2022-01-07 14:39
清除回答草稿
   您需要登录以后才能回答,未注册用户请先注册