寻求各位大佬的帮忙!!!
在进行编写spark程序时,需要从HBase中读写数据,最近HBase集群进行了kerberos的认证,我在代码中进行了HBase的kerberos认证配置之后,kerberos的相关配置文件放在resource目录下,本地(IDEA上)测试没问题。但是当提交任务到spark集群上跑时,一直报下面的问题,寻求各位大佬的帮忙。
这是出现的异常:
这是提交任务的命令:
bin/spark-submit \
--master yarn \
--deploy-mode client \
--keytab ../conf/hadoop.keytab \
--principal hadoop/hdfs01@BDSOFT.COM \
--files ../conf/hbase.keytab,../conf/krb5.conf \
envelope-0.8.0.jar 83f8876b-3b94-4f96-addc-eaca5339f50a_20210310164014016.conf
这一块是关于获取配置文件,以及进行HBase的kerberos认证的代码
public static Configuration kerberosLogin(Configuration hbConf){
LOG.info("开启kerberso 认证 ...... ");
String krb5ConfPath = KerberosCheckUtil.getKrb5Conf();
String keytabPath = KerberosCheckUtil.getKeyTabFile();
//kerbose
hbConf.set("hadoop.security.authentication", "Kerberos");
hbConf.set("hbase.security.authentication", "kerberos");
hbConf.set("hbase.master.kerberos.principal", KerberosCheckUtil.principal);
hbConf.set("hbase.regionserver.kerberos.principal", KerberosCheckUtil.principal);
System.setProperty("javax.security.auth.useSubjectCredOnly", "false");
System.setProperty("sun.security.krb5.conf", krb5ConfPath);
try{
UserGroupInformation.setConfiguration(hbConf);
if (UserGroupInformation.isLoginKeytabBased() && UserGroupInformation.getLoginUser().getUserName().equals(KerberosCheckUtil.principal)) {
LOG.info("hbase:" + hbConf.get("hbase.master")+ ",user [{}] is login already!",KerberosCheckUtil.principal);
}else {
UserGroupInformation.loginUserFromKeytab(KerberosCheckUtil.principal, keytabPath);
LOG.info("hbase:" + hbConf.get("hbase.master") + ",user [{}] login successed!",KerberosCheckUtil.principal);
}
LOG.info("Hbase Kerberos 认证成功 ...... ");
return hbConf;
}catch (Exception e){
e.printStackTrace();
LOG.error("Hbase kerberos 登录报错," + KerberosCheckUtil.getKeyTabFile());
System.exit(1);
return hbConf;
}
}
public class KerberosCheckUtil {
private static final Logger LOG = LoggerFactory.getLogger(KerberosCheckUtil.class);
//主体
public static String principal="hadoop/hdfs01@BDSOFT.COM";
//秘钥文件
public static String keyTabFileName="hadoop.keytab";
//默认配置文件
public static String krb5FileName= "krb5.conf";
public static String getKeyTabFile() {
String keyTabPath = SparkFiles.get(keyTabFileName);
LOG.info("keyTabPath ========= " + keyTabPath);
return keyTabPath;
}
public static String getKrb5Conf() {
String krb5Path = SparkFiles.get(krb5FileName);
LOG.info("krb5Path ========= " + krb5Path);
return krb5Path;
}
}
解决了吗,遇到点问题
解决了吗,我这边ok了,详情可私聊
– 木子民 2年前