相关文章推荐
力能扛鼎的饭卡  ·  写一个正则代码提取中文VBA - CSDN文库·  4 月前    · 
从容的炒面  ·  nacos版本升级和开启权限认证配置（1.3 ...·  6 月前    · 
独立的荔枝  ·  java如何让请求排队执行_mob64ca1 ...·  11 月前    · 
傲视众生的凉面  ·  log4net ...·  1 年前    · 
爱笑的感冒药  ·  json ...·  1 年前    · 
Code  ›  java - Getting error on sparkSession.getOrCreate() - Stack Overflow
test junit
https://stackoverflow.com/questions/57271767/getting-error-on-sparksession-getorcreate
火星上的楼梯
1 年前
Collectives™ on Stack Overflow

Find centralized, trusted content and collaborate around the technologies you use most.

Learn more about Collectives

Teams

Q&A for work

Connect and share knowledge within a single location that is structured and easy to search.

Learn more about Teams

When In test case I am trying to execute the below statement 

SparkSession spark = SparkSession.builder() 
.config("spark.sql.warehouse.dir", "target/spark-warehouse")    
.master("local[4]")     
.appName("App Name")                
.getOrCreate()

Getting the following error:


java.io.IOException: failure to login
org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:844) ~[hadoop-common-2.7.7.jar:na]
    at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:780) ~[hadoop-common-2.7.7.jar:na]
    at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:653) ~[hadoop-common-2.7.7.jar:na]
    at org.apache.spark.util.Utils$$anonfun$getCurrentUserName$1.apply(Utils.scala:2422) ~[spark-core_2.11-2.4.3.jar:2.4.3]
    at org.apache.spark.util.Utils$$anonfun$getCurrentUserName$1.apply(Utils.scala:2422) ~[spark-core_2.11-2.4.3.jar:2.4.3]
    at scala.Option.getOrElse(Option.scala:121) ~[scala-library-2.11.12.jar:na]
    at org.apache.spark.util.Utils$.getCurrentUserName(Utils.scala:2422) ~[spark-core_2.11-2.4.3.jar:2.4.3]
    at org.apache.spark.SparkContext.<init>(SparkContext.scala:293) ~[spark-core_2.11-2.4.3.jar:2.4.3]
    at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2520) ~[spark-core_2.11-2.4.3.jar:2.4.3]
    at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:935) ~[spark-sql_2.11-2.4.3.jar:2.4.3]
    at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:926) ~[spark-sql_2.11-2.4.3.jar:2.4.3]
    at scala.Option.getOrElse(Option.scala:121) ~[scala-library-2.11.12.jar:na]
    at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:926) ~[spark-sql_2.11-2.4.3.jar:2.4.3]
    at <implementationclass>.<methodName>(ConsumerServiceImpl.java:116) [classes/:na]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_131]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_131]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_131]
    at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_131]
    at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) [junit-4.12.jar:4.12]
    at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) [junit-4.12.jar:4.12]
    at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) [junit-4.12.jar:4.12]
    at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) [junit-4.12.jar:4.12]
    at org.springframework.test.context.junit4.statements.RunBeforeTestExecutionCallbacks.evaluate(RunBeforeTestExecutionCallbacks.java:74) [spring-test-5.1.8.RELEASE.jar:5.1.8.RELEASE]
    at org.springframework.test.context.junit4.statements.RunAfterTestExecutionCallbacks.evaluate(RunAfterTestExecutionCallbacks.java:84) [spring-test-5.1.8.RELEASE.jar:5.1.8.RELEASE]
    at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) [junit-4.12.jar:4.12]
    at org.springframework.test.context.junit4.statements.RunBeforeTestMethodCallbacks.evaluate(RunBeforeTestMethodCallbacks.java:75) [spring-test-5.1.8.RELEASE.jar:5.1.8.RELEASE]
    at org.springframework.test.context.junit4.statements.RunAfterTestMethodCallbacks.evaluate(RunAfterTestMethodCallbacks.java:86) [spring-test-5.1.8.RELEASE.jar:5.1.8.RELEASE]
    at org.springframework.test.context.junit4.statements.SpringRepeat.evaluate(SpringRepeat.java:84) [spring-test-5.1.8.RELEASE.jar:5.1.8.RELEASE]
    at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) [junit-4.12.jar:4.12]
    at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:251) [spring-test-5.1.8.RELEASE.jar:5.1.8.RELEASE]
    at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:97) [spring-test-5.1.8.RELEASE.jar:5.1.8.RELEASE]
    at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) [junit-4.12.jar:4.12]
    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) [junit-4.12.jar:4.12]
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) [junit-4.12.jar:4.12]
    at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) [junit-4.12.jar:4.12]
    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) [junit-4.12.jar:4.12]
    at org.springframework.test.context.junit4.statements.RunBeforeTestClassCallbacks.evaluate(RunBeforeTestClassCallbacks.java:61) [spring-test-5.1.8.RELEASE.jar:5.1.8.RELEASE]
    at org.springframework.test.context.junit4.statements.RunAfterTestClassCallbacks.evaluate(RunAfterTestClassCallbacks.java:70) [spring-test-5.1.8.RELEASE.jar:5.1.8.RELEASE]
    at org.junit.runners.ParentRunner.run(ParentRunner.java:363) [junit-4.12.jar:4.12]
    at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.run(SpringJUnit4ClassRunner.java:190) [spring-test-5.1.8.RELEASE.jar:5.1.8.RELEASE]
    at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365) [surefire-junit4-2.22.2.jar:2.22.2]
    at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273) [surefire-junit4-2.22.2.jar:2.22.2]
    at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238) [surefire-junit4-2.22.2.jar:2.22.2]
    at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159) [surefire-junit4-2.22.2.jar:2.22.2]
    at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384) [surefire-booter-2.22.2.jar:2.22.2]
    at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345) [surefire-booter-2.22.2.jar:2.22.2]
    at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126) [surefire-booter-2.22.2.jar:2.22.2]
    at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418) [surefire-booter-2.22.2.jar:2.22.2]
Caused by: javax.security.auth.login.LoginException: java.lang.NullPointerException: invalid null input: name
    at com.sun.security.auth.UnixPrincipal.<init>(UnixPrincipal.java:71)
    at com.sun.security.auth.module.UnixLoginModule.login(UnixLoginModule.java:133)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)
    at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195)
    at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682)
    at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
    at javax.security.auth.login.LoginContext.login(LoginContext.java:587)
    at org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:818)
    at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:780)
    at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:653)
    at org.apache.spark.util.Utils$$anonfun$getCurrentUserName$1.apply(Utils.scala:2422)
    at org.apache.spark.util.Utils$$anonfun$getCurrentUserName$1.apply(Utils.scala:2422)
    at scala.Option.getOrElse(Option.scala:121)
    at org.apache.spark.util.Utils$.getCurrentUserName(Utils.scala:2422)
    at org.apache.spark.SparkContext.<init>(SparkContext.scala:293)
    at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2520)
    at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:935)
    at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:926)
    at scala.Option.getOrElse(Option.scala:121)
    at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:926)
    at com.sap.newton.forwarding.application.service.impl.ConsumerServiceImpl.convertJsonToParquet(ConsumerServiceImpl.java:116)
    at com.sap.newton.forwarding.application.service.impl.ConsumerServiceImpl.uploadReadings(ConsumerServiceImpl.java:96)
    at com.sap.newton.forwarding.application.service.impl.ConsumerServiceImpl.processRecords(ConsumerServiceImpl.java:56)
    at com.sap.newton.forwarding.application.service.ConsumerServiceTest.testProcessSingleRecord(ConsumerServiceTest.java:101)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
    at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
    at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
    at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
    at org.springframework.test.context.junit4.statements.RunBeforeTestExecutionCallbacks.evaluate(RunBeforeTestExecutionCallbacks.java:74)
    at org.springframework.test.context.junit4.statements.RunAfterTestExecutionCallbacks.evaluate(RunAfterTestExecutionCallbacks.java:84)
    at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
    at org.springframework.test.context.junit4.statements.RunBeforeTestMethodCallbacks.evaluate(RunBeforeTestMethodCallbacks.java:75)
    at org.springframework.test.context.junit4.statements.RunAfterTestMethodCallbacks.evaluate(RunAfterTestMethodCallbacks.java:86)
    at org.springframework.test.context.junit4.statements.SpringRepeat.evaluate(SpringRepeat.java:84)
    at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
    at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:251)
    at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:97)
    at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
    at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
    at org.springframework.test.context.junit4.statements.RunBeforeTestClassCallbacks.evaluate(RunBeforeTestClassCallbacks.java:61)
    at org.springframework.test.context.junit4.statements.RunAfterTestClassCallbacks.evaluate(RunAfterTestClassCallbacks.java:70)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
    at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.run(SpringJUnit4ClassRunner.java:190)
    at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
    at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
    at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
    at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
    at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
    at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
    at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
    at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
    at javax.security.auth.login.LoginContext.invoke(LoginContext.java:856) ~[na:1.8.0_131]
    at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195) ~[na:1.8.0_131]
    at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682) ~[na:1.8.0_131]
    at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680) ~[na:1.8.0_131]
    at java.security.AccessController.doPrivileged(Native Method) ~[na:1.8.0_131]
    at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680) ~[na:1.8.0_131]
    at javax.security.auth.login.LoginContext.login(LoginContext.java:587) ~[na:1.8.0_131]
    at org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:818) ~[hadoop-common-2.7.7.jar:na]

SparkSession spark = SparkSession.builder() 
.config("spark.sql.warehouse.dir", "target/spark-warehouse")            
.master("local[4]") 
.appName("App")         
.getOrCreate()

It should execute successfully

                Can you show us your login code where you set UserGroupInformation. If you don't have it in your test, take a look at stackoverflow.com/questions/41864985/…
– mazaneicha
                Jul 30, 2019 at 13:38
                @mazaneicha : Thanks for your input, I have solved the problem by setting the UserGroupInformation realUser = UserGroupInformation.createRemoteUser("mapr");         UserGroupInformation.setLoginUser(realUser); ------------------------------------------------------------------------------------ After that I was getting error similar to stackoverflow.com/questions/50288034/… Then I found the root cause of issue by replacing the maven:3.5.0-jdk-8-alpine with  maven:3.5.0-jdk-8 Finally it worked :)
– raj03
                Jul 31, 2019 at 13:45
        
Thanks for contributing an answer to Stack Overflow!
  • Please be sure to answer the question. Provide details and share your research!
But avoid …
  • Asking for help, clarification, or responding to other answers.
  • Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
 
推荐文章
力能扛鼎的饭卡  ·  写一个正则代码提取中文VBA - CSDN文库
4 月前
从容的炒面  ·  nacos版本升级和开启权限认证配置（1.3.2—2.0.3）-CSDN博客
6 月前
独立的荔枝  ·  java如何让请求排队执行_mob64ca12d7c9ee的技术博客_51CTO博客
11 月前
傲视众生的凉面  ·  log4net 的简单配置_51CTO博客_log4net 配置
1 年前
爱笑的感冒药  ·  json 文件读写（java对象和json文件互相转换） - 个人文章 - SegmentFault 思否
1 年前
今天看啥   ·   Py中国   ·   codingpro   ·   藏经阁   ·   小百科   ·   link之家   ·   卧龙AI搜索
删除内容请联系邮箱 2879853325@qq.com
Code - 代码工具平台
© 2024 ~ 沪ICP备11025650号