请先检查代码,是否遗漏了字段,也就是要解析的字段XXX不存在。如果真的漏了,补上即可,不需要再往下看了。

具体报错日志如下:

ApplicationMaster: User class threw exception: org.apache.spark.sql.AnalysisException: Resolved attribute(s) team_id#51L missing from team_id#479L , … in operator !Join LeftOuter, (latest_secondary_team_id#328L = team_id#51L). Attribute(s) with the same name appear in the operation: team_id. Please check if the right attribute(s) are used.;;
可以看到少了team_id#51L,但日志里后面的所有字段中,是有team_id#479L的,但数字不一样。

什么场景下会出现这种问题:

A表joinB表的a\b\c字段得到C表,然后拿C表又 join了B表a\b\c字段时,此时可能会报错提示:Resolved attribute(s) 字段a#一个数字 missing from 字段a#另一个数字, …。推测是因为C join B如果能成功,会有2个a\b\c字段,但又不能有重名的字段。

解决方案一句话说明:

Reuse of the reference will create ambiguity in naming, so you will have to clone the df,或者重命名所有的列。(对引用的重复使用,会造成名字上的二义性,所以你需要克隆这个DataFrame或者重命名你用到的列)

我的代码:重命名所有需要的列

	// 选取需要的列,然后重命名所有列
    val recalculationTeamInfo = distinctRtxOrbacInfo.select("first_team_name")
      .withColumnRenamed("first_team_name", "new_" + "first_team_name")

PS: val dataframe2 = dataframe1不是克隆dataframe的正确写法。
val dataframe1Clone = dataframe1.as(“dataframe1Clone”)我试过这个方法,有时候可以解决这个问题的,但有时候不行,所以我一般推荐重命名列的方法,一了百了哈哈。

https://stackoverflow.com/questions/45713290/how-to-resolve-the-analysisexception-resolved-attributes-in-spark/53848160

Spark: org.apache.spark.sql.AnalysisException: Resolved attribute(s) ... missing from.... 背景:Spark scala代码处理数据。前言:请先检查代码,是否遗漏了字段,也就是要解析的字段XXX不存在。如果真的漏了,补上即可,不需要再往下看了。具体报错日志如下:ApplicationMaster: User class threw exception: org.apache.spark.sql.AnalysisException: Resolved attribute(s) team_id#51L missing from team_id#479L, … in operator !Join
What is New in TreeSize?​ What new features are planned for the near future? Your opinion counts. Be part of the decision team and vote here for new TreeSize features. Version 8.1.2 (7 April 2021)​ TreeSize is now also available in Bulgarian and Hungarian. The scan speed could be increased, especially for fast drives. On NetApp Storage System, snapshots were sometimes not displayed when using the "Compare with snapshots" function. This problem has been fixed. The behavior when loading XML scans via the "Open with" function of Windows Explorer has been improved. A possible application freeze when changing the selection in the directory tree during a running export has been fixed. When deleting from the Top 100 list, it could happen that other entries were duplicated. This issue has now been resolved. Scans loaded from an XML file now again show the date of the scan after the root directory name again. The behavior for the /EXPORTTITLE parameter has been corrected. The setting now affects the title of the export as expected, instead of just a subtitle. File search: The command line parameter /EXPORTTITLE is now evaluated correctly for file search. File search: A possible problem with the display of context menus has been fixed. File search: The printing of search results works again as it used to. Other minor fixes and improvements have been made.
这里写自定义目录标题项目场景:问题描述:原因分析:解决方案: 项目场景: sparksql Error in query: resolved attribute(s) XX#4245L missing from XX#4520,XX#684L 错误 问题描述: 提示:spark实际生产环境中 使用sparksql进行同表关联时 报错代码如下 select distinct a.imsi from ( select imsi,ho_time,longitude,latitude 3、然后再将这部分用户id关联回行为表,获得这部分用户的所有行为 具体代码我就不贴出来了,应该比较简单吧,就是读取行为表,过滤特定的行为,然后将这部分数据去重就得到特定行为的用户,最后把这些用户关联回行为表,就拿到了这部分特定用户的行为 写sparksql语句时出现的错误,他说该字段不在表中 Exception in thread "main" org.apache.spark.sql.AnalysisException: cannot resolve '`Class`' given input columns: [Cno, Cnam, Tno]; line 1 pos 28; 'Project [*] +- 'Filter...
org.apache.poi JAR包,解决个人的 import org.apache.commons.beanutils.PropertyUtilsBean; import org.apache.commons.lang.StringUtils; import org.apache.poi.hssf.usermodel.HSSFCell; import org.apache.poi.hssf.usermodel.HSSFCellStyle; import org.apache.poi.hssf.usermodel.HSSFFont; import org.apache.poi.hssf.usermodel.HSSFPalette; import org.apache.poi.hssf.usermodel.HSSFRichTextString; import org.apache.poi.hssf.usermodel.HSSFRow; import org.apache.poi.hssf.usermodel.HSSFSheet; import org.apache.poi.hssf.usermodel.HSSFWorkbook; import org. apache.poi.hssf.util.HSSFColor; import org.apache.poi.hssf.util.Region; import org.apache.poi.poifs.filesystem.POIFSFileSystem; import org.apache.poi.ss.util.CellRangeAddress; "The import org.apache.poi cannot be resolved"的问题
Spark sql 版本2.3.0,grouping sets之后jion相同表会出现列名不存在的问题。会导致报错 org.apache.spark.sql.AnalysisException: cannot resolve '`a.a`' given input columns: [b.b, b.c, a.a, a.b, a.c]; line 8 pos 29; org.apache.spark.sql.AnalysisException: Failure when resolving confl
更多文章,可关注微信公众号:excelwork 今天介绍的是在使用Spark SQL查询数据遇到的一些常见报错案例,为什么要说这个呢,因为,有些错误,网上搜索得到答案时间太长了。 如何快速定位问题? Exception:不啦不啦的一大堆~,熟悉不!?我们只要看Exception冒号后面说的那一句提示就好了,里面会有什么错了、错在哪了的信息。 会有哪些报错信息? 报错信息: java.sql.SQLException: org.apache.spark.sql.Analysi...
spark中遇到Exception in thread “main” org.apache.spark.sql.,这往往是所选取 'pibid'字段不存在,或者名字写错了。所以要查该字段!!! Exception in thread “main” org.apache.spark.sql.AnalysisException: cannot resolve ‘pibid’ given input columns: [spark_catalog.lg_edu_warehouse.dwd_t_baseexamh
严重: Servlet.service() for servlet jsp threw exception java.lang.UnsupportedClassVersionError: com/util/PostUtil     at org.apache.jsp.shouye.index_jsp._jspService(index_jsp.java:84)     at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)     at javax.servlet.http.HttpServlet.servic
Maven更新问题 今天Maven在更新的时候发现一直更新不成功,总结下解决方法。 在apache-maven-3.5.2/conf/setting.xml中加入以下配置即可解决 alimaven aliyun maven http://maven.aliyun.com/nexus/content/repositories/central/ central junit junit Address/ http://jcenter.bintray.com/ central alimaven aliyun maven
在Java编程中,当出现“the import org.apache.commons.codec cannot be resolved”的错误时,通常是因为没有正确地导入相关的库文件。org.apache.commons.codec库文件通常用于编码和解码操作,如Base64编码、MD5哈希、URL编码等。因此,为了解决这个错误,我们需要确保正确地导入了org.apache.commons.codec相关的库文件。这通常包括以下步骤: 首先,确保我们已经下载并安装了org.apache.commons.codec库文件。可以从官方网站或第三方软件库进行下载。 接着,我们需要在Java代码中正确地导入这些库文件。通常可以使用import语句来实现。例如,import org.apache.commons.codec.binary.Base64; 如果这些步骤都已经完成,但仍然出现“the import org.apache.commons.codec cannot be resolved”的错误,可以考虑以下几种可能的解决方案: 检查编译器的设置,确认其是否正确地链接了库文件。 检查Java项目的构建路径,确认是否正确地包含了所需的库文件和资源。 检查代码是否存在其他错误或警告,这些错误或警告可能会影响到库文件的导入。
java.net.SocketTimeoutException: failed to connect to /10.0..2.2 (port 8080) after 10000ms 68770