Exception in thread “main” org.apache.spark.sql.AnalysisException: cannot resolve ‘
pibid
’ given input columns: [spark_catalog.lg_edu_warehouse.dwd_t_baseexamhead.courseno, spark_catalog.lg_edu_warehouse.dwd_t_baseexamhead.createtime, spark_catalog.lg_edu_warehouse.dwd_t_baseexamhead.creator, spark_catalog.lg_edu_warehouse.dwd_t_baseexamhead.dataflag, spark_catalog.lg_edu_warehouse.dwd_t_baseexamhead.examendtime, spark_catalog.lg_edu_warehouse.dwd_t_baseexamhead.examlevel, spark_catalog.lg_edu_warehouse.dwd_t_baseexamhead.examstarttime, spark_catalog.lg_edu_warehouse.dwd_t_baseexamhead.fullmark, spark_catalog.lg_edu_warehouse.dwd_t_baseexamhead.globalgrade, spark_catalog.lg_edu_warehouse.dwd_t_baseexamhead.gradeid, spark_catalog.lg_edu_warehouse.dwd_t_baseexamhead.gradename, spark_catalog.lg_edu_warehouse.dwd_t_baseexamhead.havenostunum, spark_catalog.lg_edu_warehouse.dwd_t_baseexamhead.havestunum, spark_catalog.lg_edu_warehouse.dwd_t_baseexamhead.id, spark_catalog.lg_edu_warehouse.dwd_t_baseexamhead.import, spark_catalog.lg_edu_warehouse.dwd_t_baseexamhead.islatest, spark_catalog.lg_edu_warehouse.dwd_t_baseexamhead.lasteditor, spark_catalog.lg_edu_warehouse.dwd_t_baseexamhead.lastedittime, spark_catalog.lg_edu_warehouse.dwd_t_baseexamhead.pubid, spark_catalog.lg_edu_warehouse.dwd_t_baseexamhead.ratea, spark_catalog.lg_edu_warehouse.dwd_t_baseexamhead.rateb, spark_catalog.lg_edu_warehouse.dwd_t_baseexamhead.ratec, spark_catalog.lg_edu_warehouse.dwd_t_baseexamhead.remark, spark_catalog.lg_edu_warehouse.dwd_t_baseexamhead.schoolid, spark_catalog.lg_edu_warehouse.dwd_t_baseexamhead.subjectid, spark_catalog.lg_edu_warehouse.dwd_t_baseexamhead.subjectname, spark_catalog.lg_edu_warehouse.dwd_t_baseexamhead.term]; line 1 pos 19;
在spark中遇到Exception in thread “main” org.apache.spark.sql.，这往往是所选取 'pibid'字段不存在，或者名字写错了。所以要查该字段！！！Exception in thread “main” org.apache.spark.sql.AnalysisException: cannot resolve ‘pibid’ given input columns: [spark_catalog.lg_edu_warehouse.dwd_t_baseexamh
这个是报错信息
Exception
in
thread
"
main
"
org
.
apache
.
spark
.
sql
.
Analysis
Exception
: Cannot up cast `age` from bigint to int.
The type path of the target object is:
- field (class: "scala.Int", name: "age")
Exception
in
thread
"
main
"
org
.
apache
.
spark
.
sql
.
Analysis
E
Table or view not found: aaa.bbb
The column number of the existing table dmall_search.query_embedding_data_1(struct<>) doesn’t match the data schema(struct<user_id:string,dt:string,sku_list:array>);
Cannot insert into table ddw_ware.purchase_d.
Exception
in
thread
"
main
"
org
.
apache
.
spark
.
sql
.
Analysis
Exception
: Table or view not found: emp; line 1 pos 47
at
org
.
apache
.
spark
.
sql
.catalyst.analys...
从异常
中
看到是无法找到
spark
Session类，但是我在maven
中
已经引入了该依赖，却依然出现异常，这个时候可以查看pom文件引入的依赖
中
scope标签
中
的内容，其
中
scope标签
中
对应的参数有如下几种：
1.compile
默认的scope，表示 dependency 都可以在生命周期
中
使用。而且，.
1、意外退出
spark
-shell，而不是quit，然后再输入
spark
-shell命令的时候，报错：
19/04/11 13:42:32 WARN util.Utils: Service '
Spark
UI' could not bind on port 4040. Attempting port 4041.
jps查看，
Spark
Submit在，查看4040端口也是被
Spark
Submit占用...
org
.
apache
.
spark
.
Spark
Exception
: Task not serializable
at
org
.
apache
.
spark
.util.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:298)
at
org
.
apache
.
spark
.util.ClosureCleaner$.
org
$
apache
$
spark
$util$ClosureCleaner$$clean(ClosureCleaner.scala:288)
at
org
.
apache
.
spark
.util.ClosureCleaner$
import pandas as pd
from py
spark
.
sql
import
Spark
Session
from py
spark
.
sql
import
SQL
Context
from py
spark
import
Spark
Context
#初始化数据
#初始化pandas DataFrame
df = pd.DataFrame([[1, 2, 3], [4, 5, 6]], index=['row1', 'row2'], columns=['c1', 'c2', 'c3'])
#打印数据
Traceback (most recent call last):
File /home/cisco/
spark
-mongo-test.py, line 7, in
df =
spark
.read.format(com.mongodb.
spark
.
sql
.DefaultSource).load()
File /home/cisco/
spark
-2.2.0-bin-
hadoop
2.7/python/lib/py
spark
.zip/py
spark
/
sql
/readwriter.py, line 165, in load
