環(huán)境:

Ubuntu19.10

anaconda3-python3.6.10

scala 2.11.8

apache-hive-3.0.0-bin

hadoop-2.7.7

spark-2.3.1-bin-hadoop2.7

java version "1.8.0_131"

Mysql Server version: 8.0.19-0ubuntu0.19.10.3 (Ubuntu)

driver:mysql-connector-java-8.0.20.jar

[Driver link|https://mvnrepository.com/artifact/mysql/mysql-connector-java/8.0.20]

使用的代碼是:

import pandas as pd from pyspark.sql import SparkSession from pyspark import SparkContext from pyspark.sql import SQLContextdef map_extract(element):file_path, content = elementyear = file_path[-8:-4]return [(year, i) for i in content.split("\n") if i]spark = SparkSession\.builder\.appName("PythonTest")\.getOrCreate()res = spark.sparkContext.wholeTextFiles('hdfs://Desktop:9000/user/mercury/names',minPartitions=40) \.map(map_extract) \.flatMap(lambda x: x) \.map(lambda x: (x[0], int(x[1].split(',')[2]))) \.reduceByKey(lambda x,y:x+y)df = res.toDF(["key","num"]) #把已有數(shù)據(jù)列改成和目標(biāo)mysql表的列的名字相同 # print(dir(df)) df.printSchema() print(df.show()) df.printSchema()df.write.format("jdbc").options(url="jdbc:mysql://127.0.0.1:3306/leaf",driver="com.mysql.cj.jdbc.Driver",dbtable="spark",user="appleyuchi",password="appleyuchi").mode('append').save()

?

?

提交方式是(下面兩種方式都能復(fù)現(xiàn)bug):

①pyspark --master yarn（然后在交互是模式中輸入交互式代碼）

②spark-submit --master yarn --deploy-mode cluster 源碼.py

③pyspark --master yarn --conf spark.executor.extraClassPath=/home/appleyuchi/bigdata/apache-hive-3.0.0-bin/lib/mysql-connector-java-8.0.20.jar

同樣會報告類似的錯誤

?

解決方案:

https://gitee.com/appleyuchi/cluster_configuration/blob/master/物理環(huán)境配置流程-必須先看.txt

Reference:

[1]https://zhuanlan.zhihu.com/p/136777424

?

?

?

總結(jié)

以上是生活随笔為你收集整理的py4j.protocol.Py4JJavaError: An error occurred while calling o90.save的全部內(nèi)容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網(wǎng)站內(nèi)容還不錯，歡迎將生活随笔推薦給好友。