pyspark --master yarn

http://laptop:8042/node/containerlogs/container_1588409304631_0005_01_000001/appleyuchi/stderr/?start=-4096

得到:

led to connect to driver at license.sublimehq.com:41651, retrying ... 2020-05-02 16:53:04 ERROR ApplicationMaster:70 - Failed to connect to driver at license.sublimehq.com:41651, retrying ... 2020-05-02 16:53:04 ERROR ApplicationMaster:70 - Failed to connect to driver at license.sublimehq.com:41651, retrying ... 2020-05-02 16:53:04 ERROR ApplicationMaster:70 - Failed to connect to driver at license.sublimehq.com:41651, retrying ... 2020-05-02 16:53:04 ERROR ApplicationMaster:70 - Failed to connect to driver at license.sublimehq.com:41651, retrying ...

檢查:

①

python test.py

然后瀏覽器訪問license.sublimehq.com:41651

如果瀏覽器返回helloworld就證明這個端口已經打開了。

?

②

在第一步確認無誤以后,

$SPARK_HOME/conf/spark-defaults.conf

?中加入[2]:

spark.driver.port=20002
spark.driver.host=Desktop

#------------------------------------------------------------

不用重啟spark，直接再次輸入:

pyspark --master yarn即可

?

Reference:

[1]failing to connect to spark driver when submitting job to spark in yarn mode

?

?

總結

以上是生活随笔為你收集整理的Failed to connect to driver at XXXXXXx的全部內容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網站內容還不錯，歡迎將生活随笔推薦給好友。