關(guān)于Spark集群部署參考：http://blog.csdn.net/tototuzuoquan/article/details/74481570

一、Spark遠(yuǎn)程調(diào)試配置：

#調(diào)試Master，在master節(jié)點(diǎn)的spark-env.sh中添加SPARK_MASTER_OPTS變量 export SPARK_MASTER_OPTS="-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=10000" #啟動(dòng)Master sbin/start-master.sh#調(diào)試Worker，在worker節(jié)點(diǎn)的spark-env.sh中添加SPARK_WORKER_OPTS變量 export SPARK_WORKER_OPTS="-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=10001" #啟動(dòng)Worker sbin/start-slave.sh 1 spark://hadoop1:7077#調(diào)試spark-submit + app bin/spark-submit --class cn.itcast.spark.WordCount --master spark://hadoop1:7077 --driver-java-options "-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=10002" /root/wc.jar hdfs://mycluster/wordcount/input/2.txt hdfs://mycluster/out2 #調(diào)試spark-submit + app + executor bin/spark-submit --class cn.itcast.spark.WordCount --master spark://hadoop1:7077 --conf "spark.executor.extraJavaOptions=-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=10003" --driver-java-options "-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=10002" /root/wc.jar hdfs://mycluster/wordcount/input/2.txt hdfs://mycluster/out2

二、編寫(xiě)程序進(jìn)行調(diào)試

接著：

與50位技術(shù)專家面對(duì)面20年技術(shù)見(jiàn)證，附贈(zèng)技術(shù)全景圖

總結(jié)

以上是生活随笔為你收集整理的Spark远程调试配置，在IDEA中的配置的全部?jī)?nèi)容，希望文章能夠幫你解決所遇到的問(wèn)題。

如果覺(jué)得生活随笔網(wǎng)站內(nèi)容還不錯(cuò)，歡迎將生活随笔推薦給好友。