1. coalesce

def coalesce(numPartitions: Int, shuffle: Boolean = false)(implicit ord: Ordering[T] = null): RDD[T]

該函數用于將RDD進行重分區，使用HashPartitioner。第一個參數為重分區的數目，第二個為是否進行shuffle，默認為false.

代碼測試如下：

scala> var data = sc.textFile("example.txt") data: org.apache.spark.rdd.RDD[String] = MapPartitionsRDD[53] at textFile at :21scala> data.collect res1: Array[String] = Array(hello world, hello spark, hello hive, hi spark)scala> data.partitions.size res2: Int = 2 //RDD data默認有兩個分區scala> var rdd1 = data.coalesce(1) rdd1: org.apache.spark.rdd.RDD[String] = CoalescedRDD[2] at coalesce at :23scala> rdd1.partitions.size res3: Int = 1 //rdd1的分區數為1scala> var rdd1 = data.coalesce(4) rdd1: org.apache.spark.rdd.RDD[String] = CoalescedRDD[3] at coalesce at :23scala> rdd1.partitions.size res4: Int = 2 //如果重分區的數目大于原來的分區數，那么必須指定shuffle參數為true，否則，分區數不變scala> var rdd1 = data.coalesce(4,true) rdd1: org.apache.spark.rdd.RDD[String] = MapPartitionsRDD[7] at coalesce at :23scala> rdd1.partitions.size res5: Int = 4

2.?repartition

def repartition(numPartitions: Int)(implicit ord: Ordering[T] = null): RDD[T]

該函數其實就是coalesce函數第二個參數為true的實現

代碼測試如下：

scala> var rdd2 = data.repartition(1) rdd2: org.apache.spark.rdd.RDD[String] = MapPartitionsRDD[11] at repartition at :23scala> rdd2.partitions.size res6: Int = 1scala> var rdd2 = data.repartition(4) rdd2: org.apache.spark.rdd.RDD[String] = MapPartitionsRDD[15] at repartition at :23scala> rdd2.partitions.size res7: Int = 4

?

與50位技術專家面對面20年技術見證，附贈技術全景圖

總結

以上是生活随笔為你收集整理的Spark算子：RDD基本转换操作–coalesce、repartition的全部內容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網站內容還不錯，歡迎將生活随笔推薦給好友。