日志報錯（WARN類型最后執行成功可以忽略）：

19/04/15 12:35:37 INFO memory.MemoryStore: Will not store rdd_2_5119/04/15 12:35:37 WARN memory.MemoryStore: Not enough space to cache rdd_2_51 in memory! (computed 1109.7 MB so far)19/04/15 12:35:37 INFO memory.MemoryStore: Memory use = 627.7 MB (blocks) + 1109.7 MB (scratch space shared across 1 tasks(s)) = 1737.4 MB. Storage limit = 2004.6 MB.19/04/15 12:35:37 WARN storage.BlockManager: Persisting block rdd_2_51 to disk instead.19/04/15 13:01:46 INFO zookeeper.ReadOnlyZKClient: Close zookeeper connection 0x331abd57 to fwqml016.zh:2181,fwqml018.zh:2181,fwqml009.zh:218119/04/15 13:01:49 INFO memory.MemoryStore: Will not store rdd_2_5119/04/15 13:01:49 WARN memory.MemoryStore: Not enough space to cache rdd_2_51 in memory! (computed 1109.7 MB so far)19/04/15 13:01:49 INFO memory.MemoryStore: Memory use = 627.7 MB (blocks) + 1109.7 MB (scratch space shared across 1 tasks(s)) = 1737.4 MB. Storage limit = 2004.6 MB.

webUI頁面報錯：

異常提示內存空間不足

解決方法：

1.將"spark.yarn.executor.memoryOverhead"設置為最大值，可以考慮一下4096。這個數值一般都是2的次冪。

2.將rdd進行重新分區，這里可以考慮200k。在spark2.3的版本中，rdd成為了dateframe格式的數據。

3.將"executor.cores"從8設置為4。將core的個數調小，防止cpu不足。

4.將"executor.memory"從8g設置為12g。將內存調大。

?

一般引起內存的問題都可使用該參數進行調整，

如常見異常：

WARN yarn.YarnAllocator: Container killed by YARN for exceeding memory limits. 2.2 GB of 2.1 GB virtual memory used. Consider boosting spark.yarn.executor.memoryOverhead.

當executor的內存使用大于executor-memory與executor.memoryOverhead的加和時，Yarn會干掉這些executor，

修改上面描述參數也可以解決該問題

?

?

最初導致異常提交的配置為：

spark-submit \ --master yarn \ --deploy-mode cluster \ --class com.data.filter.ArticleFilter \ --jars $(echo ./lib/*.jar | tr ' ' ',') \ --num-executors 50 \ --executor-cores 2 \ --conf spark.yarn.queue=etl \ --executor-memory 4g \ --driver-memory 1g \ --conf spark.kryoserializer.buffer.max=2000 \ --conf spark.akka.frameSize=500 \ --conf spark.sql.shuffle.partitions=100 \ --conf spark.default.parallelism=100 \ --conf spark.storage.memoryFraction=0.3 \ --conf spark.shuffle.memoryFraction=0.7 \ --conf spark.shuffle.safetyFraction=0.8 \ --conf spark.shuffle.spill=true \ --conf spark.yarn.queue=etl \ --conf spark.serializer=org.apache.spark.serializer.KryoSerializer \ ./data-filter.jar

修改Executor-memory后執行成功的參數，修改為 --executor-memory 8g?

spark-submit \ --master yarn \ --deploy-mode cluster \ --class com.data.filter.ArticleFilter \ --jars $(echo ./lib/*.jar | tr ' ' ',') \ --num-executors 50 \ --executor-cores 2 \ --conf spark.yarn.queue=etl \ --executor-memory 8g \ --driver-memory 1g \ --conf spark.kryoserializer.buffer.max=2000 \ --conf spark.akka.frameSize=500 \ --conf spark.sql.shuffle.partitions=100 \ --conf spark.default.parallelism=100 \ --conf spark.storage.memoryFraction=0.3 \ --conf spark.shuffle.memoryFraction=0.7 \ --conf spark.shuffle.safetyFraction=0.8 \ --conf spark.shuffle.spill=true \ --conf spark.yarn.queue=etl \ --conf spark.serializer=org.apache.spark.serializer.KryoSerializer \ ./data-filter.jar

或修改--conf spark.yarn.executor.memoryOverhead=4096參數同樣也執行成功了

spark-submit \ --master yarn \ --deploy-mode cluster \ --class com.data.filter.ArticleFilter \ --jars $(echo ./lib/*.jar | tr ' ' ',') \ --num-executors 50 \ --executor-cores 2 \ --conf spark.yarn.queue=etl \ --executor-memory 4g \ --driver-memory 1g \ --conf spark.yarn.executor.memoryOverhead=4096 \ --conf spark.kryoserializer.buffer.max=2000 \ --conf spark.akka.frameSize=500 \ --conf spark.sql.shuffle.partitions=100 \ --conf spark.default.parallelism=100 \ --conf spark.storage.memoryFraction=0.3 \ --conf spark.shuffle.memoryFraction=0.7 \ --conf spark.shuffle.safetyFraction=0.8 \ --conf spark.shuffle.spill=true \ --conf spark.yarn.queue=etl \ --conf spark.serializer=org.apache.spark.serializer.KryoSerializer \ ./data-filter.jar

?

?

總結

以上是生活随笔為你收集整理的Spark 运行内存不足Not enough space to cache rdd in memory，Container killed by YARN for exceeding memory的全部內容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網站內容還不錯，歡迎將生活随笔推薦給好友。