報錯：

17/06/24 01:30:25 ERROR Utils: Exception encountered py4j.Py4JException: Cannot obtain a new communication channelat py4j.CallbackClient.sendCommand(CallbackClient.java:340)at py4j.CallbackClient.sendCommand(CallbackClient.java:316)at py4j.reflection.PythonProxyHandler.invoke(PythonProxyHandler.java:103)at com.sun.proxy.$Proxy8.dumps(Unknown Source)at org.apache.spark.streaming.api.python.PythonTransformFunctionSerializer$.serialize(PythonDStream.scala:141)at org.apache.spark.streaming.api.python.TransformFunction$$anonfun$writeObject$1.apply$mcV$sp(PythonDStream.scala:101)at org.apache.spark.streaming.api.python.TransformFunction$$anonfun$writeObject$1.apply(PythonDStream.scala:100)at org.apache.spark.streaming.api.python.TransformFunction$$anonfun$writeObject$1.apply(PythonDStream.scala:100)at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1269)at org.apache.spark.streaming.api.python.TransformFunction.writeObject(PythonDStream.scala:100)at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source)at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)at java.lang.reflect.Method.invoke(Method.java:498)at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:1028)at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1496)at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)at java.io.ObjectOutputStream.writeArray(ObjectOutputStream.java:1378)at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1174)at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)

因為代碼中設置了local directory：

ssc.checkpoint("checkpoint")

需要設置成HDFS directory。

?

?

【參考】：http://45.55.83.242:4000/blog/2016/02/22/note-Some-Spark-issues-and-solutions/

轉載于:https://www.cnblogs.com/zhzhang/p/7073064.html

總結

以上是生活随笔為你收集整理的spark-streaming问题集锦的全部內容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網站內容還不錯，歡迎將生活随笔推薦給好友。