ES报错:Connection reset by peer 解决经历
http://nicethemes.cn/news/txtlist_i28391v.html
這次來分享一下ES報錯:java.io.IOException: Connection reset by peer 的解決經歷
問題描述
本人最近負責了定時獲取Prometheus Metrics并發送到ES做持久化存儲的任務。然而在Metrics采集粒度從3分鐘變為1小時后(這里使用的是Springboot的定時采集組件Scheduler),報了如下的錯誤:
java.io.IOException: Connection reset by peerat org.elasticsearch.client.RestClient.extractAndWrapCause(RestClient.java:793) ~[elasticsearch-rest-client-7.4.0.jar!/:7.4.0]at org.elasticsearch.client.RestClient.performRequest(RestClient.java:218) ~[elasticsearch-rest-client-7.4.0.jar!/:7.4.0]at org.elasticsearch.client.RestClient.performRequest(RestClient.java:205) ~[elasticsearch-rest-client-7.4.0.jar!/:7.4.0]at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1454) ~[elasticsearch-rest-high-level-client-7.4.0.jar!/:7.4.0]at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1439) ~[elasticsearch-rest-high-level-client-7.4.0.jar!/:7.4.0]at org.elasticsearch.client.IndicesClient.exists(IndicesClient.java:785) ~[elasticsearch-rest-high-level-client-7.4.0.jar!/:7.4.0]at com.free4inno.scheduler.adapter.service.elasticsearch.GenericEsService.isIndexExists(GenericEsService.java:60) ~[classes!/:0.0.1-SNAPSHOT]at com.free4inno.scheduler.adapter.service.elasticsearch.MetricsToEsService.getTodayIndex(MetricsToEsService.java:56) ~[classes!/:0.0.1-SNAPSHOT]at com.free4inno.scheduler.adapter.service.elasticsearch.MetricsToEsService.insert(MetricsToEsService.java:49) ~[classes!/:0.0.1-SNAPSHOT]at com.free4inno.scheduler.adapter.service.Scheduler.sendMetrics(Scheduler.java:64) [classes!/:0.0.1-SNAPSHOT]at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_261]at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_261]at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_261]at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_261]at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:84) [spring-context-5.2.8.RELEASE.jar!/:5.2.8.RELEASE]at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) [spring-context-5.2.8.RELEASE.jar!/:5.2.8.RELEASE]at org.springframework.scheduling.concurrent.ReschedulingRunnable.run(ReschedulingRunnable.java:93) [spring-context-5.2.8.RELEASE.jar!/:5.2.8.RELEASE]at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_261]at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_261]at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_261]at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_261]at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_261]at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_261]at java.lang.Thread.run(Thread.java:748) [?:1.8.0_261] Caused by: java.io.IOException: Connection reset by peerat sun.nio.ch.FileDispatcherImpl.read0(Native Method) ~[?:1.8.0_261]at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) ~[?:1.8.0_261]at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) ~[?:1.8.0_261]at sun.nio.ch.IOUtil.read(IOUtil.java:197) ~[?:1.8.0_261]at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:378) ~[?:1.8.0_261]at org.apache.http.impl.nio.reactor.SessionInputBufferImpl.fill(SessionInputBufferImpl.java:231) ~[httpcore-nio-4.4.13.jar!/:4.4.13]at org.apache.http.impl.nio.codecs.AbstractMessageParser.fillBuffer(AbstractMessageParser.java:136) ~[httpcore-nio-4.4.13.jar!/:4.4.13]at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:241) ~[httpcore-nio-4.4.13.jar!/:4.4.13]at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:81) ~[httpasyncclient-4.1.4.jar!/:4.1.4]at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:39) ~[httpasyncclient-4.1.4.jar!/:4.1.4]at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:114) ~[httpcore-nio-4.4.13.jar!/:4.4.13]at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162) ~[httpcore-nio-4.4.13.jar!/:4.4.13]at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337) ~[httpcore-nio-4.4.13.jar!/:4.4.13]at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315) ~[httpcore-nio-4.4.13.jar!/:4.4.13]at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276) ~[httpcore-nio-4.4.13.jar!/:4.4.13]at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104) ~[httpcore-nio-4.4.13.jar!/:4.4.13]at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591) ~[httpcore-nio-4.4.13.jar!/:4.4.13]... 1 more更令人費解的是,在采集Metrics發送至ES的過程中,在單數的整點寫入ES是正常的,然而一到雙數的整點就不正常了,例如0:00開始發送是可以發送到ES的,1:00就發送不到報錯,2:00就重新恢復正常了…
后面經過調研,是因為esClient自動設置的KeepAlive時間為-1,也就是持續連接,然而這回受到外界的影響比如Firewall,會將TCP連接單方面斷開,從而會導致Connection reset by peer的報錯。
解決方案
在設置RestHighLevelClient的時候手動設置一下setKeepAliveStrategy的方法,在這里設置的是3分鐘的時間(不要太長)
@Beanpublic RestHighLevelClient restHighLevelClient(@Autowired RestClientBuilder restClientBuilder){return new RestHighLevelClient(restClientBuilder.setHttpClientConfigCallback(requestConfig ->requestConfig.setKeepAliveStrategy((response, context) -> TimeUnit.MINUTES.toMillis(3))));}或者也可以使用心跳,隔一段時間就去獲取一次es的連接狀態,以此來保持連接的活躍,個人認為這是下策,這里就沒有進行嘗試。
https://blog.csdn.net/qq_33999844/article/details/113843845
Elasticsearch出現Connection reset by peer分析
1.異常:
Caused by: java.io.IOException: Connection reset by peer
at org.elasticsearch.client.RestClient.extractAndWrapCause(RestClient.java:793)
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:218)
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:205)
at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1454)
at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1424)
2.分析
RestHighLevelClient客戶端,使用的是apache httpclient,版本為4.x,keepAlive默認為-1(客戶端會一直保持session);
當服務端因為超時或者其他原因關閉session,客戶端仍然認為長連接存在,拋出異常;
注:當ES服務端的keepAlive短于ES客戶端的keepAlive,也會導致:服務端已經關閉了連接,客戶端繼續復用該連接,拋出異常。
3.解決
手動設置KeepAliveStrategy來配置keepAlive,保證客戶端keepAlive小于服務端keepAlive,讓客戶端先于服務端關閉連接
————————————————
版權聲明:本文為CSDN博主「夜-NULL(zmc)」的原創文章,遵循CC 4.0 BY-SA版權協議,轉載請附上原文出處鏈接及本聲明。
原文鏈接:https://blog.csdn.net/qq_33999844/article/details/113843845
總結
以上是生活随笔為你收集整理的ES报错:Connection reset by peer 解决经历的全部內容,希望文章能夠幫你解決所遇到的問題。
- 上一篇: 笔记本玩魔兽世界要什么配置?
- 下一篇: 批量删除文件