应用系统瓶颈排查和分析的思考-Arthas 实战
作者 | 一啦米
【Arthas 官方社區正在舉行征文活動,參加即有獎品拿~點擊投稿】
背景
業務應用系統接入流程引擎來處理業務應用的流程執行,流程引擎提供多線程高性能異步化來執行流程元素的執行,但是如何設置流程引擎的線程池線程數執行,以及執行線程數和任務數,應用機器資源使用情況之間的關系如何,目前只能通過接入方的人工經驗評估,比較粗泛評估,參數的合理性也很難直觀的評估,現通過現有的應用系統默認的參數配置進行問題說明。
性能監控
查看 CPU 核數
應用機器的配置是 2 核 CPU 配置:
查看應用機器的負載情況
應用程序的近 15 分鐘內的負載嚴重的超載(平均超出 CPU 執行的 10 倍之多),應用程序對應的 CPU 和 MEM 的利用率都很高,按照本次單機模擬線上的流量(QPS:10) 不至于出現如此的負載情況吧。
查看應用機器的綜合統計
CPU 運行隊列中也嚴重的堵篩(Procs列中指標),應用的 CPU 有效利用率其實并不算太高(US+SY),但是系統出現了嚴重的 CS(上下文切換),配合 BI,BO 執行,也間接反饋出一些端倪,具體反饋到應用程序中,還需要進一步分析。
查看應用服務(PID=512)的進程 CPU 執行情況
查看應用機器的磁盤 IO 使用情況
應用系統的線程統計概要
應用系統中居然有如此多的線程(1285),應用系統接入流程 SDK 中的線程池數據設置也不會超過80個線程,為何有如何大的偏差,考慮到應用系統中還存在 RPC,kafka 等相關的技術棧,對于應用系統的線程分布還需要進一步的分析。
應用系統執行 dashboard
GC 回收線程監控統計
應用系統線程執行棧的統計
阻塞線程 ThreadDump 信息
org.apache.log4j.spi.RootLogger?is blocking 379 threads.
Pigeon-Server-Request-Processor-46-thread-318
Stack Trace is:
java.lang.Thread.State: RUNNABLE
at java.lang.Throwable.printStackTrace(Throwable.java:665)
- locked <0x000000075332fde8> (a java.io.PrintWriter)
at java.lang.Throwable.printStackTrace(Throwable.java:721)
at org.apache.log4j.DefaultThrowableRenderer.render(DefaultThrowableRenderer.java:60)
at org.apache.log4j.spi.ThrowableInformation.getThrowableStrRep(ThrowableInformation.java:87) - locked <0x000000074ae96550> (a org.apache.log4j.spi.ThrowableInformation)
at org.apache.log4j.spi.LoggingEvent.getThrowableStrRep(LoggingEvent.java:413)
at org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:313)
at org.apache.log4j.WriterAppender.append(WriterAppender.java:162)
at com.dianping.combiz.misc.ExtendedConsoleAppender.append(ExtendedConsoleAppender.java:82)
at org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251) - eliminated <0x000000075f880030> (a com.dianping.combiz.misc.ExtendedConsoleAppender)
at com.dianping.combiz.misc.ExtendedConsoleAppender.doAppend(ExtendedConsoleAppender.java:75) - locked <0x000000075f880030> (a com.dianping.combiz.misc.ExtendedConsoleAppender)
at org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66)
at org.apache.log4j.Category.callAppenders(Category.java:206)
- locked <0x000000075ed2bd58> (a org.apache.log4j.spi.RootLogger)
at org.apache.log4j.Category.forcedLog(Category.java:391)
at org.apache.log4j.Category.log(Category.java:856)
at org.slf4j.impl.Log4jLoggerAdapter.error(Log4jLoggerAdapter.java:576)
at com.meituan.mafka.client.producer.DefaultProducerProcessor.doSend(DefaultProducerProcessor.java:832)
at com.meituan.mafka.client.producer.DefaultProducerProcessor.sendMessage(DefaultProducerProcessor.java:603)
at com.meituan.mafka.client.producer.DefaultProducerProcessor.sendMessage(DefaultProducerProcessor.java:584)
at com.meituan.mafka.client.producer.DefaultProducerProcessor.sendMessage(DefaultProducerProcessor.java:567)
at com.meituan.mafka.client.producer.DefaultProducerProcessor.sendMessage(DefaultProducerProcessor.java:562)
at com.dianping.poi.flowcenter.mq.WorkflowEventMQProducer.sendSyncMessage(WorkflowEventMQProducer.java:98)
at com.dianping.poi.flowcenter.WorkflowEngine.createFlowInstance(WorkflowEngine.java:642)
at com.dianping.poi.flowcenter.WorkflowEngine.startWorkflow0(WorkflowEngine.java:560)
at com.dianping.poi.flowcenter.WorkflowEngine.startWorkflow(WorkflowEngine.java:504)
at com.dianping.poi.mainflow.clearengine.engine.create.impl.CreateWorkflowServiceImpl.createWorkflow(CreateWorkflowServiceImpl.java:66)
at com.dianping.poi.mainflow.core.business.WorkflowBusiness.createWorkflow(WorkflowBusiness.java:277)
at com.dianping.poi.mainflow.core.business.WorkflowBusiness.createWorkflow(WorkflowBusiness.java:183)
at com.dianping.poi.mainflow.core.business.WorkflowBusiness.createWorkflow(WorkflowBusiness.java:176)
at com.dianping.poi.mainflow.core.business.WorkflowBusinessFastClassBySpringCGLIBFastClassBySpringCGLIBFastClassBySpringCGLIBf3bb47a4.invoke()
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
at org.springframework.aop.framework.CglibAopProxyCglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:736)atorg.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)atorg.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)atorg.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)atorg.springframework.aop.framework.CglibAopProxyCglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:736) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at org.springframework.aop.framework.CglibAopProxyCglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:736)atorg.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)atorg.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)atorg.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)atorg.springframework.aop.framework.CglibAopProxyDynamicAdvisedInterceptor.intercept(CglibAopProxy.java:671)
at com.dianping.poi.mainflow.core.business.WorkflowBusinessEnhancerBySpringCGLIBEnhancerBySpringCGLIBEnhancerBySpringCGLIBe721a2aa.createWorkflow()
at com.dianping.poi.mainflow.update.biz.business.MainUpdateFlowBusiness.update(MainUpdateFlowBusiness.java:220)
at com.dianping.poi.mainflow.update.biz.business.MainUpdateFlowBusiness.update(MainUpdateFlowBusiness.java:179)
at com.dianping.poi.mainflow.feedback.biz.serviceImpl.FeedbackForPoiServiceImpl.dpFeedbackBase(FeedbackForPoiServiceImpl.java:1233)
at com.dianping.poi.mainflow.feedback.biz.serviceImpl.FeedbackForPoiServiceImpl.dpTerminalFeedback(FeedbackForPoiServiceImpl.java:486)
at sun.reflect.GeneratedMethodAccessor144.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at com.dianping.pigeon.remoting.provider.service.method.ServiceMethod.invoke(ServiceMethod.java:193)
at com.dianping.pigeon.remoting.provider.process.filter.BusinessProcessFilter.invoke(BusinessProcessFilter.java:79)
at com.dianping.pigeon.remoting.provider.process.filter.BusinessProcessFilter.invoke(BusinessProcessFilter.java:34)
at com.dianping.pigeon.remoting.provider.process.ProviderProcessHandlerFactory$1.handle(ProviderProcessHandlerFactory.java:99)
at com.dianping.pigeon.remoting.provider.process.filter.GatewayProcessFilter.invoke(GatewayProcessFilter.java:242)
at com.dianping.pigeon.remoting.provider.process.filter.GatewayProcessFilter.invoke(GatewayProcessFilter.java:51)
at com.dianping.pigeon.remoting.provider.process.ProviderProcessHandlerFactory$1.handle(ProviderProcessHandlerFactory.java:99)
at com.dianping.pigeon.remoting.provider.process.filter.SecurityFilter.invoke(SecurityFilter.java:31)
at com.dianping.pigeon.remoting.provider.process.filter.SecurityFilter.invoke(SecurityFilter.java:15)
at com.dianping.pigeon.remoting.provider.process.ProviderProcessHandlerFactory$1.handle(ProviderProcessHandlerFactory.java:99)
at com.dianping.pigeon.remoting.provider.process.filter.GenericProcessFilter.invoke(GenericProcessFilter.java:41)
at com.dianping.pigeon.remoting.provider.process.filter.GenericProcessFilter.invoke(GenericProcessFilter.java:24)
at com.dianping.pigeon.remoting.provider.process.ProviderProcessHandlerFactory$1.handle(ProviderProcessHandlerFactory.java:99)
at com.dianping.pigeon.remoting.provider.process.filter.ExceptionProcessFilter.invoke(ExceptionProcessFilter.java:44)
at com.dianping.pigeon.remoting.provider.process.filter.ExceptionProcessFilter.invoke(ExceptionProcessFilter.java:27)
at com.dianping.pigeon.remoting.provider.process.ProviderProcessHandlerFactory$1.handle(ProviderProcessHandlerFactory.java:99)
at com.dianping.pigeon.remoting.provider.process.filter.ContextTransferProcessFilter.invoke(ContextTransferProcessFilter.java:43)
at com.dianping.pigeon.remoting.provider.process.filter.ContextTransferProcessFilter.invoke(ContextTransferProcessFilter.java:28)
at com.dianping.pigeon.remoting.provider.process.ProviderProcessHandlerFactory$1.handle(ProviderProcessHandlerFactory.java:99)
at com.dianping.pigeon.remoting.provider.process.filter.WriteResponseProcessFilter.invoke(WriteResponseProcessFilter.java:36)
at com.dianping.pigeon.remoting.provider.process.filter.WriteResponseProcessFilter.invoke(WriteResponseProcessFilter.java:26)
at com.dianping.pigeon.remoting.provider.process.ProviderProcessHandlerFactory$1.handle(ProviderProcessHandlerFactory.java:99)
at com.dianping.pigeon.remoting.provider.process.filter.TrafficRecordFilter.invoke(TrafficRecordFilter.java:34)
at com.dianping.pigeon.remoting.provider.process.filter.TrafficRecordFilter.invoke(TrafficRecordFilter.java:14)
at com.dianping.pigeon.remoting.provider.process.ProviderProcessHandlerFactory$1.handle(ProviderProcessHandlerFactory.java:99)
at com.dianping.pigeon.remoting.provider.process.filter.InnerTracerFilter.invoke(InnerTracerFilter.java:29)
at com.dianping.pigeon.remoting.provider.process.filter.InnerTracerFilter.invoke(InnerTracerFilter.java:15)
at com.dianping.pigeon.remoting.provider.process.ProviderProcessHandlerFactory$1.handle(ProviderProcessHandlerFactory.java:99)
at com.dianping.pigeon.remoting.provider.process.filter.MonitorProcessFilter.invoke(MonitorProcessFilter.java:101)
at com.dianping.pigeon.remoting.provider.process.filter.MonitorProcessFilter.invoke(MonitorProcessFilter.java:35)
at com.dianping.pigeon.remoting.provider.process.ProviderProcessHandlerFactory$1.handle(ProviderProcessHandlerFactory.java:99)
at com.dianping.pigeon.remoting.provider.process.filter.TracerFilter.invoke(TracerFilter.java:37)
at com.dianping.pigeon.remoting.provider.process.filter.TracerFilter.invoke(TracerFilter.java:15)
at com.dianping.pigeon.remoting.provider.process.ProviderProcessHandlerFactory$1.handle(ProviderProcessHandlerFactory.java:99)
at com.dianping.pigeon.remoting.provider.process.filter.ProviderContextFilter.invoke(ProviderContextFilter.java:28)
at com.dianping.pigeon.remoting.provider.process.filter.ProviderContextFilter.invoke(ProviderContextFilter.java:18)
at com.dianping.pigeon.remoting.provider.process.ProviderProcessHandlerFactory$1.handle(ProviderProcessHandlerFactory.java:99)
at com.dianping.pigeon.remoting.provider.process.threadpool.RequestThreadPoolProcessor$2.call(RequestThreadPoolProcessor.java:254)
at com.dianping.pigeon.remoting.provider.process.threadpool.RequestThreadPoolProcessor2.call(RequestThreadPoolProcessor.java:243)atjava.util.concurrent.FutureTask.run(FutureTask.java:266)atjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)atjava.util.concurrent.ThreadPoolExecutor2.call(RequestThreadPoolProcessor.java:243) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor2.call(RequestThreadPoolProcessor.java:243)atjava.util.concurrent.FutureTask.run(FutureTask.java:266)atjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)atjava.util.concurrent.ThreadPoolExecutorWorker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers: - <0x00000007ab2b6228> (a java.util.concurrent.ThreadPoolExecutor$Worker)
Pigeon 阻塞線程 ThreadDump 信息
priority:5 - threadId:0x00007fa599636000 - nativeId:0x2ec8 - nativeId (decimal):11976 - state:BLOCKED
stackTrace:
java.lang.Thread.State: BLOCKED (on object monitor)
at deps.redis.clients.util.RedisInputStream.ensureFill(RedisInputStream.java:207)
at deps.redis.clients.util.RedisInputStream.readByte(RedisInputStream.java:47)
at deps.redis.clients.jedis.Protocol.process(Protocol.java:204)
at deps.redis.clients.jedis.Protocol.read(Protocol.java:271)
at deps.redis.clients.jedis.Connection.readProtocolWithCheckingBroken(Connection.java:305)
at deps.redis.clients.jedis.Connection.getBinaryBulkReply(Connection.java:236)
at deps.redis.clients.jedis.BinaryJedis.hget(BinaryJedis.java:788)
at deps.redis.clients.jedis.BinaryJedisCluster$13.execute(BinaryJedisCluster.java:202)
at deps.redis.clients.jedis.BinaryJedisCluster$13.execute(BinaryJedisCluster.java:199)
at deps.redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:64)
at deps.redis.clients.jedis.JedisClusterCommand.runBinary(JedisClusterCommand.java:41)
at deps.redis.clients.jedis.BinaryJedisCluster.hget(BinaryJedisCluster.java:199)
at com.dianping.squirrel.client.impl.redis.RedisStoreClientImpl41.excute(RedisStoreClientImpl.java:1022)atcom.dianping.squirrel.client.impl.AbstractStoreClient41.excute(RedisStoreClientImpl.java:1022) at com.dianping.squirrel.client.impl.AbstractStoreClient41.excute(RedisStoreClientImpl.java:1022)atcom.dianping.squirrel.client.impl.AbstractStoreClientMonitorCommand.run(AbstractStoreClient.java:1066)
at com.dianping.squirrel.client.impl.redis.RedisStoreClientImpl.hget(RedisStoreClientImpl.java:1018)
at com.dianping.poi.mainflow.core.redisdao.TimestampRedisDao.getUpdateTime(TimestampRedisDao.java:18)
at com.dianping.poi.mainflow.common.RefreshableCache.checkUpdate(RefreshableCache.java:99)
- locked?<0x00000007623197a0>?(a com.dianping.poi.mainflow.core.cache.FlowTemplateRuleCache)
at com.dianping.poi.mainflow.common.RefreshableCache.getValue(RefreshableCache.java:53)
at com.dianping.poi.mainflow.common.RefreshableCache.getValue(RefreshableCache.java:43)
at com.dianping.poi.mainflow.core.business.WorkflowBusiness.getTemplateRule(WorkflowBusiness.java:160)
at com.dianping.poi.mainflow.core.business.WorkflowBusiness.setTemplateByWorkflow(WorkflowBusiness.java:134)
at com.dianping.poi.mainflow.core.business.WorkflowBusiness.createWorkflow(WorkflowBusiness.java:253)
at com.dianping.poi.mainflow.core.business.WorkflowBusiness.createWorkflow(WorkflowBusiness.java:183)
at com.dianping.poi.mainflow.core.business.WorkflowBusiness.createWorkflow(WorkflowBusiness.java:176)
at com.dianping.poi.mainflow.core.business.WorkflowBusinessFastClassBySpringCGLIBFastClassBySpringCGLIBFastClassBySpringCGLIBf3bb47a4.invoke()
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
at org.springframework.aop.framework.CglibAopProxyCglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:736)atorg.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)atorg.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)atorg.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)atorg.springframework.aop.framework.CglibAopProxyCglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:736) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at org.springframework.aop.framework.CglibAopProxyCglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:736)atorg.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)atorg.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)atorg.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)atorg.springframework.aop.framework.CglibAopProxyDynamicAdvisedInterceptor.intercept(CglibAopProxy.java:671)
at com.dianping.poi.mainflow.core.business.WorkflowBusinessEnhancerBySpringCGLIBEnhancerBySpringCGLIBEnhancerBySpringCGLIBe721a2aa.createWorkflow()
at com.dianping.poi.mainflow.update.biz.business.MainUpdateFlowBusiness.update(MainUpdateFlowBusiness.java:220)
at com.dianping.poi.mainflow.update.biz.business.MainUpdateFlowBusiness.update(MainUpdateFlowBusiness.java:179)
at com.dianping.poi.mainflow.feedback.biz.serviceImpl.FeedbackForPoiServiceImpl.dpFeedbackBase(FeedbackForPoiServiceImpl.java:1233)
at com.dianping.poi.mainflow.feedback.biz.serviceImpl.FeedbackForPoiServiceImpl.dpTerminalFeedback(FeedbackForPoiServiceImpl.java:486)
at sun.reflect.GeneratedMethodAccessor144.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at com.dianping.pigeon.remoting.provider.service.method.ServiceMethod.invoke(ServiceMethod.java:193)
at com.dianping.pigeon.remoting.provider.process.filter.BusinessProcessFilter.invoke(BusinessProcessFilter.java:79)
at com.dianping.pigeon.remoting.provider.process.filter.BusinessProcessFilter.invoke(BusinessProcessFilter.java:34)
at com.dianping.pigeon.remoting.provider.process.ProviderProcessHandlerFactory$1.handle(ProviderProcessHandlerFactory.java:99)
at com.dianping.pigeon.remoting.provider.process.filter.GatewayProcessFilter.invoke(GatewayProcessFilter.java:242)
at com.dianping.pigeon.remoting.provider.process.filter.GatewayProcessFilter.invoke(GatewayProcessFilter.java:51)
at com.dianping.pigeon.remoting.provider.process.ProviderProcessHandlerFactory$1.handle(ProviderProcessHandlerFactory.java:99)
at com.dianping.pigeon.remoting.provider.process.filter.SecurityFilter.invoke(SecurityFilter.java:31)
at com.dianping.pigeon.remoting.provider.process.filter.SecurityFilter.invoke(SecurityFilter.java:15)
at com.dianping.pigeon.remoting.provider.process.ProviderProcessHandlerFactory$1.handle(ProviderProcessHandlerFactory.java:99)
at com.dianping.pigeon.remoting.provider.process.filter.GenericProcessFilter.invoke(GenericProcessFilter.java:41)
at com.dianping.pigeon.remoting.provider.process.filter.GenericProcessFilter.invoke(GenericProcessFilter.java:24)
at com.dianping.pigeon.remoting.provider.process.ProviderProcessHandlerFactory$1.handle(ProviderProcessHandlerFactory.java:99)
at com.dianping.pigeon.remoting.provider.process.filter.ExceptionProcessFilter.invoke(ExceptionProcessFilter.java:44)
at com.dianping.pigeon.remoting.provider.process.filter.ExceptionProcessFilter.invoke(ExceptionProcessFilter.java:27)
at com.dianping.pigeon.remoting.provider.process.ProviderProcessHandlerFactory$1.handle(ProviderProcessHandlerFactory.java:99)
at com.dianping.pigeon.remoting.provider.process.filter.ContextTransferProcessFilter.invoke(ContextTransferProcessFilter.java:43)
at com.dianping.pigeon.remoting.provider.process.filter.ContextTransferProcessFilter.invoke(ContextTransferProcessFilter.java:28)
at com.dianping.pigeon.remoting.provider.process.ProviderProcessHandlerFactory$1.handle(ProviderProcessHandlerFactory.java:99)
at com.dianping.pigeon.remoting.provider.process.filter.WriteResponseProcessFilter.invoke(WriteResponseProcessFilter.java:36)
at com.dianping.pigeon.remoting.provider.process.filter.WriteResponseProcessFilter.invoke(WriteResponseProcessFilter.java:26)
at com.dianping.pigeon.remoting.provider.process.ProviderProcessHandlerFactory$1.handle(ProviderProcessHandlerFactory.java:99)
at com.dianping.pigeon.remoting.provider.process.filter.TrafficRecordFilter.invoke(TrafficRecordFilter.java:34)
at com.dianping.pigeon.remoting.provider.process.filter.TrafficRecordFilter.invoke(TrafficRecordFilter.java:14)
at com.dianping.pigeon.remoting.provider.process.ProviderProcessHandlerFactory$1.handle(ProviderProcessHandlerFactory.java:99)
at com.dianping.pigeon.remoting.provider.process.filter.InnerTracerFilter.invoke(InnerTracerFilter.java:29)
at com.dianping.pigeon.remoting.provider.process.filter.InnerTracerFilter.invoke(InnerTracerFilter.java:15)
at com.dianping.pigeon.remoting.provider.process.ProviderProcessHandlerFactory$1.handle(ProviderProcessHandlerFactory.java:99)
at com.dianping.pigeon.remoting.provider.process.filter.MonitorProcessFilter.invoke(MonitorProcessFilter.java:101)
at com.dianping.pigeon.remoting.provider.process.filter.MonitorProcessFilter.invoke(MonitorProcessFilter.java:35)
at com.dianping.pigeon.remoting.provider.process.ProviderProcessHandlerFactory$1.handle(ProviderProcessHandlerFactory.java:99)
at com.dianping.pigeon.remoting.provider.process.filter.TracerFilter.invoke(TracerFilter.java:37)
at com.dianping.pigeon.remoting.provider.process.filter.TracerFilter.invoke(TracerFilter.java:15)
at com.dianping.pigeon.remoting.provider.process.ProviderProcessHandlerFactory$1.handle(ProviderProcessHandlerFactory.java:99)
at com.dianping.pigeon.remoting.provider.process.filter.ProviderContextFilter.invoke(ProviderContextFilter.java:28)
at com.dianping.pigeon.remoting.provider.process.filter.ProviderContextFilter.invoke(ProviderContextFilter.java:18)
at com.dianping.pigeon.remoting.provider.process.ProviderProcessHandlerFactory$1.handle(ProviderProcessHandlerFactory.java:99)
at com.dianping.pigeon.remoting.provider.process.threadpool.RequestThreadPoolProcessor$2.call(RequestThreadPoolProcessor.java:254)
at com.dianping.pigeon.remoting.provider.process.threadpool.RequestThreadPoolProcessor2.call(RequestThreadPoolProcessor.java:243)atjava.util.concurrent.FutureTask.run(FutureTask.java:266)atjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)atjava.util.concurrent.ThreadPoolExecutor2.call(RequestThreadPoolProcessor.java:243) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor2.call(RequestThreadPoolProcessor.java:243)atjava.util.concurrent.FutureTask.run(FutureTask.java:266)atjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)atjava.util.concurrent.ThreadPoolExecutorWorker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
-?<0x0000000735ab3bc0>?(a java.util.concurrent.ThreadPoolExecutor$Worker)
問題說明
Arthas 征文活動火熱進行中
Arthas 官方正在舉行征文活動,如果你有:
- 使用 Arthas 排查過的問題
- 對 Arthas 進行源碼解讀
- 對 Arthas 提出建議
- 不限,其它與 Arthas 有關的內容
歡迎參加征文活動,還有獎品拿哦~點擊投稿
“阿里巴巴云原生關注微服務、Serverless、容器、Service Mesh 等技術領域、聚焦云原生流行技術趨勢、云原生大規模的落地實踐,做最懂云原生開發者的公眾號。”
總結
以上是生活随笔為你收集整理的应用系统瓶颈排查和分析的思考-Arthas 实战的全部內容,希望文章能夠幫你解決所遇到的問題。
- 上一篇: 从零入门 Serverless | Se
- 下一篇: Fluid — 云原生环境下的高效“数据