日韩av黄I国产麻豆传媒I国产91av视频在线观看I日韩一区二区三区在线看I美女国产在线I麻豆视频国产在线观看I成人黄色短片

歡迎訪問 生活随笔!

生活随笔

當前位置: 首頁 > 编程资源 > 编程问答 >内容正文

编程问答

马士兵 jvm调优

發布時間:2023/12/15 编程问答 43 豆豆
生活随笔 收集整理的這篇文章主要介紹了 马士兵 jvm调优 小編覺得挺不錯的,現在分享給大家,幫大家做個參考.

GC和GC Tuning

作者:馬士兵教育 http://mashibing.com

GC的基礎知識

1.什么是垃圾

C語言申請內存:malloc free

C++: new delete

c/C++ 手動回收內存

Java: new ?

自動內存回收,編程上簡單,系統不容易出錯,手動釋放內存,容易出兩種類型的問題:

  • 忘記回收
  • 多次回收
  • 沒有任何引用指向的一個對象或者多個對象(循環引用)

    2.如何定位垃圾

  • 引用計數(ReferenceCount)
  • 根可達算法(RootSearching)
  • 3.常見的垃圾回收算法

  • 標記清除(mark sweep) - 位置不連續 產生碎片 效率偏低(兩遍掃描)
  • 拷貝算法 (copying) - 沒有碎片,浪費空間
  • 標記壓縮(mark compact) - 沒有碎片,效率偏低(兩遍掃描,指針需要調整)
  • 4.JVM內存分代模型(用于分代垃圾回收算法)

  • 部分垃圾回收器使用的模型

    除Epsilon ZGC Shenandoah之外的GC都是使用邏輯分代模型

    G1是邏輯分代,物理不分代

    除此之外不僅邏輯分代,而且物理分代

  • 新生代 + 老年代 + 永久代(1.7)Perm Generation/ 元數據區(1.8) Metaspace

  • 永久代 元數據 - Class
  • 永久代必須指定大小限制 ,元數據可以設置,也可以不設置,無上限(受限于物理內存)
  • 字符串常量 1.7 - 永久代,1.8 - 堆
  • MethodArea邏輯概念 - 永久代、元數據
  • 新生代 = Eden + 2個suvivor區

  • YGC回收之后,大多數的對象會被回收,活著的進入s0
  • 再次YGC,活著的對象eden + s0 -> s1
  • 再次YGC,eden + s1 -> s0
  • 年齡足夠 -> 老年代 (15 CMS 6)
  • s區裝不下 -> 老年代
  • 老年代

  • 頑固分子
  • 老年代滿了FGC Full GC
  • GC Tuning (Generation)

  • 盡量減少FGC
  • MinorGC = YGC
  • MajorGC = FGC
  • 對象分配過程圖
    [外鏈圖片轉存失敗,源站可能有防盜鏈機制,建議將圖片保存下來直接上傳(img-suj2fBed-1597918507248)(對象分配過程詳解.png)]

  • 動態年齡:(不重要)
    https://www.jianshu.com/p/989d3b06a49d

  • 分配擔保:(不重要)
    YGC期間 survivor區空間不夠了 空間擔保直接進入老年代
    參考:https://cloud.tencent.com/developer/article/1082730

  • 5.常見的垃圾回收器

    [外鏈圖片轉存失敗,源站可能有防盜鏈機制,建議將圖片保存下來直接上傳(img-0Pl9cxAn-1597918507250)(常用垃圾回收器.png)]

  • 垃圾回收器的發展路線,是隨著內存越來越大的過程而演進
    從分代算法演化到不分代算法
    Serial算法 幾十兆
    Parallel算法 幾個G
    CMS 幾十個G - 承上啟下,開始并發回收 -
    .- 三色標記 -
  • JDK誕生 Serial追隨 提高效率,誕生了PS,為了配合CMS,誕生了PN,CMS是1.4版本后期引入,CMS是里程碑式的GC,它開啟了并發回收的過程,但是CMS毛病較多,因此目前任何一個JDK版本默認是CMS
    并發垃圾回收是因為無法忍受STW
  • Serial 年輕代 串行回收
  • PS 年輕代 并行回收
  • ParNew 年輕代 配合CMS的并行回收
  • SerialOld
  • ParallelOld
  • ConcurrentMarkSweep 老年代 并發的, 垃圾回收和應用程序同時運行,降低STW的時間(200ms)
    CMS問題比較多,所以現在沒有一個版本默認是CMS,只能手工指定
    CMS既然是MarkSweep,就一定會有碎片化的問題,碎片到達一定程度,CMS的老年代分配對象分配不下的時候,使用SerialOld 進行老年代回收
    想象一下:
    PS + PO -> 加內存 換垃圾回收器 -> PN + CMS + SerialOld(幾個小時 - 幾天的STW)
    幾十個G的內存,單線程回收 -> G1 + FGC 幾十個G -> 上T內存的服務器 ZGC
    算法:三色標記 + Incremental Update
  • G1(200ms - 10ms)
    算法:三色標記 + SATB
  • ZGC (10ms - 1ms) PK C++
    算法:ColoredPointers + LoadBarrier
  • Shenandoah
    算法:ColoredPointers + WriteBarrier
  • Eplison
  • PS 和 PN區別的延伸閱讀:
    ?https://docs.oracle.com/en/java/javase/13/gctuning/ergonomics.html#GUID-3D0BB91E-9BFF-4EBB-B523-14493A860E73
  • 垃圾收集器跟內存大小的關系
  • Serial 幾十兆
  • PS 上百兆 - 幾個G
  • CMS - 20G
  • G1 - 上百G
  • ZGC - 4T - 16T(JDK13)
  • 1.8默認的垃圾回收:PS + ParallelOld

    常見垃圾回收器組合參數設定:(1.8)

    • -XX:+UseSerialGC = Serial New (DefNew) + Serial Old

      • 小型程序。默認情況下不會是這種選項,HotSpot會根據計算及配置和JDK版本自動選擇收集器
    • -XX:+UseParNewGC = ParNew + SerialOld

      • 這個組合已經很少用(在某些版本中已經廢棄)
      • https://stackoverflow.com/questions/34962257/why-remove-support-for-parnewserialold-anddefnewcms-in-the-future
    • -XX:+UseConc(urrent)MarkSweepGC = ParNew + CMS + Serial Old

    • -XX:+UseParallelGC = Parallel Scavenge + Parallel Old (1.8默認) 【PS + SerialOld】

    • -XX:+UseParallelOldGC = Parallel Scavenge + Parallel Old

    • -XX:+UseG1GC = G1

    • Linux中沒找到默認GC的查看方法,而windows中會打印UseParallelGC

      • java +XX:+PrintCommandLineFlags -version
      • 通過GC的日志來分辨
    • Linux下1.8版本默認的垃圾回收器到底是什么?

      • 1.8.0_181 默認(看不出來)Copy MarkCompact
      • 1.8.0_222 默認 PS + PO

    JVM調優第一步,了解JVM常用命令行參數

    • JVM的命令行參數參考:https://docs.oracle.com/javase/8/docs/technotes/tools/unix/java.html

    • HotSpot參數分類

      標準: - 開頭,所有的HotSpot都支持

      非標準:-X 開頭,特定版本HotSpot支持特定命令

      不穩定:-XX 開頭,下個版本可能取消

      java -version

      java -X

      java -XX:+PrintFlagsWithComments //只有debug版本能用

      試驗用程序:

      <span style="color:#000000"><span style="background-color:#282c34"><code class="language-java"><span style="color:#c678dd">import</span> java<span style="color:#999999">.</span>util<span style="color:#999999">.</span>List<span style="color:#999999">;</span> <span style="color:#c678dd">import</span> java<span style="color:#999999">.</span>util<span style="color:#999999">.</span>LinkedList<span style="color:#999999">;</span><span style="color:#c678dd">public</span> <span style="color:#c678dd">class</span> HelloGC <span style="color:#999999">{</span><span style="color:#c678dd">public</span> <span style="color:#c678dd">static</span> <span style="color:#c678dd">void</span> <span style="color:#61aeee">main</span><span style="color:#999999">(</span>String<span style="color:#999999">[</span><span style="color:#999999">]</span> args<span style="color:#999999">)</span> <span style="color:#999999">{</span>System<span style="color:#999999">.</span>out<span style="color:#999999">.</span><span style="color:#61aeee">println</span><span style="color:#999999">(</span><span style="color:#669900">"HelloGC!"</span><span style="color:#999999">)</span><span style="color:#999999">;</span>List list <span style="color:#669900">=</span> <span style="color:#c678dd">new</span> LinkedList<span style="color:#999999">(</span><span style="color:#999999">)</span><span style="color:#999999">;</span><span style="color:#c678dd">for</span><span style="color:#999999">(</span><span style="color:#999999">;</span><span style="color:#999999">;</span><span style="color:#999999">)</span> <span style="color:#999999">{</span><span style="color:#c678dd">byte</span><span style="color:#999999">[</span><span style="color:#999999">]</span> b <span style="color:#669900">=</span> <span style="color:#c678dd">new</span> byte<span style="color:#999999">[</span><span style="color:#98c379">1024</span><span style="color:#669900">*</span><span style="color:#98c379">1024</span><span style="color:#999999">]</span><span style="color:#999999">;</span>list<span style="color:#999999">.</span><span style="color:#61aeee">add</span><span style="color:#999999">(</span>b<span style="color:#999999">)</span><span style="color:#999999">;</span><span style="color:#999999">}</span><span style="color:#999999">}</span> </code></span></span>
      • 1
      • 2
      • 3
      • 4
      • 5
      • 6
      • 7
      • 8
      • 9
      • 10
      • 11
      • 12

    }

    <span style="color:#000000"><span style="background-color:#282c34"><code>1. 區分概念:內存泄漏memory leak,內存溢出out of memory 2. java -XX:+PrintCommandLineFlags HelloGC 3. java -Xmn10M -Xms40M -Xmx60M -XX:+PrintCommandLineFlags -XX:+PrintGC HelloGCPrintGCDetails PrintGCTimeStamps PrintGCCauses 4. java -XX:+UseConcMarkSweepGC -XX:+PrintCommandLineFlags HelloGC 5. java -XX:+PrintFlagsInitial 默認參數值 6. java -XX:+PrintFlagsFinal 最終參數值 7. java -XX:+PrintFlagsFinal | grep xxx 找到對應的參數 8. java -XX:+PrintFlagsFinal -version |grep GC 9. java -XX:+PrintFlagsFinal -version | wc -l 共728個參數### PS GC日志詳解每種垃圾回收器的日志格式是不同的!PS日志格式[外鏈圖片轉存失敗,源站可能有防盜鏈機制,建議將圖片保存下來直接上傳(img-7Rc34d7R-1597918507250)(./GC日志詳解.png)]heap dump部分:```java eden space 5632K, 94% used [0x00000000ff980000,0x00000000ffeb3e28,0x00000000fff00000)后面的內存地址指的是,起始地址,使用空間結束地址,整體空間結束地址 </code></span></span>
    • 1
    • 2
    • 3
    • 4
    • 5
    • 6
    • 7
    • 8
    • 9
    • 10
    • 11
    • 12
    • 13
    • 14
    • 15
    • 16
    • 17
    • 18
    • 19
    • 20
    • 21
    • 22
    • 23
    • 24
    • 25
    • 26
    • 27
    • 28

    [外鏈圖片轉存失敗,源站可能有防盜鏈機制,建議將圖片保存下來直接上傳(img-oFKjgwrX-1597918507252)(GCHeapDump.png)]

    total = eden + 1個survivor

    調優前的基礎概念:

  • 吞吐量:用戶代碼時間 /(用戶代碼執行時間 + 垃圾回收時間)
  • 響應時間:STW越短,響應時間越好
  • 所謂調優,首先確定,追求啥?吞吐量優先,還是響應時間優先?還是在滿足一定的響應時間的情況下,要求達到多大的吞吐量…

    問題:

    科學計算,吞吐量。數據挖掘,thrput。吞吐量優先的一般:(PS + PO)

    響應時間:網站 GUI API (1.8 G1)

    什么是調優?

  • 根據需求進行JVM規劃和預調優
  • 優化運行JVM運行環境(慢,卡頓)
  • 解決JVM運行過程中出現的各種問題(OOM)
  • 調優,從規劃開始

    • 調優,從業務場景開始,沒有業務場景的調優都是耍流氓

    • 無監控(壓力測試,能看到結果),不調優

    • 步驟:

    • 熟悉業務場景(沒有最好的垃圾回收器,只有最合適的垃圾回收器)
    • 響應時間、停頓時間 [CMS G1 ZGC] (需要給用戶作響應)
    • 吞吐量 = 用戶時間 /( 用戶時間 + GC時間) [PS]
    • 選擇回收器組合
    • 計算內存需求(經驗值 1.5G 16G)
    • 選定CPU(越高越好)
    • 設定年代大小、升級年齡
    • 設定日志參數
    • -Xloggc:/opt/xxx/logs/xxx-xxx-gc-%t.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=20M -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCCause
    • 或者每天產生一個日志文件
    • 觀察日志情況
    • 案例1:垂直電商,最高每日百萬訂單,處理訂單系統需要什么樣的服務器配置?

      這個問題比較業余,因為很多不同的服務器配置都能支撐(1.5G 16G)

      1小時360000集中時間段, 100個訂單/秒,(找一小時內的高峰期,1000訂單/秒)

      經驗值,

      非要計算:一個訂單產生需要多少內存?512K * 1000 500M內存

      專業一點兒問法:要求響應時間100ms

      壓測!

    • 案例2:12306遭遇春節大規模搶票應該如何支撐?

      12306應該是中國并發量最大的秒殺網站:

      號稱并發量100W最高

      CDN -> LVS -> NGINX -> 業務系統 -> 每臺機器1W并發(10K問題) 100臺機器

      普通電商訂單 -> 下單 ->訂單系統(IO)減庫存 ->等待用戶付款

      12306的一種可能的模型: 下單 -> 減庫存 和 訂單(redis kafka) 同時異步進行 ->等付款

      減庫存最后還會把壓力壓到一臺服務器

      可以做分布式本地庫存 + 單獨服務器做庫存均衡

      大流量的處理方法:分而治之

    • 怎么得到一個事務會消耗多少內存?

    • 弄臺機器,看能承受多少TPS?是不是達到目標?擴容或調優,讓它達到

    • 用壓測來確定

    優化環境

  • 有一個50萬PV的資料類網站(從磁盤提取文檔到內存)原服務器32位,1.5G
    的堆,用戶反饋網站比較緩慢,因此公司決定升級,新的服務器為64位,16G
    的堆內存,結果用戶反饋卡頓十分嚴重,反而比以前效率更低了
  • 為什么原網站慢?
    很多用戶瀏覽數據,很多數據load到內存,內存不足,頻繁GC,STW長,響應時間變慢
  • 為什么會更卡頓?
    內存越大,FGC時間越長
  • 咋辦?
    PS -> PN + CMS 或者 G1
  • 系統CPU經常100%,如何調優?(面試高頻)
    CPU100%那么一定有線程在占用系統資源,
  • 找出哪個進程cpu高(top)
  • 該進程中的哪個線程cpu高(top -Hp)
  • 導出該線程的堆棧 (jstack)
  • 查找哪個方法(棧幀)消耗時間 (jstack)
  • 工作線程占比高 | 垃圾回收線程占比高
  • 系統內存飆高,如何查找問題?(面試高頻)
  • 導出堆內存 (jmap)
  • 分析 (jhat jvisualvm mat jprofiler … )
  • 如何監控JVM
  • jstat jvisualvm jprofiler arthas top…
  • 解決JVM運行中的問題

    一個案例理解常用工具

  • 測試代碼:

    <span style="color:#000000"><span style="background-color:#282c34"><code class="language-java"><span style="color:#c678dd">package</span> com<span style="color:#999999">.</span>mashibing<span style="color:#999999">.</span>jvm<span style="color:#999999">.</span>gc<span style="color:#999999">;</span><span style="color:#c678dd">import</span> java<span style="color:#999999">.</span>math<span style="color:#999999">.</span>BigDecimal<span style="color:#999999">;</span> <span style="color:#c678dd">import</span> java<span style="color:#999999">.</span>util<span style="color:#999999">.</span>ArrayList<span style="color:#999999">;</span> <span style="color:#c678dd">import</span> java<span style="color:#999999">.</span>util<span style="color:#999999">.</span>Date<span style="color:#999999">;</span> <span style="color:#c678dd">import</span> java<span style="color:#999999">.</span>util<span style="color:#999999">.</span>List<span style="color:#999999">;</span> <span style="color:#c678dd">import</span> java<span style="color:#999999">.</span>util<span style="color:#999999">.</span>concurrent<span style="color:#999999">.</span>ScheduledThreadPoolExecutor<span style="color:#999999">;</span> <span style="color:#c678dd">import</span> java<span style="color:#999999">.</span>util<span style="color:#999999">.</span>concurrent<span style="color:#999999">.</span>ThreadPoolExecutor<span style="color:#999999">;</span> <span style="color:#c678dd">import</span> java<span style="color:#999999">.</span>util<span style="color:#999999">.</span>concurrent<span style="color:#999999">.</span>TimeUnit<span style="color:#999999">;</span><span style="color:#5c6370">/*** 從數據庫中讀取信用數據,套用模型,并把結果進行記錄和傳輸*/</span><span style="color:#c678dd">public</span> <span style="color:#c678dd">class</span> T15_FullGC_Problem01 <span style="color:#999999">{</span><span style="color:#c678dd">private</span> <span style="color:#c678dd">static</span> <span style="color:#c678dd">class</span> CardInfo <span style="color:#999999">{</span>BigDecimal price <span style="color:#669900">=</span> <span style="color:#c678dd">new</span> BigDecimal<span style="color:#999999">(</span><span style="color:#98c379">0.0</span><span style="color:#999999">)</span><span style="color:#999999">;</span>String name <span style="color:#669900">=</span> <span style="color:#669900">"張三"</span><span style="color:#999999">;</span><span style="color:#c678dd">int</span> age <span style="color:#669900">=</span> <span style="color:#98c379">5</span><span style="color:#999999">;</span>Date birthdate <span style="color:#669900">=</span> <span style="color:#c678dd">new</span> Date<span style="color:#999999">(</span><span style="color:#999999">)</span><span style="color:#999999">;</span><span style="color:#c678dd">public</span> <span style="color:#c678dd">void</span> <span style="color:#61aeee">m</span><span style="color:#999999">(</span><span style="color:#999999">)</span> <span style="color:#999999">{</span><span style="color:#999999">}</span><span style="color:#999999">}</span><span style="color:#c678dd">private</span> <span style="color:#c678dd">static</span> ScheduledThreadPoolExecutor executor <span style="color:#669900">=</span> <span style="color:#c678dd">new</span> ScheduledThreadPoolExecutor<span style="color:#999999">(</span><span style="color:#98c379">50</span><span style="color:#999999">,</span><span style="color:#c678dd">new</span> ThreadPoolExecutor<span style="color:#999999">.</span>DiscardOldestPolicy<span style="color:#999999">(</span><span style="color:#999999">)</span><span style="color:#999999">)</span><span style="color:#999999">;</span><span style="color:#c678dd">public</span> <span style="color:#c678dd">static</span> <span style="color:#c678dd">void</span> <span style="color:#61aeee">main</span><span style="color:#999999">(</span>String<span style="color:#999999">[</span><span style="color:#999999">]</span> args<span style="color:#999999">)</span> <span style="color:#c678dd">throws</span> Exception <span style="color:#999999">{</span>executor<span style="color:#999999">.</span><span style="color:#61aeee">setMaximumPoolSize</span><span style="color:#999999">(</span><span style="color:#98c379">50</span><span style="color:#999999">)</span><span style="color:#999999">;</span><span style="color:#c678dd">for</span> <span style="color:#999999">(</span><span style="color:#999999">;</span><span style="color:#999999">;</span><span style="color:#999999">)</span><span style="color:#999999">{</span><span style="color:#61aeee">modelFit</span><span style="color:#999999">(</span><span style="color:#999999">)</span><span style="color:#999999">;</span>Thread<span style="color:#999999">.</span><span style="color:#61aeee">sleep</span><span style="color:#999999">(</span><span style="color:#98c379">100</span><span style="color:#999999">)</span><span style="color:#999999">;</span><span style="color:#999999">}</span><span style="color:#999999">}</span><span style="color:#c678dd">private</span> <span style="color:#c678dd">static</span> <span style="color:#c678dd">void</span> <span style="color:#61aeee">modelFit</span><span style="color:#999999">(</span><span style="color:#999999">)</span><span style="color:#999999">{</span>List<span style="color:#61aeee"><span style="color:#999999"><</span>CardInfo<span style="color:#999999">></span></span> taskList <span style="color:#669900">=</span> <span style="color:#61aeee">getAllCardInfo</span><span style="color:#999999">(</span><span style="color:#999999">)</span><span style="color:#999999">;</span>taskList<span style="color:#999999">.</span><span style="color:#61aeee">forEach</span><span style="color:#999999">(</span>info <span style="color:#669900">-</span><span style="color:#669900">></span> <span style="color:#999999">{</span><span style="color:#5c6370">// do something</span>executor<span style="color:#999999">.</span><span style="color:#61aeee">scheduleWithFixedDelay</span><span style="color:#999999">(</span><span style="color:#999999">(</span><span style="color:#999999">)</span> <span style="color:#669900">-</span><span style="color:#669900">></span> <span style="color:#999999">{</span><span style="color:#5c6370">//do sth with info</span>info<span style="color:#999999">.</span><span style="color:#61aeee">m</span><span style="color:#999999">(</span><span style="color:#999999">)</span><span style="color:#999999">;</span><span style="color:#999999">}</span><span style="color:#999999">,</span> <span style="color:#98c379">2</span><span style="color:#999999">,</span> <span style="color:#98c379">3</span><span style="color:#999999">,</span> TimeUnit<span style="color:#999999">.</span>SECONDS<span style="color:#999999">)</span><span style="color:#999999">;</span><span style="color:#999999">}</span><span style="color:#999999">)</span><span style="color:#999999">;</span><span style="color:#999999">}</span><span style="color:#c678dd">private</span> <span style="color:#c678dd">static</span> List<span style="color:#61aeee"><span style="color:#999999"><</span>CardInfo<span style="color:#999999">></span></span> <span style="color:#61aeee">getAllCardInfo</span><span style="color:#999999">(</span><span style="color:#999999">)</span><span style="color:#999999">{</span>List<span style="color:#61aeee"><span style="color:#999999"><</span>CardInfo<span style="color:#999999">></span></span> taskList <span style="color:#669900">=</span> <span style="color:#c678dd">new</span> ArrayList<span style="color:#669900"><</span><span style="color:#669900">></span><span style="color:#999999">(</span><span style="color:#999999">)</span><span style="color:#999999">;</span><span style="color:#c678dd">for</span> <span style="color:#999999">(</span><span style="color:#c678dd">int</span> i <span style="color:#669900">=</span> <span style="color:#98c379">0</span><span style="color:#999999">;</span> i <span style="color:#669900"><</span> <span style="color:#98c379">100</span><span style="color:#999999">;</span> i<span style="color:#669900">++</span><span style="color:#999999">)</span> <span style="color:#999999">{</span>CardInfo ci <span style="color:#669900">=</span> <span style="color:#c678dd">new</span> CardInfo<span style="color:#999999">(</span><span style="color:#999999">)</span><span style="color:#999999">;</span>taskList<span style="color:#999999">.</span><span style="color:#61aeee">add</span><span style="color:#999999">(</span>ci<span style="color:#999999">)</span><span style="color:#999999">;</span><span style="color:#999999">}</span><span style="color:#c678dd">return</span> taskList<span style="color:#999999">;</span><span style="color:#999999">}</span> <span style="color:#999999">}</span></code></span></span>
    • 1
    • 2
    • 3
    • 4
    • 5
    • 6
    • 7
    • 8
    • 9
    • 10
    • 11
    • 12
    • 13
    • 14
    • 15
    • 16
    • 17
    • 18
    • 19
    • 20
    • 21
    • 22
    • 23
    • 24
    • 25
    • 26
    • 27
    • 28
    • 29
    • 30
    • 31
    • 32
    • 33
    • 34
    • 35
    • 36
    • 37
    • 38
    • 39
    • 40
    • 41
    • 42
    • 43
    • 44
    • 45
    • 46
    • 47
    • 48
    • 49
    • 50
    • 51
    • 52
    • 53
    • 54
    • 55
    • 56
    • 57
    • 58
    • 59
    • 60
    • 61
  • java -Xms200M -Xmx200M -XX:+PrintGC com.mashibing.jvm.gc.T15_FullGC_Problem01

  • 一般是運維團隊首先受到報警信息(CPU Memory)

  • top命令觀察到問題:內存不斷增長 CPU占用率居高不下

  • top -Hp 觀察進程中的線程,哪個線程CPU和內存占比高

  • jps定位具體java進程
    jstack 定位線程狀況,重點關注:WAITING BLOCKED
    eg.
    waiting on <0x0000000088ca3310> (a java.lang.Object)
    假如有一個進程中100個線程,很多線程都在waiting on ,一定要找到是哪個線程持有這把鎖
    怎么找?搜索jstack dump的信息,找 ,看哪個線程持有這把鎖RUNNABLE
    作業:1:寫一個死鎖程序,用jstack觀察 2 :寫一個程序,一個線程持有鎖不釋放,其他線程等待

  • 為什么阿里規范里規定,線程的名稱(尤其是線程池)都要寫有意義的名稱
    怎么樣自定義線程池里的線程名稱?(自定義ThreadFactory)

  • jinfo pid

  • jstat -gc 動態觀察gc情況 / 閱讀GC日志發現頻繁GC / arthas觀察 / jconsole/jvisualVM/ Jprofiler(最好用)
    jstat -gc 4655 500 : 每個500個毫秒打印GC的情況
    如果面試官問你是怎么定位OOM問題的?如果你回答用圖形界面(錯誤)
    1:已經上線的系統不用圖形界面用什么?(cmdline arthas)
    2:圖形界面到底用在什么地方?測試!測試的時候進行監控!(壓測觀察)

  • jmap - histo 4655 | head -20,查找有多少對象產生

  • jmap -dump:format=b,file=xxx pid :

    線上系統,內存特別大,jmap執行期間會對進程產生很大影響,甚至卡頓(電商不適合)
    1:設定了參數HeapDump,OOM的時候會自動產生堆轉儲文件(不是很專業,因為多有監控,內存增長就會報警)
    2:很多服務器備份(高可用),停掉這臺服務器對其他服務器不影響
    3:在線定位(一般小點兒公司用不到)

    4:在測試環境中壓測(產生類似內存增長問題,在堆還不是很大的時候進行轉儲)

  • java -Xms20M -Xmx20M -XX:+UseParallelGC -XX:+HeapDumpOnOutOfMemoryError com.mashibing.jvm.gc.T15_FullGC_Problem01

  • 使用MAT / jhat /jvisualvm 進行dump文件分析
    https://www.cnblogs.com/baihuitestsoftware/articles/6406271.html
    jhat -J-mx512M xxx.dump
    http://192.168.17.11:7000
    拉到最后:找到對應鏈接
    可以使用OQL查找特定問題對象

  • 找到代碼的問題

  • jconsole遠程連接

  • 程序啟動加入參數:

    <span style="color:#000000"><span style="background-color:#282c34"><code class="language-shell">java -Djava.rmi.server.hostname<span style="color:#669900">=</span>192.168.17.11 -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port<span style="color:#669900">=</span>11111 -Dcom.sun.management.jmxremote.authenticate<span style="color:#669900">=</span>false -Dcom.sun.management.jmxremote.ssl<span style="color:#669900">=</span>false XXX </code></span></span>
    • 1
  • 如果遭遇 Local host name unknown:XXX的錯誤,修改/etc/hosts文件,把XXX加入進去

    <span style="color:#000000"><span style="background-color:#282c34"><code class="language-java"><span style="color:#98c379">192.168</span><span style="color:#98c379">.17</span><span style="color:#98c379">.11</span> basic localhost localhost<span style="color:#999999">.</span>localdomain localhost4 localhost4<span style="color:#999999">.</span>localdomain4 <span style="color:#669900">:</span><span style="color:#669900">:</span><span style="color:#98c379">1</span> localhost localhost<span style="color:#999999">.</span>localdomain localhost6 localhost6<span style="color:#999999">.</span>localdomain6 </code></span></span>
    • 1
    • 2
  • 關閉linux防火墻(實戰中應該打開對應端口)

    <span style="color:#000000"><span style="background-color:#282c34"><code class="language-shell"><span style="color:#61aeee">service</span> iptables stop <span style="color:#61aeee">chkconfig</span> iptables off <span style="color:#5c6370">#永久關閉</span> </code></span></span>
    • 1
    • 2
  • windows上打開 jconsole遠程連接 192.168.17.11:11111

  • jvisualvm遠程連接

    https://www.cnblogs.com/liugh/p/7620336.html (簡單做法)

    jprofiler (收費)

    arthas在線排查工具

    • 為什么需要在線排查?
      在生產上我們經常會碰到一些不好排查的問題,例如線程安全問題,用最簡單的threaddump或者heapdump不好查到問題原因。為了排查這些問題,有時我們會臨時加一些日志,比如在一些關鍵的函數里打印出入參,然后重新打包發布,如果打了日志還是沒找到問題,繼續加日志,重新打包發布。對于上線流程復雜而且審核比較嚴的公司,從改代碼到上線需要層層的流轉,會大大影響問題排查的進度。
    • jvm觀察jvm信息
    • thread定位線程問題
    • dashboard 觀察系統情況
    • heapdump + jhat分析
    • jad反編譯
      動態代理生成類的問題定位
      第三方的類(觀察代碼)
      版本問題(確定自己最新提交的版本是不是被使用)
    • redefine 熱替換
      目前有些限制條件:只能改方法實現(方法已經運行完成),不能改方法名, 不能改屬性
      m() -> mm()
    • sc - search class
    • watch - watch method
    • 沒有包含的功能:jmap

    GC算法的基礎概念

    • Card Table
      由于做YGC時,需要掃描整個OLD區,效率非常低,所以JVM設計了CardTable, 如果一個OLD區CardTable中有對象指向Y區,就將它設為Dirty,下次掃描時,只需要掃描Dirty Card
      在結構上,Card Table用BitMap來實現

    CMS

    CMS的問題

  • Memory Fragmentation

    -XX:+UseCMSCompactAtFullCollection
    -XX:CMSFullGCsBeforeCompaction 默認為0 指的是經過多少次FGC才進行壓縮

  • Floating Garbage

    Concurrent Mode Failure
    產生:if the concurrent collector is unable to finish reclaiming the unreachable objects before the tenured generation fills up, or if an allocation cannot be satisfiedwith the available free space blocks in the tenured generation, then theapplication is paused and the collection is completed with all the applicationthreads stopped

    解決方案:降低觸發CMS的閾值

    PromotionFailed

    解決方案類似,保持老年代有足夠的空間

    –XX:CMSInitiatingOccupancyFraction 92% 可以降低這個值,讓CMS保持老年代足夠的空間

  • CMS日志分析

    執行命令:java -Xms20M -Xmx20M -XX:+PrintGCDetails -XX:+UseConcMarkSweepGC com.mashibing.jvm.gc.T15_FullGC_Problem01

    [GC (Allocation Failure) [ParNew: 6144K->640K(6144K), 0.0265885 secs] 6585K->2770K(19840K), 0.0268035 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]

    ParNew:年輕代收集器

    6144->640:收集前后的對比

    (6144):整個年輕代容量

    6585 -> 2770:整個堆的情況

    (19840):整個堆大小

    <span style="color:#000000"><span style="background-color:#282c34"><code class="language-java"><span style="color:#999999">[</span>GC <span style="color:#999999">(</span>CMS Initial Mark<span style="color:#999999">)</span> <span style="color:#999999">[</span><span style="color:#98c379">1</span> CMS<span style="color:#669900">-</span>initial<span style="color:#669900">-</span>mark<span style="color:#669900">:</span> <span style="color:#61aeee">8511K</span><span style="color:#999999">(</span><span style="color:#98c379">13696</span>K<span style="color:#999999">)</span><span style="color:#999999">]</span> <span style="color:#61aeee">9866K</span><span style="color:#999999">(</span><span style="color:#98c379">19840</span>K<span style="color:#999999">)</span><span style="color:#999999">,</span> <span style="color:#98c379">0.0040321</span> secs<span style="color:#999999">]</span> <span style="color:#999999">[</span>Times<span style="color:#669900">:</span> user<span style="color:#669900">=</span><span style="color:#98c379">0.01</span> sys<span style="color:#669900">=</span><span style="color:#98c379">0.00</span><span style="color:#999999">,</span> real<span style="color:#669900">=</span><span style="color:#98c379">0.00</span> secs<span style="color:#999999">]</span> <span style="color:#5c6370">//8511 (13696) : 老年代使用(最大)</span><span style="color:#5c6370">//9866 (19840) : 整個堆使用(最大)</span> <span style="color:#999999">[</span>CMS<span style="color:#669900">-</span>concurrent<span style="color:#669900">-</span>mark<span style="color:#669900">-</span>start<span style="color:#999999">]</span> <span style="color:#999999">[</span>CMS<span style="color:#669900">-</span>concurrent<span style="color:#669900">-</span>mark<span style="color:#669900">:</span> <span style="color:#98c379">0.018</span><span style="color:#669900">/</span><span style="color:#98c379">0.018</span> secs<span style="color:#999999">]</span> <span style="color:#999999">[</span>Times<span style="color:#669900">:</span> user<span style="color:#669900">=</span><span style="color:#98c379">0.01</span> sys<span style="color:#669900">=</span><span style="color:#98c379">0.00</span><span style="color:#999999">,</span> real<span style="color:#669900">=</span><span style="color:#98c379">0.02</span> secs<span style="color:#999999">]</span> <span style="color:#5c6370">//這里的時間意義不大,因為是并發執行</span> <span style="color:#999999">[</span>CMS<span style="color:#669900">-</span>concurrent<span style="color:#669900">-</span>preclean<span style="color:#669900">-</span>start<span style="color:#999999">]</span> <span style="color:#999999">[</span>CMS<span style="color:#669900">-</span>concurrent<span style="color:#669900">-</span>preclean<span style="color:#669900">:</span> <span style="color:#98c379">0.000</span><span style="color:#669900">/</span><span style="color:#98c379">0.000</span> secs<span style="color:#999999">]</span> <span style="color:#999999">[</span>Times<span style="color:#669900">:</span> user<span style="color:#669900">=</span><span style="color:#98c379">0.00</span> sys<span style="color:#669900">=</span><span style="color:#98c379">0.00</span><span style="color:#999999">,</span> real<span style="color:#669900">=</span><span style="color:#98c379">0.00</span> secs<span style="color:#999999">]</span> <span style="color:#5c6370">//標記Card為Dirty,也稱為Card Marking</span> <span style="color:#999999">[</span>GC <span style="color:#999999">(</span>CMS Final Remark<span style="color:#999999">)</span> <span style="color:#999999">[</span>YG occupancy<span style="color:#669900">:</span> <span style="color:#98c379">1597</span> K <span style="color:#999999">(</span><span style="color:#98c379">6144</span> K<span style="color:#999999">)</span><span style="color:#999999">]</span><span style="color:#999999">[</span>Rescan <span style="color:#999999">(</span>parallel<span style="color:#999999">)</span> <span style="color:#999999">,</span> <span style="color:#98c379">0.0008396</span> secs<span style="color:#999999">]</span><span style="color:#999999">[</span>weak refs processing<span style="color:#999999">,</span> <span style="color:#98c379">0.0000138</span> secs<span style="color:#999999">]</span><span style="color:#999999">[</span><span style="color:#c678dd">class</span> unloading<span style="color:#999999">,</span> <span style="color:#98c379">0.0005404</span> secs<span style="color:#999999">]</span><span style="color:#999999">[</span>scrub symbol table<span style="color:#999999">,</span> <span style="color:#98c379">0.0006169</span> secs<span style="color:#999999">]</span><span style="color:#999999">[</span>scrub string table<span style="color:#999999">,</span> <span style="color:#98c379">0.0004903</span> secs<span style="color:#999999">]</span><span style="color:#999999">[</span><span style="color:#98c379">1</span> CMS<span style="color:#669900">-</span>remark<span style="color:#669900">:</span> <span style="color:#61aeee">8511K</span><span style="color:#999999">(</span><span style="color:#98c379">13696</span>K<span style="color:#999999">)</span><span style="color:#999999">]</span> <span style="color:#61aeee">10108K</span><span style="color:#999999">(</span><span style="color:#98c379">19840</span>K<span style="color:#999999">)</span><span style="color:#999999">,</span> <span style="color:#98c379">0.0039567</span> secs<span style="color:#999999">]</span> <span style="color:#999999">[</span>Times<span style="color:#669900">:</span> user<span style="color:#669900">=</span><span style="color:#98c379">0.00</span> sys<span style="color:#669900">=</span><span style="color:#98c379">0.00</span><span style="color:#999999">,</span> real<span style="color:#669900">=</span><span style="color:#98c379">0.00</span> secs<span style="color:#999999">]</span> <span style="color:#5c6370">//STW階段,YG occupancy:年輕代占用及容量</span><span style="color:#5c6370">//[Rescan (parallel):STW下的存活對象標記</span><span style="color:#5c6370">//weak refs processing: 弱引用處理</span><span style="color:#5c6370">//class unloading: 卸載用不到的class</span><span style="color:#5c6370">//scrub symbol(string) table: </span><span style="color:#5c6370">//cleaning up symbol and string tables which hold class-level metadata and </span><span style="color:#5c6370">//internalized string respectively</span><span style="color:#5c6370">//CMS-remark: 8511K(13696K): 階段過后的老年代占用及容量</span><span style="color:#5c6370">//10108K(19840K): 階段過后的堆占用及容量</span><span style="color:#999999">[</span>CMS<span style="color:#669900">-</span>concurrent<span style="color:#669900">-</span>sweep<span style="color:#669900">-</span>start<span style="color:#999999">]</span> <span style="color:#999999">[</span>CMS<span style="color:#669900">-</span>concurrent<span style="color:#669900">-</span>sweep<span style="color:#669900">:</span> <span style="color:#98c379">0.005</span><span style="color:#669900">/</span><span style="color:#98c379">0.005</span> secs<span style="color:#999999">]</span> <span style="color:#999999">[</span>Times<span style="color:#669900">:</span> user<span style="color:#669900">=</span><span style="color:#98c379">0.00</span> sys<span style="color:#669900">=</span><span style="color:#98c379">0.00</span><span style="color:#999999">,</span> real<span style="color:#669900">=</span><span style="color:#98c379">0.01</span> secs<span style="color:#999999">]</span> <span style="color:#5c6370">//標記已經完成,進行并發清理</span> <span style="color:#999999">[</span>CMS<span style="color:#669900">-</span>concurrent<span style="color:#669900">-</span>reset<span style="color:#669900">-</span>start<span style="color:#999999">]</span> <span style="color:#999999">[</span>CMS<span style="color:#669900">-</span>concurrent<span style="color:#669900">-</span>reset<span style="color:#669900">:</span> <span style="color:#98c379">0.000</span><span style="color:#669900">/</span><span style="color:#98c379">0.000</span> secs<span style="color:#999999">]</span> <span style="color:#999999">[</span>Times<span style="color:#669900">:</span> user<span style="color:#669900">=</span><span style="color:#98c379">0.00</span> sys<span style="color:#669900">=</span><span style="color:#98c379">0.00</span><span style="color:#999999">,</span> real<span style="color:#669900">=</span><span style="color:#98c379">0.00</span> secs<span style="color:#999999">]</span><span style="color:#5c6370">//重置內部結構,為下次GC做準備</span> </code></span></span>
    • 1
    • 2
    • 3
    • 4
    • 5
    • 6
    • 7
    • 8
    • 9
    • 10
    • 11
    • 12
    • 13
    • 14
    • 15
    • 16
    • 17
    • 18
    • 19
    • 20
    • 21
    • 22
    • 23
    • 24
    • 25
    • 26

    G1

  • ?https://www.oracle.com/technical-resources/articles/java/g1gc.html
  • G1日志詳解

    <span style="color:#000000"><span style="background-color:#282c34"><code class="language-java"><span style="color:#999999">[</span>GC pause <span style="color:#999999">(</span>G1 Evacuation Pause<span style="color:#999999">)</span> <span style="color:#999999">(</span>young<span style="color:#999999">)</span> <span style="color:#999999">(</span>initial<span style="color:#669900">-</span>mark<span style="color:#999999">)</span><span style="color:#999999">,</span> <span style="color:#98c379">0.0015790</span> secs<span style="color:#999999">]</span> <span style="color:#5c6370">//young -> 年輕代 Evacuation-> 復制存活對象 </span> <span style="color:#5c6370">//initial-mark 混合回收的階段,這里是YGC混合老年代回收</span><span style="color:#999999">[</span>Parallel Time<span style="color:#669900">:</span> <span style="color:#98c379">1.5</span> ms<span style="color:#999999">,</span> GC Workers<span style="color:#669900">:</span> <span style="color:#98c379">1</span><span style="color:#999999">]</span> <span style="color:#5c6370">//一個GC線程</span><span style="color:#999999">[</span>GC Worker Start <span style="color:#999999">(</span>ms<span style="color:#999999">)</span><span style="color:#669900">:</span> <span style="color:#98c379">92635.7</span><span style="color:#999999">]</span><span style="color:#999999">[</span>Ext Root Scanning <span style="color:#999999">(</span>ms<span style="color:#999999">)</span><span style="color:#669900">:</span> <span style="color:#98c379">1.1</span><span style="color:#999999">]</span><span style="color:#999999">[</span>Update RS <span style="color:#999999">(</span>ms<span style="color:#999999">)</span><span style="color:#669900">:</span> <span style="color:#98c379">0.0</span><span style="color:#999999">]</span><span style="color:#999999">[</span>Processed Buffers<span style="color:#669900">:</span> <span style="color:#98c379">1</span><span style="color:#999999">]</span><span style="color:#999999">[</span>Scan RS <span style="color:#999999">(</span>ms<span style="color:#999999">)</span><span style="color:#669900">:</span> <span style="color:#98c379">0.0</span><span style="color:#999999">]</span><span style="color:#999999">[</span>Code Root Scanning <span style="color:#999999">(</span>ms<span style="color:#999999">)</span><span style="color:#669900">:</span> <span style="color:#98c379">0.0</span><span style="color:#999999">]</span><span style="color:#999999">[</span>Object Copy <span style="color:#999999">(</span>ms<span style="color:#999999">)</span><span style="color:#669900">:</span> <span style="color:#98c379">0.1</span><span style="color:#999999">]</span><span style="color:#999999">[</span>Termination <span style="color:#999999">(</span>ms<span style="color:#999999">)</span><span style="color:#669900">:</span> <span style="color:#98c379">0.0</span><span style="color:#999999">]</span><span style="color:#999999">[</span>Termination Attempts<span style="color:#669900">:</span> <span style="color:#98c379">1</span><span style="color:#999999">]</span><span style="color:#999999">[</span>GC Worker Other <span style="color:#999999">(</span>ms<span style="color:#999999">)</span><span style="color:#669900">:</span> <span style="color:#98c379">0.0</span><span style="color:#999999">]</span><span style="color:#999999">[</span>GC Worker Total <span style="color:#999999">(</span>ms<span style="color:#999999">)</span><span style="color:#669900">:</span> <span style="color:#98c379">1.2</span><span style="color:#999999">]</span><span style="color:#999999">[</span>GC Worker End <span style="color:#999999">(</span>ms<span style="color:#999999">)</span><span style="color:#669900">:</span> <span style="color:#98c379">92636.9</span><span style="color:#999999">]</span><span style="color:#999999">[</span>Code Root Fixup<span style="color:#669900">:</span> <span style="color:#98c379">0.0</span> ms<span style="color:#999999">]</span><span style="color:#999999">[</span>Code Root Purge<span style="color:#669900">:</span> <span style="color:#98c379">0.0</span> ms<span style="color:#999999">]</span><span style="color:#999999">[</span>Clear CT<span style="color:#669900">:</span> <span style="color:#98c379">0.0</span> ms<span style="color:#999999">]</span><span style="color:#999999">[</span>Other<span style="color:#669900">:</span> <span style="color:#98c379">0.1</span> ms<span style="color:#999999">]</span><span style="color:#999999">[</span>Choose CSet<span style="color:#669900">:</span> <span style="color:#98c379">0.0</span> ms<span style="color:#999999">]</span><span style="color:#999999">[</span>Ref Proc<span style="color:#669900">:</span> <span style="color:#98c379">0.0</span> ms<span style="color:#999999">]</span><span style="color:#999999">[</span>Ref Enq<span style="color:#669900">:</span> <span style="color:#98c379">0.0</span> ms<span style="color:#999999">]</span><span style="color:#999999">[</span>Redirty Cards<span style="color:#669900">:</span> <span style="color:#98c379">0.0</span> ms<span style="color:#999999">]</span><span style="color:#999999">[</span>Humongous Register<span style="color:#669900">:</span> <span style="color:#98c379">0.0</span> ms<span style="color:#999999">]</span><span style="color:#999999">[</span>Humongous Reclaim<span style="color:#669900">:</span> <span style="color:#98c379">0.0</span> ms<span style="color:#999999">]</span><span style="color:#999999">[</span>Free CSet<span style="color:#669900">:</span> <span style="color:#98c379">0.0</span> ms<span style="color:#999999">]</span><span style="color:#999999">[</span>Eden<span style="color:#669900">:</span> <span style="color:#98c379">0.</span><span style="color:#61aeee">0B</span><span style="color:#999999">(</span><span style="color:#98c379">1024.0</span>K<span style="color:#999999">)</span><span style="color:#669900">-</span><span style="color:#669900">></span><span style="color:#98c379">0.</span><span style="color:#61aeee">0B</span><span style="color:#999999">(</span><span style="color:#98c379">1024.0</span>K<span style="color:#999999">)</span> Survivors<span style="color:#669900">:</span> <span style="color:#98c379">0.0</span>B<span style="color:#669900">-</span><span style="color:#669900">></span><span style="color:#98c379">0.0</span>B Heap<span style="color:#669900">:</span> <span style="color:#98c379">18.</span><span style="color:#61aeee">8M</span><span style="color:#999999">(</span><span style="color:#98c379">20.0</span>M<span style="color:#999999">)</span><span style="color:#669900">-</span><span style="color:#669900">></span><span style="color:#98c379">18.</span><span style="color:#61aeee">8M</span><span style="color:#999999">(</span><span style="color:#98c379">20.0</span>M<span style="color:#999999">)</span><span style="color:#999999">]</span><span style="color:#999999">[</span>Times<span style="color:#669900">:</span> user<span style="color:#669900">=</span><span style="color:#98c379">0.00</span> sys<span style="color:#669900">=</span><span style="color:#98c379">0.00</span><span style="color:#999999">,</span> real<span style="color:#669900">=</span><span style="color:#98c379">0.00</span> secs<span style="color:#999999">]</span> <span style="color:#5c6370">//以下是混合回收其他階段</span> <span style="color:#999999">[</span>GC concurrent<span style="color:#669900">-</span>root<span style="color:#669900">-</span>region<span style="color:#669900">-</span>scan<span style="color:#669900">-</span>start<span style="color:#999999">]</span> <span style="color:#999999">[</span>GC concurrent<span style="color:#669900">-</span>root<span style="color:#669900">-</span>region<span style="color:#669900">-</span>scan<span style="color:#669900">-</span>end<span style="color:#999999">,</span> <span style="color:#98c379">0.0000078</span> secs<span style="color:#999999">]</span> <span style="color:#999999">[</span>GC concurrent<span style="color:#669900">-</span>mark<span style="color:#669900">-</span>start<span style="color:#999999">]</span> <span style="color:#5c6370">//無法evacuation,進行FGC</span> <span style="color:#999999">[</span>Full GC <span style="color:#999999">(</span>Allocation Failure<span style="color:#999999">)</span> <span style="color:#98c379">18</span>M<span style="color:#669900">-</span><span style="color:#669900">></span><span style="color:#61aeee">18M</span><span style="color:#999999">(</span><span style="color:#98c379">20</span>M<span style="color:#999999">)</span><span style="color:#999999">,</span> <span style="color:#98c379">0.0719656</span> secs<span style="color:#999999">]</span><span style="color:#999999">[</span>Eden<span style="color:#669900">:</span> <span style="color:#98c379">0.</span><span style="color:#61aeee">0B</span><span style="color:#999999">(</span><span style="color:#98c379">1024.0</span>K<span style="color:#999999">)</span><span style="color:#669900">-</span><span style="color:#669900">></span><span style="color:#98c379">0.</span><span style="color:#61aeee">0B</span><span style="color:#999999">(</span><span style="color:#98c379">1024.0</span>K<span style="color:#999999">)</span> Survivors<span style="color:#669900">:</span> <span style="color:#98c379">0.0</span>B<span style="color:#669900">-</span><span style="color:#669900">></span><span style="color:#98c379">0.0</span>B Heap<span style="color:#669900">:</span> <span style="color:#98c379">18.</span><span style="color:#61aeee">8M</span><span style="color:#999999">(</span><span style="color:#98c379">20.0</span>M<span style="color:#999999">)</span><span style="color:#669900">-</span><span style="color:#669900">></span><span style="color:#98c379">18.</span><span style="color:#61aeee">8M</span><span style="color:#999999">(</span><span style="color:#98c379">20.0</span>M<span style="color:#999999">)</span><span style="color:#999999">]</span><span style="color:#999999">,</span> <span style="color:#999999">[</span>Metaspace<span style="color:#669900">:</span> <span style="color:#98c379">38</span> <span style="color:#98c379">76</span>K<span style="color:#669900">-</span><span style="color:#669900">></span><span style="color:#61aeee">3876K</span><span style="color:#999999">(</span><span style="color:#98c379">1056768</span>K<span style="color:#999999">)</span><span style="color:#999999">]</span> <span style="color:#999999">[</span>Times<span style="color:#669900">:</span> user<span style="color:#669900">=</span><span style="color:#98c379">0.07</span> sys<span style="color:#669900">=</span><span style="color:#98c379">0.00</span><span style="color:#999999">,</span> real<span style="color:#669900">=</span><span style="color:#98c379">0.07</span> secs<span style="color:#999999">]</span></code></span></span>
    • 1
    • 2
    • 3
    • 4
    • 5
    • 6
    • 7
    • 8
    • 9
    • 10
    • 11
    • 12
    • 13
    • 14
    • 15
    • 16
    • 17
    • 18
    • 19
    • 20
    • 21
    • 22
    • 23
    • 24
    • 25
    • 26
    • 27
    • 28
    • 29
    • 30
    • 31
    • 32
    • 33
    • 34
    • 35
    • 36
    • 37
    • 38

    案例匯總

    OOM產生的原因多種多樣,有些程序未必產生OOM,不斷FGC(CPU飆高,但內存回收特別少) (上面案例)

  • 硬件升級系統反而卡頓的問題(見上)

  • 線程池不當運用產生OOM問題(見上)
    不斷的往List里加對象(實在太LOW)

  • smile jira問題
    實際系統不斷重啟
    解決問題 加內存 + 更換垃圾回收器 G1
    真正問題在哪兒?不知道

  • tomcat http-header-size過大問題(Hector)

  • lambda表達式導致方法區溢出問題(MethodArea / Perm Metaspace)
    LambdaGC.java -XX:MaxMetaspaceSize=9M -XX:+PrintGCDetails

    <span style="color:#000000"><span style="background-color:#282c34"><code class="language-java"><span style="color:#669900">"C:\Program Files\Java\jdk1.8.0_181\bin\java.exe"</span> <span style="color:#669900">-</span>XX<span style="color:#669900">:</span>MaxMetaspaceSize<span style="color:#669900">=</span><span style="color:#98c379">9</span>M <span style="color:#669900">-</span>XX<span style="color:#669900">:</span><span style="color:#669900">+</span>PrintGCDetails <span style="color:#669900">"-javaagent:C:\Program Files\JetBrains\IntelliJ IDEA Community Edition 2019.1\lib\idea_rt.jar=49316:C:\Program Files\JetBrains\IntelliJ IDEA Community Edition 2019.1\bin"</span> <span style="color:#669900">-</span>Dfile<span style="color:#999999">.</span>encoding<span style="color:#669900">=</span>UTF<span style="color:#669900">-</span><span style="color:#98c379">8</span> <span style="color:#669900">-</span>classpath <span style="color:#669900">"C:\Program Files\Java\jdk1.8.0_181\jre\lib\charsets.jar;C:\Program Files\Java\jdk1.8.0_181\jre\lib\deploy.jar;C:\Program Files\Java\jdk1.8.0_181\jre\lib\ext\access-bridge-64.jar;C:\Program Files\Java\jdk1.8.0_181\jre\lib\ext\cldrdata.jar;C:\Program Files\Java\jdk1.8.0_181\jre\lib\ext\dnsns.jar;C:\Program Files\Java\jdk1.8.0_181\jre\lib\ext\jaccess.jar;C:\Program Files\Java\jdk1.8.0_181\jre\lib\ext\jfxrt.jar;C:\Program Files\Java\jdk1.8.0_181\jre\lib\ext\localedata.jar;C:\Program Files\Java\jdk1.8.0_181\jre\lib\ext\nashorn.jar;C:\Program Files\Java\jdk1.8.0_181\jre\lib\ext\sunec.jar;C:\Program Files\Java\jdk1.8.0_181\jre\lib\ext\sunjce_provider.jar;C:\Program Files\Java\jdk1.8.0_181\jre\lib\ext\sunmscapi.jar;C:\Program Files\Java\jdk1.8.0_181\jre\lib\ext\sunpkcs11.jar;C:\Program Files\Java\jdk1.8.0_181\jre\lib\ext\zipfs.jar;C:\Program Files\Java\jdk1.8.0_181\jre\lib\javaws.jar;C:\Program Files\Java\jdk1.8.0_181\jre\lib\jce.jar;C:\Program Files\Java\jdk1.8.0_181\jre\lib\jfr.jar;C:\Program Files\Java\jdk1.8.0_181\jre\lib\jfxswt.jar;C:\Program Files\Java\jdk1.8.0_181\jre\lib\jsse.jar;C:\Program Files\Java\jdk1.8.0_181\jre\lib\management-agent.jar;C:\Program Files\Java\jdk1.8.0_181\jre\lib\plugin.jar;C:\Program Files\Java\jdk1.8.0_181\jre\lib\resources.jar;C:\Program Files\Java\jdk1.8.0_181\jre\lib\rt.jar;C:\work\ijprojects\JVM\out\production\JVM;C:\work\ijprojects\ObjectSize\out\artifacts\ObjectSize_jar\ObjectSize.jar"</span> com<span style="color:#999999">.</span>mashibing<span style="color:#999999">.</span>jvm<span style="color:#999999">.</span>gc<span style="color:#999999">.</span>LambdaGC <span style="color:#999999">[</span>GC <span style="color:#999999">(</span>Metadata GC Threshold<span style="color:#999999">)</span> <span style="color:#999999">[</span>PSYoungGen<span style="color:#669900">:</span> <span style="color:#98c379">11341</span>K<span style="color:#669900">-</span><span style="color:#669900">></span><span style="color:#61aeee">1880K</span><span style="color:#999999">(</span><span style="color:#98c379">38400</span>K<span style="color:#999999">)</span><span style="color:#999999">]</span> <span style="color:#98c379">11341</span>K<span style="color:#669900">-</span><span style="color:#669900">></span><span style="color:#61aeee">1888K</span><span style="color:#999999">(</span><span style="color:#98c379">125952</span>K<span style="color:#999999">)</span><span style="color:#999999">,</span> <span style="color:#98c379">0.0022190</span> secs<span style="color:#999999">]</span> <span style="color:#999999">[</span>Times<span style="color:#669900">:</span> user<span style="color:#669900">=</span><span style="color:#98c379">0.00</span> sys<span style="color:#669900">=</span><span style="color:#98c379">0.00</span><span style="color:#999999">,</span> real<span style="color:#669900">=</span><span style="color:#98c379">0.00</span> secs<span style="color:#999999">]</span> <span style="color:#999999">[</span>Full GC <span style="color:#999999">(</span>Metadata GC Threshold<span style="color:#999999">)</span> <span style="color:#999999">[</span>PSYoungGen<span style="color:#669900">:</span> <span style="color:#98c379">1880</span>K<span style="color:#669900">-</span><span style="color:#669900">></span><span style="color:#61aeee">0K</span><span style="color:#999999">(</span><span style="color:#98c379">38400</span>K<span style="color:#999999">)</span><span style="color:#999999">]</span> <span style="color:#999999">[</span>ParOldGen<span style="color:#669900">:</span> <span style="color:#98c379">8</span>K<span style="color:#669900">-</span><span style="color:#669900">></span><span style="color:#61aeee">1777K</span><span style="color:#999999">(</span><span style="color:#98c379">35328</span>K<span style="color:#999999">)</span><span style="color:#999999">]</span> <span style="color:#98c379">1888</span>K<span style="color:#669900">-</span><span style="color:#669900">></span><span style="color:#61aeee">1777K</span><span style="color:#999999">(</span><span style="color:#98c379">73728</span>K<span style="color:#999999">)</span><span style="color:#999999">,</span> <span style="color:#999999">[</span>Metaspace<span style="color:#669900">:</span> <span style="color:#98c379">8164</span>K<span style="color:#669900">-</span><span style="color:#669900">></span><span style="color:#61aeee">8164K</span><span style="color:#999999">(</span><span style="color:#98c379">1056768</span>K<span style="color:#999999">)</span><span style="color:#999999">]</span><span style="color:#999999">,</span> <span style="color:#98c379">0.0100681</span> secs<span style="color:#999999">]</span> <span style="color:#999999">[</span>Times<span style="color:#669900">:</span> user<span style="color:#669900">=</span><span style="color:#98c379">0.02</span> sys<span style="color:#669900">=</span><span style="color:#98c379">0.00</span><span style="color:#999999">,</span> real<span style="color:#669900">=</span><span style="color:#98c379">0.01</span> secs<span style="color:#999999">]</span> <span style="color:#999999">[</span>GC <span style="color:#999999">(</span>Last ditch collection<span style="color:#999999">)</span> <span style="color:#999999">[</span>PSYoungGen<span style="color:#669900">:</span> <span style="color:#98c379">0</span>K<span style="color:#669900">-</span><span style="color:#669900">></span><span style="color:#61aeee">0K</span><span style="color:#999999">(</span><span style="color:#98c379">38400</span>K<span style="color:#999999">)</span><span style="color:#999999">]</span> <span style="color:#98c379">1777</span>K<span style="color:#669900">-</span><span style="color:#669900">></span><span style="color:#61aeee">1777K</span><span style="color:#999999">(</span><span style="color:#98c379">73728</span>K<span style="color:#999999">)</span><span style="color:#999999">,</span> <span style="color:#98c379">0.0005698</span> secs<span style="color:#999999">]</span> <span style="color:#999999">[</span>Times<span style="color:#669900">:</span> user<span style="color:#669900">=</span><span style="color:#98c379">0.00</span> sys<span style="color:#669900">=</span><span style="color:#98c379">0.00</span><span style="color:#999999">,</span> real<span style="color:#669900">=</span><span style="color:#98c379">0.00</span> secs<span style="color:#999999">]</span> <span style="color:#999999">[</span>Full GC <span style="color:#999999">(</span>Last ditch collection<span style="color:#999999">)</span> <span style="color:#999999">[</span>PSYoungGen<span style="color:#669900">:</span> <span style="color:#98c379">0</span>K<span style="color:#669900">-</span><span style="color:#669900">></span><span style="color:#61aeee">0K</span><span style="color:#999999">(</span><span style="color:#98c379">38400</span>K<span style="color:#999999">)</span><span style="color:#999999">]</span> <span style="color:#999999">[</span>ParOldGen<span style="color:#669900">:</span> <span style="color:#98c379">1777</span>K<span style="color:#669900">-</span><span style="color:#669900">></span><span style="color:#61aeee">1629K</span><span style="color:#999999">(</span><span style="color:#98c379">67584</span>K<span style="color:#999999">)</span><span style="color:#999999">]</span> <span style="color:#98c379">1777</span>K<span style="color:#669900">-</span><span style="color:#669900">></span><span style="color:#61aeee">1629K</span><span style="color:#999999">(</span><span style="color:#98c379">105984</span>K<span style="color:#999999">)</span><span style="color:#999999">,</span> <span style="color:#999999">[</span>Metaspace<span style="color:#669900">:</span> <span style="color:#98c379">8164</span>K<span style="color:#669900">-</span><span style="color:#669900">></span><span style="color:#61aeee">8156K</span><span style="color:#999999">(</span><span style="color:#98c379">1056768</span>K<span style="color:#999999">)</span><span style="color:#999999">]</span><span style="color:#999999">,</span> <span style="color:#98c379">0.0124299</span> secs<span style="color:#999999">]</span> <span style="color:#999999">[</span>Times<span style="color:#669900">:</span> user<span style="color:#669900">=</span><span style="color:#98c379">0.06</span> sys<span style="color:#669900">=</span><span style="color:#98c379">0.00</span><span style="color:#999999">,</span> real<span style="color:#669900">=</span><span style="color:#98c379">0.01</span> secs<span style="color:#999999">]</span> java<span style="color:#999999">.</span>lang<span style="color:#999999">.</span>reflect<span style="color:#999999">.</span>InvocationTargetExceptionat sun<span style="color:#999999">.</span>reflect<span style="color:#999999">.</span>NativeMethodAccessorImpl<span style="color:#999999">.</span><span style="color:#61aeee">invoke0</span><span style="color:#999999">(</span>Native Method<span style="color:#999999">)</span>at sun<span style="color:#999999">.</span>reflect<span style="color:#999999">.</span>NativeMethodAccessorImpl<span style="color:#999999">.</span><span style="color:#61aeee">invoke</span><span style="color:#999999">(</span>NativeMethodAccessorImpl<span style="color:#999999">.</span>java<span style="color:#669900">:</span><span style="color:#98c379">62</span><span style="color:#999999">)</span>at sun<span style="color:#999999">.</span>reflect<span style="color:#999999">.</span>DelegatingMethodAccessorImpl<span style="color:#999999">.</span><span style="color:#61aeee">invoke</span><span style="color:#999999">(</span>DelegatingMethodAccessorImpl<span style="color:#999999">.</span>java<span style="color:#669900">:</span><span style="color:#98c379">43</span><span style="color:#999999">)</span>at java<span style="color:#999999">.</span>lang<span style="color:#999999">.</span>reflect<span style="color:#999999">.</span>Method<span style="color:#999999">.</span><span style="color:#61aeee">invoke</span><span style="color:#999999">(</span>Method<span style="color:#999999">.</span>java<span style="color:#669900">:</span><span style="color:#98c379">498</span><span style="color:#999999">)</span>at sun<span style="color:#999999">.</span>instrument<span style="color:#999999">.</span>InstrumentationImpl<span style="color:#999999">.</span><span style="color:#61aeee">loadClassAndStartAgent</span><span style="color:#999999">(</span>InstrumentationImpl<span style="color:#999999">.</span>java<span style="color:#669900">:</span><span style="color:#98c379">388</span><span style="color:#999999">)</span>at sun<span style="color:#999999">.</span>instrument<span style="color:#999999">.</span>InstrumentationImpl<span style="color:#999999">.</span><span style="color:#61aeee">loadClassAndCallAgentmain</span><span style="color:#999999">(</span>InstrumentationImpl<span style="color:#999999">.</span>java<span style="color:#669900">:</span><span style="color:#98c379">411</span><span style="color:#999999">)</span> Caused by<span style="color:#669900">:</span> java<span style="color:#999999">.</span>lang<span style="color:#999999">.</span>OutOfMemoryError<span style="color:#669900">:</span> Compressed <span style="color:#c678dd">class</span> spaceat sun<span style="color:#999999">.</span>misc<span style="color:#999999">.</span>Unsafe<span style="color:#999999">.</span><span style="color:#61aeee">defineClass</span><span style="color:#999999">(</span>Native Method<span style="color:#999999">)</span>at sun<span style="color:#999999">.</span>reflect<span style="color:#999999">.</span>ClassDefiner<span style="color:#999999">.</span><span style="color:#61aeee">defineClass</span><span style="color:#999999">(</span>ClassDefiner<span style="color:#999999">.</span>java<span style="color:#669900">:</span><span style="color:#98c379">63</span><span style="color:#999999">)</span>at sun<span style="color:#999999">.</span>reflect<span style="color:#999999">.</span>MethodAccessorGenerator$<span style="color:#98c379">1.</span><span style="color:#61aeee">run</span><span style="color:#999999">(</span>MethodAccessorGenerator<span style="color:#999999">.</span>java<span style="color:#669900">:</span><span style="color:#98c379">399</span><span style="color:#999999">)</span>at sun<span style="color:#999999">.</span>reflect<span style="color:#999999">.</span>MethodAccessorGenerator$<span style="color:#98c379">1.</span><span style="color:#61aeee">run</span><span style="color:#999999">(</span>MethodAccessorGenerator<span style="color:#999999">.</span>java<span style="color:#669900">:</span><span style="color:#98c379">394</span><span style="color:#999999">)</span>at java<span style="color:#999999">.</span>security<span style="color:#999999">.</span>AccessController<span style="color:#999999">.</span><span style="color:#61aeee">doPrivileged</span><span style="color:#999999">(</span>Native Method<span style="color:#999999">)</span>at sun<span style="color:#999999">.</span>reflect<span style="color:#999999">.</span>MethodAccessorGenerator<span style="color:#999999">.</span><span style="color:#61aeee">generate</span><span style="color:#999999">(</span>MethodAccessorGenerator<span style="color:#999999">.</span>java<span style="color:#669900">:</span><span style="color:#98c379">393</span><span style="color:#999999">)</span>at sun<span style="color:#999999">.</span>reflect<span style="color:#999999">.</span>MethodAccessorGenerator<span style="color:#999999">.</span><span style="color:#61aeee">generateSerializationConstructor</span><span style="color:#999999">(</span>MethodAccessorGenerator<span style="color:#999999">.</span>java<span style="color:#669900">:</span><span style="color:#98c379">112</span><span style="color:#999999">)</span>at sun<span style="color:#999999">.</span>reflect<span style="color:#999999">.</span>ReflectionFactory<span style="color:#999999">.</span><span style="color:#61aeee">generateConstructor</span><span style="color:#999999">(</span>ReflectionFactory<span style="color:#999999">.</span>java<span style="color:#669900">:</span><span style="color:#98c379">398</span><span style="color:#999999">)</span>at sun<span style="color:#999999">.</span>reflect<span style="color:#999999">.</span>ReflectionFactory<span style="color:#999999">.</span><span style="color:#61aeee">newConstructorForSerialization</span><span style="color:#999999">(</span>ReflectionFactory<span style="color:#999999">.</span>java<span style="color:#669900">:</span><span style="color:#98c379">360</span><span style="color:#999999">)</span>at java<span style="color:#999999">.</span>io<span style="color:#999999">.</span>ObjectStreamClass<span style="color:#999999">.</span><span style="color:#61aeee">getSerializableConstructor</span><span style="color:#999999">(</span>ObjectStreamClass<span style="color:#999999">.</span>java<span style="color:#669900">:</span><span style="color:#98c379">1574</span><span style="color:#999999">)</span>at java<span style="color:#999999">.</span>io<span style="color:#999999">.</span>ObjectStreamClass<span style="color:#999999">.</span>access$<span style="color:#61aeee">1500</span><span style="color:#999999">(</span>ObjectStreamClass<span style="color:#999999">.</span>java<span style="color:#669900">:</span><span style="color:#98c379">79</span><span style="color:#999999">)</span>at java<span style="color:#999999">.</span>io<span style="color:#999999">.</span>ObjectStreamClass$<span style="color:#98c379">3.</span><span style="color:#61aeee">run</span><span style="color:#999999">(</span>ObjectStreamClass<span style="color:#999999">.</span>java<span style="color:#669900">:</span><span style="color:#98c379">519</span><span style="color:#999999">)</span>at java<span style="color:#999999">.</span>io<span style="color:#999999">.</span>ObjectStreamClass$<span style="color:#98c379">3.</span><span style="color:#61aeee">run</span><span style="color:#999999">(</span>ObjectStreamClass<span style="color:#999999">.</span>java<span style="color:#669900">:</span><span style="color:#98c379">494</span><span style="color:#999999">)</span>at java<span style="color:#999999">.</span>security<span style="color:#999999">.</span>AccessController<span style="color:#999999">.</span><span style="color:#61aeee">doPrivileged</span><span style="color:#999999">(</span>Native Method<span style="color:#999999">)</span>at java<span style="color:#999999">.</span>io<span style="color:#999999">.</span>ObjectStreamClass<span style="color:#999999">.</span><span style="color:#61aeee"><span style="color:#999999"><</span>init<span style="color:#999999">></span></span><span style="color:#999999">(</span>ObjectStreamClass<span style="color:#999999">.</span>java<span style="color:#669900">:</span><span style="color:#98c379">494</span><span style="color:#999999">)</span>at java<span style="color:#999999">.</span>io<span style="color:#999999">.</span>ObjectStreamClass<span style="color:#999999">.</span><span style="color:#61aeee">lookup</span><span style="color:#999999">(</span>ObjectStreamClass<span style="color:#999999">.</span>java<span style="color:#669900">:</span><span style="color:#98c379">391</span><span style="color:#999999">)</span>at java<span style="color:#999999">.</span>io<span style="color:#999999">.</span>ObjectOutputStream<span style="color:#999999">.</span><span style="color:#61aeee">writeObject0</span><span style="color:#999999">(</span>ObjectOutputStream<span style="color:#999999">.</span>java<span style="color:#669900">:</span><span style="color:#98c379">1134</span><span style="color:#999999">)</span>at java<span style="color:#999999">.</span>io<span style="color:#999999">.</span>ObjectOutputStream<span style="color:#999999">.</span><span style="color:#61aeee">defaultWriteFields</span><span style="color:#999999">(</span>ObjectOutputStream<span style="color:#999999">.</span>java<span style="color:#669900">:</span><span style="color:#98c379">1548</span><span style="color:#999999">)</span>at java<span style="color:#999999">.</span>io<span style="color:#999999">.</span>ObjectOutputStream<span style="color:#999999">.</span><span style="color:#61aeee">writeSerialData</span><span style="color:#999999">(</span>ObjectOutputStream<span style="color:#999999">.</span>java<span style="color:#669900">:</span><span style="color:#98c379">1509</span><span style="color:#999999">)</span>at java<span style="color:#999999">.</span>io<span style="color:#999999">.</span>ObjectOutputStream<span style="color:#999999">.</span><span style="color:#61aeee">writeOrdinaryObject</span><span style="color:#999999">(</span>ObjectOutputStream<span style="color:#999999">.</span>java<span style="color:#669900">:</span><span style="color:#98c379">1432</span><span style="color:#999999">)</span>at java<span style="color:#999999">.</span>io<span style="color:#999999">.</span>ObjectOutputStream<span style="color:#999999">.</span><span style="color:#61aeee">writeObject0</span><span style="color:#999999">(</span>ObjectOutputStream<span style="color:#999999">.</span>java<span style="color:#669900">:</span><span style="color:#98c379">1178</span><span style="color:#999999">)</span>at java<span style="color:#999999">.</span>io<span style="color:#999999">.</span>ObjectOutputStream<span style="color:#999999">.</span><span style="color:#61aeee">writeObject</span><span style="color:#999999">(</span>ObjectOutputStream<span style="color:#999999">.</span>java<span style="color:#669900">:</span><span style="color:#98c379">348</span><span style="color:#999999">)</span>at javax<span style="color:#999999">.</span>management<span style="color:#999999">.</span>remote<span style="color:#999999">.</span>rmi<span style="color:#999999">.</span>RMIConnectorServer<span style="color:#999999">.</span><span style="color:#61aeee">encodeJRMPStub</span><span style="color:#999999">(</span>RMIConnectorServer<span style="color:#999999">.</span>java<span style="color:#669900">:</span><span style="color:#98c379">727</span><span style="color:#999999">)</span>at javax<span style="color:#999999">.</span>management<span style="color:#999999">.</span>remote<span style="color:#999999">.</span>rmi<span style="color:#999999">.</span>RMIConnectorServer<span style="color:#999999">.</span><span style="color:#61aeee">encodeStub</span><span style="color:#999999">(</span>RMIConnectorServer<span style="color:#999999">.</span>java<span style="color:#669900">:</span><span style="color:#98c379">719</span><span style="color:#999999">)</span>at javax<span style="color:#999999">.</span>management<span style="color:#999999">.</span>remote<span style="color:#999999">.</span>rmi<span style="color:#999999">.</span>RMIConnectorServer<span style="color:#999999">.</span><span style="color:#61aeee">encodeStubInAddress</span><span style="color:#999999">(</span>RMIConnectorServer<span style="color:#999999">.</span>java<span style="color:#669900">:</span><span style="color:#98c379">690</span><span style="color:#999999">)</span>at javax<span style="color:#999999">.</span>management<span style="color:#999999">.</span>remote<span style="color:#999999">.</span>rmi<span style="color:#999999">.</span>RMIConnectorServer<span style="color:#999999">.</span><span style="color:#61aeee">start</span><span style="color:#999999">(</span>RMIConnectorServer<span style="color:#999999">.</span>java<span style="color:#669900">:</span><span style="color:#98c379">439</span><span style="color:#999999">)</span>at sun<span style="color:#999999">.</span>management<span style="color:#999999">.</span>jmxremote<span style="color:#999999">.</span>ConnectorBootstrap<span style="color:#999999">.</span><span style="color:#61aeee">startLocalConnectorServer</span><span style="color:#999999">(</span>ConnectorBootstrap<span style="color:#999999">.</span>java<span style="color:#669900">:</span><span style="color:#98c379">550</span><span style="color:#999999">)</span>at sun<span style="color:#999999">.</span>management<span style="color:#999999">.</span>Agent<span style="color:#999999">.</span><span style="color:#61aeee">startLocalManagementAgent</span><span style="color:#999999">(</span>Agent<span style="color:#999999">.</span>java<span style="color:#669900">:</span><span style="color:#98c379">137</span><span style="color:#999999">)</span></code></span></span>
    • 1
    • 2
    • 3
    • 4
    • 5
    • 6
    • 7
    • 8
    • 9
    • 10
    • 11
    • 12
    • 13
    • 14
    • 15
    • 16
    • 17
    • 18
    • 19
    • 20
    • 21
    • 22
    • 23
    • 24
    • 25
    • 26
    • 27
    • 28
    • 29
    • 30
    • 31
    • 32
    • 33
    • 34
    • 35
    • 36
    • 37
    • 38
    • 39
    • 40
    • 41
    • 42
  • 直接內存溢出問題(少見)
    《深入理解Java虛擬機》P59,使用Unsafe分配直接內存,或者使用NIO的問題

  • 棧溢出問題
    -Xss設定太小

  • 比較一下這兩段程序的異同,分析哪一個是更優的寫法:

    <span style="color:#000000"><span style="background-color:#282c34"><code class="language-java">Object o <span style="color:#669900">=</span> null<span style="color:#999999">;</span> <span style="color:#c678dd">for</span><span style="color:#999999">(</span><span style="color:#c678dd">int</span> i<span style="color:#669900">=</span><span style="color:#98c379">0</span><span style="color:#999999">;</span> i<span style="color:#669900"><</span><span style="color:#98c379">100</span><span style="color:#999999">;</span> i<span style="color:#669900">++</span><span style="color:#999999">)</span> <span style="color:#999999">{</span>o <span style="color:#669900">=</span> <span style="color:#c678dd">new</span> Object<span style="color:#999999">(</span><span style="color:#999999">)</span><span style="color:#999999">;</span><span style="color:#5c6370">//業務處理</span> <span style="color:#999999">}</span> </code></span></span>
    • 1
    • 2
    • 3
    • 4
    • 5
    <span style="color:#000000"><span style="background-color:#282c34"><code class="language-java"><span style="color:#c678dd">for</span><span style="color:#999999">(</span><span style="color:#c678dd">int</span> i<span style="color:#669900">=</span><span style="color:#98c379">0</span><span style="color:#999999">;</span> i<span style="color:#669900"><</span><span style="color:#98c379">100</span><span style="color:#999999">;</span> i<span style="color:#669900">++</span><span style="color:#999999">)</span> <span style="color:#999999">{</span>Object o <span style="color:#669900">=</span> <span style="color:#c678dd">new</span> Object<span style="color:#999999">(</span><span style="color:#999999">)</span><span style="color:#999999">;</span> <span style="color:#999999">}</span> </code></span></span>
    • 1
    • 2
    • 3
  • 重寫finalize引發頻繁GC
    小米云,HBase同步系統,系統通過nginx訪問超時報警,最后排查,C++程序員重寫finalize引發頻繁GC問題
    為什么C++程序員會重寫finalize?(new delete)
    finalize耗時比較長(200ms)

  • 如果有一個系統,內存一直消耗不超過10%,但是觀察GC日志,發現FGC總是頻繁產生,會是什么引起的?
    System.gc() (這個比較Low)

  • Distuptor有個可以設置鏈的長度,如果過大,然后對象大,消費完不主動釋放,會溢出 (來自 死物風情)

  • 用jvm都會溢出,mycat用崩過,1.6.5某個臨時版本解析sql子查詢算法有問題,9個exists的聯合sql就導致生成幾百萬的對象(來自 死物風情)

  • new 大量線程,會產生 native thread OOM,(low)應該用線程池,
    解決方案:減少堆空間(太TMlow了),預留更多內存產生native thread
    JVM內存占物理內存比例 50% - 80%

  • GC常用參數

    • -Xmn -Xms -Xmx -Xss
      年輕代 最小堆 最大堆 棧空間
    • -XX:+UseTLAB
      使用TLAB,默認打開
    • -XX:+PrintTLAB
      打印TLAB的使用情況
    • -XX:TLABSize
      設置TLAB大小
    • -XX:+DisableExplictGC
      System.gc()不管用 ,FGC
    • -XX:+PrintGC
    • -XX:+PrintGCDetails
    • -XX:+PrintHeapAtGC
    • -XX:+PrintGCTimeStamps
    • -XX:+PrintGCApplicationConcurrentTime (低)
      打印應用程序時間
    • -XX:+PrintGCApplicationStoppedTime (低)
      打印暫停時長
    • -XX:+PrintReferenceGC (重要性低)
      記錄回收了多少種不同引用類型的引用
    • -verbose:class
      類加載詳細過程
    • -XX:+PrintVMOptions
    • -XX:+PrintFlagsFinal -XX:+PrintFlagsInitial
      必須會用
    • -Xloggc:opt/log/gc.log
    • -XX:MaxTenuringThreshold
      升代年齡,最大值15
    • 鎖自旋次數 -XX:PreBlockSpin 熱點代碼檢測參數-XX:CompileThreshold 逃逸分析 標量替換 …
      這些不建議設置

    Parallel常用參數

    • -XX:SurvivorRatio
    • -XX:PreTenureSizeThreshold
      大對象到底多大
    • -XX:MaxTenuringThreshold
    • -XX:+ParallelGCThreads
      并行收集器的線程數,同樣適用于CMS,一般設為和CPU核數相同
    • -XX:+UseAdaptiveSizePolicy
      自動選擇各區大小比例

    CMS常用參數

    • -XX:+UseConcMarkSweepGC
    • -XX:ParallelCMSThreads
      CMS線程數量
    • -XX:CMSInitiatingOccupancyFraction
      使用多少比例的老年代后開始CMS收集,默認是68%(近似值),如果頻繁發生SerialOld卡頓,應該調小,(頻繁CMS回收)
    • -XX:+UseCMSCompactAtFullCollection
      在FGC時進行壓縮
    • -XX:CMSFullGCsBeforeCompaction
      多少次FGC之后進行壓縮
    • -XX:+CMSClassUnloadingEnabled
    • -XX:CMSInitiatingPermOccupancyFraction
      達到什么比例時進行Perm回收
    • GCTimeRatio
      設置GC時間占用程序運行時間的百分比
    • -XX:MaxGCPauseMillis
      停頓時間,是一個建議時間,GC會嘗試用各種手段達到這個時間,比如減小年輕代

    G1常用參數

    • -XX:+UseG1GC
    • -XX:MaxGCPauseMillis
      建議值,G1會嘗試調整Young區的塊數來達到這個值
    • -XX:GCPauseIntervalMillis
      ?GC的間隔時間
    • -XX:+G1HeapRegionSize
      分區大小,建議逐漸增大該值,1 2 4 8 16 32。
      隨著size增加,垃圾的存活時間更長,GC間隔更長,但每次GC的時間也會更長
      ZGC做了改進(動態區塊大小)
    • G1NewSizePercent
      新生代最小比例,默認為5%
    • G1MaxNewSizePercent
      新生代最大比例,默認為60%
    • GCTimeRatio
      GC時間建議比例,G1會根據這個值調整堆空間
    • ConcGCThreads
      線程數量
    • InitiatingHeapOccupancyPercent
      啟動G1的堆空間占用比例

    作業

  • -XX:MaxTenuringThreshold控制的是什么?
    A: 對象升入老年代的年齡
    B: 老年代觸發FGC時的內存垃圾比例

  • 生產環境中,傾向于將最大堆內存和最小堆內存設置為:(為什么?)
    A: 相同 B:不同

  • JDK1.8默認的垃圾回收器是:
    A: ParNew + CMS
    B: G1
    C: PS + ParallelOld
    D: 以上都不是

  • 什么是響應時間優先?

  • 什么是吞吐量優先?

  • ParNew和PS的區別是什么?

  • ParNew和ParallelOld的區別是什么?(年代不同,算法不同)

  • 長時間計算的場景應該選擇:A:停頓時間 B: 吞吐量

  • 大規模電商網站應該選擇:A:停頓時間 B: 吞吐量

  • HotSpot的垃圾收集器最常用有哪些?

  • 常見的HotSpot垃圾收集器組合有哪些?

  • JDK1.7 1.8 1.9的默認垃圾回收器是什么?如何查看?

  • 所謂調優,到底是在調什么?

  • 如果采用PS + ParrallelOld組合,怎么做才能讓系統基本不產生FGC

  • 如果采用ParNew + CMS組合,怎樣做才能夠讓系統基本不產生FGC

    1.加大JVM內存

    2.加大Young的比例

    3.提高Y-O的年齡

    4.提高S區比例

    5.避免代碼內存泄漏

  • G1是否分代?G1垃圾回收器會產生FGC嗎?

  • 如果G1產生FGC,你應該做什么?

  • 擴內存
  • 提高CPU性能(回收的快,業務邏輯產生對象的速度固定,垃圾回收越快,內存空間越大)
  • 降低MixedGC觸發的閾值,讓MixedGC提早發生(默認是45%)
  • 問:生產環境中能夠隨隨便便的dump嗎?
    小堆影響不大,大堆會有服務暫停或卡頓(加live可以緩解),dump前會有FGC

  • 問:常見的OOM問題有哪些?
    棧 堆 MethodArea 直接內存

  • 參考資料

  • https://blogs.oracle.com/jonthecollector/our-collectors
  • https://docs.oracle.com/javase/8/docs/technotes/tools/unix/java.html
  • http://java.sun.com/javase/technologies/hotspot/vmoptions.jsp
  • JVM調優參考文檔:https://docs.oracle.com/en/java/javase/13/gctuning/introduction-garbage-collection-tuning.html#GUID-8A443184-7E07-4B71-9777-4F12947C8184
  • https://www.cnblogs.com/nxlhero/p/11660854.html 在線排查工具
  • https://www.jianshu.com/p/507f7e0cc3a3 arthas常用命令
  • Arthas手冊:
  • 啟動arthas java -jar arthas-boot.jar
  • 綁定java進程
  • dashboard命令觀察系統整體情況
  • help 查看幫助
  • help xx 查看具體命令幫助
  • jmap命令參考: https://www.jianshu.com/p/507f7e0cc3a3
  • jmap -heap pid
  • jmap -histo pid
  • jmap -clstats pid
  • https://blog.csdn.net/chenssy/article/details/78271744?JVM致命錯誤日志(hs_err_pid.log)分析

    總結

    以上是生活随笔為你收集整理的马士兵 jvm调优的全部內容,希望文章能夠幫你解決所遇到的問題。

    如果覺得生活随笔網站內容還不錯,歡迎將生活随笔推薦給好友。

    综合伊人av | 中文字幕 国产 一区 | 国产精品99久久久久久小说 | 色综合天天狠天天透天天伊人 | 午夜精品久久久久久久久久久久久久 | 国产精华国产精品 | 人人草人| 中文字幕91视频 | www好男人| 精品一区电影国产 | 亚洲免费av在线 | 亚洲精品中文字幕视频 | 日韩欧美视频二区 | 日本久久久久久久久久久 | 成人黄性视频 | 人人舔人人爱 | 久久er99热精品一区二区 | av一二三区 | 精品高清美女精品国产区 | 久久综合加勒比 | 国产精品久久久久久久久久久久久久 | 日韩手机在线观看 | 在线电影91 | 欧美人交a欧美精品 | 中文成人字幕 | 久久激情五月丁香伊人 | 日韩综合在线观看 | 久久久久麻豆 | 大型av综合网站 | 色综合天天视频在线观看 | 在线观看视频一区二区三区 | 日韩毛片久久久 | 久久激情视频免费观看 | 久草在线在线视频 | 欧美精品中文在线免费观看 | 亚洲va欧美va人人爽 | 不卡av电影在线观看 | 国产亚洲精品女人久久久久久 | 久久一区二区三区国产精品 | 亚洲人成在线电影 | 亚洲精品中文在线资源 | 中文字幕超清在线免费 | 精品1区2区3区 | 五月天丁香视频 | 欧美最猛性xxxxx免费 | 国内外成人在线 | 99色在线观看视频 | 97影视 | 最近2019中文免费高清视频观看www99 | 五月婷婷另类国产 | 97精品视频在线播放 | 色国产在线 | 久久亚洲国产精品 | 久久人人97超碰精品888 | 久久久www成人免费毛片 | 激情久久网 | 久草免费在线观看视频 | av免费网站在线观看 | 免费日韩三级 | 午夜av剧场 | 国产高清一级 | 欧美最猛性xxx | 亚洲专区欧美 | 久久经典国产视频 | 麻豆成人精品视频 | 国产精品美女久久久 | 狠狠久久伊人 | 亚洲日本精品视频 | 久免费 | 日韩欧美在线观看一区二区 | 成人a视频在线观看 | 国产视频一区二区三区在线 | 乱子伦av| 九九热在线精品视频 | 色多多污污在线观看 | 国产小视频在线免费观看 | 国产精品福利在线 | 五月婷丁香网 | 国产一级免费观看视频 | 激情综合五月网 | 天天干国产 | 最近中文字幕在线 | 天天搞天天干天天色 | 久久精品一二三区 | 黄网站色成年免费观看 | 亚洲国产三级在线 | 亚洲 欧美 综合 在线 精品 | 亚洲天天在线日亚洲洲精 | 夜夜夜夜夜夜操 | 成人免费在线观看电影 | 99精品国产兔费观看久久99 | 中文字幕二区三区 | 最近免费中文字幕大全高清10 | 日韩综合精品 | 精品国偷自产在线 | www激情com | 色偷偷网站视频 | 婷香五月 | 日韩高清在线一区二区三区 | 超碰97在线资源 | 高清精品在线 | 国产亚洲精品电影 | 麻豆极品 | 夜夜操狠狠操 | 日韩视频a| 国产美女精品在线 | 欧美性护士 | 亚洲国产字幕 | 国产精品粉嫩 | 久久手机精品视频 | 菠萝菠萝蜜在线播放 | 国内精品久久久久 | 韩国一区二区在线观看 | 成人av在线直播 | 91精品中文字幕 | 中文字幕在线观看你懂的 | 欧美日韩国产精品一区二区亚洲 | 久久久久国产精品免费 | 在线观看精品一区 | 成年人免费看的视频 | 日日夜夜天天射 | 91精品在线免费视频 | 久草在线视频免赞 | 欧美一二三视频 | 在线探花| 97国产电影 | 日韩亚洲欧美中文字幕 | 97精品在线 | 成人黄色中文字幕 | 欧美久久久久久久久久久久久 | av亚洲产国偷v产偷v自拍小说 | 婷婷丁香色 | www免费 | 中文字幕在线成人 | 九色视频网址 | 亚洲日本韩国一区二区 | 日本精品视频免费观看 | 日本黄色免费观看 | 午夜视频在线观看一区二区三区 | 国产一级在线 | 日韩视频一区二区在线 | 久久久久久国产精品 | 精品久久久久久国产偷窥 | 99re国产 | 九九国产视频 | 国产91精品在线观看 | 久久综合干 | 在线观看蜜桃视频 | 国产夫妻av在线 | 香蕉视频免费在线播放 | 亚洲成人av在线播放 | 久久高清国产视频 | 91插插插网站 | 国产精品第7页 | 97超碰成人在线 | 国产69精品久久99不卡的观看体验 | 日韩av女优视频 | 天堂av官网 | 成人黄色电影在线 | 在线国产91 | 国产精品久久艹 | 日韩三级在线观看 | www日韩欧美 | 久久精品99国产国产精 | 在线探花 | 麻豆一区在线观看 | 亚洲午夜久久久久 | 国产精品免费一区二区三区 | 国产字幕av | 日韩av影视在线 | 精品少妇一区二区三区在线 | 国产精品系列在线 | 亚洲一二三区精品 | 久久久三级视频 | 久久国产精品色av免费看 | 国产成人精品在线观看 | 色全色在线资源网 | 国产精品久久久久久a | 国产在线播放不卡 | 丰满少妇在线观看网站 | 亚洲有 在线 | av免费在线观 | 天天躁天天狠天天透 | 人人澡人摸人人添学生av | 久久成人午夜 | 激情视频国产 | 98精品国产自产在线观看 | 在线成人免费电影 | 九色91视频 | 最近的中文字幕大全免费版 | 国产麻豆剧传媒免费观看 | 久久久96 | 欧美精品色 | 人人射人人澡 | 日本性xxx| 91麻豆精品国产91久久久无需广告 | 热99在线 | 日韩午夜网站 | 黄色日本免费 | 亚洲无吗视频在线 | 99久久婷婷国产精品综合 | 美女又爽又黄 | 97人人精品 | 黄色小说18| 91视频在线看 | 五月婷婷av在线 | 国产精品av一区二区 | 日韩欧美在线播放 | 奇米影视999 | 久草com| 亚洲国产播放 | 天天干天天干天天射 | 国产精品久久久一区二区 | 久久毛片高清国产 | 午夜色站| japanesefreesex中国少妇 | 欧美人人爱 | 怡红院av久久久久久久 | 精品久久91 | 丁香午夜婷婷 | 婷婷久月| 国产精品岛国久久久久久久久红粉 | 福利网在线| 久久久久国产成人免费精品免费 | 久久久久久久久久免费 | 香蕉影视 | 成人观看| 91视频午夜 | 久久国产精品影视 | 国产成人99av超碰超爽 | 日韩视 | 亚州精品天堂中文字幕 | 成人av电影免费在线播放 | 国产福利91精品张津瑜 | 亚洲精品玖玖玖av在线看 | 黄色成人影视 | 六月丁香在线视频 | 成人在线一区二区 | 六月色| 欧美日韩亚洲第一页 | 在线观看国产成人av片 | 超碰最新网址 | 国产一区二区三区四区在线 | 99精品国产99久久久久久福利 | 久久av伊人 | 国产精品原创 | 久久久久久免费视频 | 亚洲三级影院 | 不卡av在线 | 成人av在线影视 | 久久精品5| ,午夜性刺激免费看视频 | 西西www4444大胆视频 | 欧美伦理电影一区二区 | 日本在线中文 | 国产精品成人一区二区三区 | 久久精品xxx | 欧美另类高清 videos | 国产主播99 | 日韩精品一区二区在线 | 午夜精品电影一区二区在线 | 91亚瑟视频 | a级片久久 | 天天视频色版 | 99c视频高清免费观看 | 免费网站污 | 成年人视频在线免费观看 | 日韩精品一区在线观看 | 免费黄色特级片 | 亚洲人成综合 | 91九色蝌蚪视频网站 | 亚洲激情校园春色 | 91精品国产麻豆国产自产影视 | 久久婷婷精品 | 182午夜在线观看 | 91精品久久香蕉国产线看观看 | 一区二区不卡高清 | 国产福利在线免费观看 | 成年人电影毛片 | 日韩网站视频 | 欧美日本啪啪无遮挡网站 | 婷婷六月在线 | 亚洲欧洲精品视频 | 国产精品久久久久久久久久久免费看 | 亚洲精品国产品国语在线 | 国产精品中文久久久久久久 | 亚洲自拍偷拍色图 | av动态图片 | 免费视频18 | 亚洲精品字幕 | 久久高清毛片 | 久久国产精品精品国产色婷婷 | 国产午夜精品免费一区二区三区视频 | 欧美日韩高清免费 | 99性视频| 手机av网站 | 在线蜜桃视频 | 免费激情网 | 久久成人免费视频 | 米奇影视7777| 午夜精品久久久久久久久久久久久久 | 色婷在线 | 国产一区二区高清不卡 | 2019中文字幕网站 | 黄污污网站| 黄色大片日本 | 日韩高清免费电影 | 国产69精品久久久久99尤 | 久久久激情网 | 日韩欧美精品在线视频 | 欧美xxxxx在线视频 | 香蕉影视在线观看 | 国产精品女人久久久久久 | 日韩最新中文字幕 | 国产精品久久久久久一区二区三区 | 久久在线免费观看视频 | 麻豆小视频在线观看 | 久久一区二区免费视频 | 99久久精| 17videosex性欧美| 国产日韩中文字幕 | 91爱爱视频 | 亚洲dvd | 激情五月激情综合网 | 国产小视频在线看 | 在线电影 你懂得 | 青青河边草免费直播 | a视频免费看 | 国产二区免费视频 | 成人在线电影观看 | 久艹在线观看视频 | 极品美女被弄高潮视频网站 | 婷婷中文字幕在线观看 | 97人人人人| 久久免费99精品久久久久久 | 97av.com| av色综合| 国产高清在线视频 | 中文字幕一区2区3区 | 色偷偷男人的天堂av | 色综合中文综合网 | 精品久久美女 | 又黄又爽又无遮挡的视频 | 波多野结衣日韩 | 亚洲综合成人专区片 | 国产 日韩 在线 亚洲 字幕 中文 | 久久久伊人网 | 中文字幕一区二区三区久久 | 久草精品电影 | 成年人免费av网站 | 在线免费观看黄色 | 国产精品无av码在线观看 | 五月婷婷久久丁香 | 久久免费在线观看视频 | 国产视频 亚洲精品 | 久久婷婷视频 | 九九九热精品免费视频观看 | 黄免费在线观看 | 国产美女视频一区 | 日韩中文字幕在线观看 | 成人av免费在线看 | 亚洲一区二区天堂 | 人人爽人人爽人人爽人人爽 | 成年人免费看片 | 日韩精品久久久免费观看夜色 | 欧美日韩亚洲在线观看 | 国产精品视频不卡 | 国产一二三四在线视频 | 黄色av电影网 | 国产精品毛片一区 | 亚洲电影一区二区 | 国产经典 欧美精品 | 激情综合亚洲 | 亚洲色图av | 成人av影视| 欧美日韩一区久久 | 成年人电影免费在线观看 | 97色在线观看免费视频 | 精品久久久999 | 黄色成人免费电影 | 婷婷深爱激情 | 国产精品久久久久久久久久99 | 婷婷中文字幕 | av九九| 九热在线 | 国产99久久久国产精品成人免费 | 涩涩爱夜夜爱 | japanesefreesex中国少妇 | 久久久久久草 | 亚洲精品成人在线 | 精品国产一区二区三区日日嗨 | 91喷水| 色综合欧洲 | 天天天天色综合 | 国内综合精品午夜久久资源 | 日韩一区二区三区在线看 | 色在线亚洲 | 99热精品免费观看 | 中文久草| 国产精品18久久久久久首页狼 | 久久精品看片 | 91在线国产观看 | 日韩免费在线播放 | aa级黄色大片| 国产日韩欧美视频 | 日韩女同一区二区三区在线观看 | 天天干夜夜干 | 人人干网站 | 亚洲国产操 | 人人爽人人爽人人爽 | 天天综合天天综合 | 我要色综合天天 | 久精品视频在线 | 日韩欧美精品一区二区 | 久久久精品日本 | 久久久精品小视频 | 色婷婷六月 | 波多野结衣网址 | 福利视频午夜 | 久操中文字幕在线观看 | .精品久久久麻豆国产精品 亚洲va欧美 | 成人福利在线 | 丝袜一区在线 | 9在线观看免费高清完整版在线观看明 | 欧美精品中文字幕亚洲专区 | 成人久久精品 | 精品久久久久久亚洲综合网站 | 国产精品毛片网 | 91视频中文字幕 | 日日夜夜综合网 | av免费福利 | 日韩专区一区二区 | 日日摸日日爽 | 久久视频这里有久久精品视频11 | 久久天堂网站 | 最近中文字幕完整视频高清1 | 色香天天 | 国产二区视频在线观看 | 国产在线一区二区 | 日韩精品在线免费观看 | 亚洲欧美观看 | 国产成人精品999在线观看 | av女优中文字幕在线观看 | 九九天堂 | 亚洲电影一区二区 | 久久精品在线 | 99精品国产高清在线观看 | 精品一区二区在线看 | 久草免费在线视频 | va视频在线观看 | 一区二区三区国产欧美 | 91福利在线导航 | 精品久久久久国产免费第一页 | 波多野结衣在线观看一区 | 又色又爽的网站 | 亚洲在线免费视频 | 久久精品www人人爽人人 | 黄色片亚洲 | 偷拍视频一区 | 九九综合九九 | 久久影院一区 | 一区二区视频电影在线观看 | 不卡的av中文字幕 | 成人四虎 | 成人四虎| 探花视频免费观看 | 九色自拍视频 | 免费在线国产黄色 | 在线观看成人福利 | 韩国视频一区二区三区 | 黄色亚洲大片免费在线观看 | 欧美日韩精品免费观看视频 | 8x成人免费视频 | 久久免费一级片 | 国产麻豆精品一区二区 | 久久人人爽人人爽人人片 | 国产精品综合在线观看 | 六月激情久久 | 日韩av电影国产 | 久久国产精品久久国产精品 | 午夜影视一区 | 激情九九| 三级av在线播放 | 久久99久久精品国产 | 国产成人资源 | 久久综合九色综合97婷婷女人 | 亚洲国产视频直播 | 91在线日韩 | 国产爽视频 | 中文字幕欧美激情 | 九九综合九九 | 久草在线看片 | 五月婷婷在线观看视频 | 毛片在线播放网址 | 国产特级毛片aaaaaa高清 | 久草成人在线 | 亚洲最新毛片 | 精品欧美小视频在线观看 | 夜夜夜夜猛噜噜噜噜噜初音未来 | 一区二区三区四区在线免费观看 | 精品一区二区6 | 91在线国内视频 | 日韩欧美一级二级 | 国产精品日韩欧美 | 国产精品成人久久 | 97在线播放视频 | 精品久久久影院 | 国产一区二区三区在线 | 91禁在线看 | 激情视频一区 | 日韩激情视频在线 | 国产高清av免费在线观看 | 久久国产色 | 91av精品 | 欧美色婷 | 九九久久免费视频 | 日韩二区三区在线 | 三上悠亚在线免费 | 操操操操网 | 亚洲精品99久久久久中文字幕 | 国产99黄| 国产日韩视频在线播放 | 久久久久久看片 | 久久精品4 | 国产一区在线视频观看 | 视频精品一区二区三区 | 久久在线看 | 国产999视频 | 国产69精品久久久久99尤 | 国产日产欧美在线观看 | 国产精品一区二区果冻传媒 | 九九热在线观看 | 97精品欧美91久久久久久 | av电影在线观看完整版一区二区 | 丁香六月色 | 久久精彩视频 | 欧美性天天 | 色网站国产精品 | 六月色播 | 激情视频免费在线观看 | 91麻豆精品国产午夜天堂 | 超碰精品在线观看 | 人人射av | 99久久er热在这里只有精品66 | 特级毛片aaa | 伊人五月天婷婷 | 精选久久| 久久国产精品99久久久久 | 国产专区精品视频 | 五月婷婷激情 | 国产中文字幕在线看 | 最近中文字幕久久 | 91在线免费视频观看 | 视频在线99re | 国产高清视频在线免费观看 | 亚洲国产黄色片 | 久久99在线观看 | 亚洲最新毛片 | 热久久免费国产视频 | 91精品久久久久久久久久入口 | 亚洲国产片色 | 国产精品99久久免费黑人 | 91成人观看| 91av原创| 日韩精品视频久久 | 日日夜精品 | 999国产在线 | 麻豆国产在线播放 | 99久久久久久久久 | 国产午夜精品一区二区三区嫩草 | 激情久久综合 | 在线观看你懂的网址 | 国产精品一区二区吃奶在线观看 | 日韩在线观看你懂得 | 激情网五月 | 麻豆影视在线播放 | 国产高清 不卡 | 亚洲影院国产 | 久久高清视频免费 | 国产视频97 | 免费看毛片在线 | 国产资源免费在线观看 | 97成人资源站 | 伊人狠狠 | 日本黄色黄网站 | 久久福利| 天天综合五月天 | 日韩在线观看网址 | 国产高清av在线播放 | 四虎影视成人永久免费观看视频 | 91福利专区 | 精品网站999www | 顶级欧美色妇4khd | a黄色大片 | 91av短视频 | 欧美最新大片在线看 | 亚洲欧美精品在线 | 亚洲成a人片在线观看中文 中文字幕在线视频第一页 狠狠色丁香婷婷综合 | 国偷自产中文字幕亚洲手机在线 | 日本特黄特色aaa大片免费 | 午夜精选视频 | 国产在线日本 | 人人射人人爽 | 激情婷婷久久 | 麻豆精品传媒视频 | 久草在线免费看视频 | 亚洲 av网站 | 国产在线黄色 | 亚洲国产欧美在线人成大黄瓜 | 久久天天拍 | 国产综合婷婷 | 91免费观看视频在线 | 亚洲理论片 | 欧美一二三区在线播放 | 激情网在线视频 | 久久精品亚洲精品国产欧美 | 日韩中文字幕免费视频 | 国产一级在线观看 | 免费在线色视频 | 18国产精品白浆在线观看免费 | 久久综合久久八八 | 天天操天天色天天 | 欧美中文字幕第一页 | 国产探花视频在线播放 | 国产高清久久久久 | 97精品国产91久久久久久久 | 综合久久2023 | 日本爱爱免费 | 91久久人澡人人添人人爽欧美 | 欧美激情精品久久久久久 | 97久久精品午夜一区二区 | 久久草网 | 国产乱码精品一区二区蜜臀 | 丁香六月婷婷激情 | 91在线欧美 | 久久综合九色综合97_ 久久久 | 国产精品美女免费视频 | 国产裸体永久免费视频网站 | 99久久影视 | www.色婷婷| 国产精品午夜免费福利视频 | 日韩久久电影 | 国产精品密入口果冻 | 在线免费av播放 | 成年人网站免费观看 | 精品国产_亚洲人成在线 | 97国产大学生情侣白嫩酒店 | av黄色成人 | 在线视频你懂得 | 亚洲三级毛片 | 992tv人人草 黄色国产区 | 欧美成亚洲 | 国内丰满少妇猛烈精品播 | 久久精品国产一区 | 日韩电影在线看 | 九九九在线观看视频 | 日本一区二区高清不卡 | 日本中文不卡 | 久久99久久99精品免观看粉嫩 | 在线天堂视频 | 欧美一区二区在线 | 日韩精品视频在线观看网址 | 黄网站免费久久 | 欧美午夜一区二区福利视频 | 中文字幕亚洲精品在线观看 | 99人久久精品视频最新地址 | 日日夜夜av | 狠狠色综合网站久久久久久久 | 日韩中文字幕视频在线 | 久久久精品免费观看 | 在线电影播放 | 欧美成人影音 | avwww在线观看 | 九九免费在线观看 | 国产一级片一区二区三区 | 欧美性生交大片免网 | 99热这里精品 | 五月综合在线观看 | 国产成在线观看免费视频 | 国产精品日韩久久久久 | 日韩有码在线播放 | 深夜福利视频在线观看 | 精品毛片久久久久久 | 国产在线精品一区二区 | 欧美a级成人淫片免费看 | 精品黄色视 | 亚洲资源在线网 | 日韩簧片在线观看 | 97伊人网| 成人午夜黄色影院 | av爱干 | 国产精品乱码久久久久久1区2区 | av在线等 | 久久精品美女视频网站 | 狠狠干天天干 | 日本论理电影 | 91污在线| 国产精品剧情在线亚洲 | 中文字幕4 | 91欧美国产| av电影中文字幕在线观看 | 天堂资源在线观看视频 | 在线免费观看麻豆视频 | 综合网伊人 | 成年人网站免费在线观看 | 国产高清日韩欧美 | 亚洲综合色视频在线观看 | 亚洲精品无 | 国产视频日本 | 国产理论影院 | 五月天婷亚洲天综合网精品偷 | 欧美综合在线视频 | 美女黄频在线观看 | av电影在线不卡 | 精品久久精品久久 | 午夜视频色 | 激情综合啪啪 | 亚洲成aⅴ人片久久青草影院 | 欧美黑吊大战白妞欧美 | 亚洲午夜av | 国产视频久 | 久久久影片 | 国产97视频| 国产伦理久久精品久久久久_ | 国产精品99久久99久久久二8 | 国产亚洲午夜高清国产拍精品 | 在线观看91久久久久久 | 精品国产片 | 小草av在线播放 | 在线观看视频一区二区三区 | 欧美日韩亚洲在线 | 国产精品视频你懂的 | 久久69av | 超碰999 | 超碰人人草| 91精品伦理 | 天天激情天天干 | 久久久久国产免费免费 | 精品久久久久久久久久久久久久久久 | 亚洲欧洲一级 | 黄色大片免费播放 | 国产美女精品视频免费观看 | 免费视频一区二区 | 亚洲视频 一区 | 91成人在线观看喷潮 | 蜜桃视频成人在线观看 | 国产精品免费在线 | 不卡精品 | 亚洲免费黄色 | 久久这里有| 一个色综合网站 | 久久久三级视频 | 在线观看国产中文字幕 | 在线观看视频免费播放 | 五月天激情在线 | 99热亚洲精品 | 亚洲国产日本 | 91在线视频在线观看 | 日本黄区免费视频观看 | 97电影在线观看 | 00av视频 | 中文字幕在线日 | 国产精品资源 | 丁香婷婷激情国产高清秒播 | 日本激情视频中文字幕 | 久久免费视频在线观看6 | 欧美片一区二区三区 | 日韩精品首页 | 99精品免费久久久久久日本 | 久久99热这里只有精品 | 碰超人人 | 中文字幕一区在线观看视频 | 久久久精品99 | 黄色福利网站 | 久久人91精品久久久久久不卡 | 中文字幕成人一区 | 日韩国产欧美在线视频 | 国产一区二区久久久久 | 亚洲国产精品99久久久久久久久 | 久久免费观看少妇a级毛片 久久久久成人免费 | 日本三级不卡视频 | 懂色av懂色av粉嫩av分享吧 | 亚洲精品在线视频观看 | 超碰在线亚洲 | 深夜免费小视频 | 深爱婷婷 | 国产五月色婷婷六月丁香视频 | av免费电影在线 | 欧美日韩在线免费视频 | 天天操欧美| 国产精品一区二区在线免费观看 | 91精品一区二区在线观看 | 高清视频一区二区三区 | 国产欧美综合视频 | 久久久精品视频网站 | 日本久久高清视频 | 免费看黄网站在线 | 日日操夜夜操狠狠操 | 一区二区三区在线视频观看58 | 日本久久高清视频 | 日韩av电影国产 | 中文字幕网址 | 特级毛片aaa | 黄色片软件网站 | 超碰在线1| 免费视频91 | 人人插人人插 | 99视频在线观看免费 | 久久激情网站 | 欧美精品在线观看一区 | 国产精品18久久久 | 久久免费国产 | 狠狠色噜噜狠狠狠狠 | 欧美最猛性xxxxx免费 | 欧美日韩国产在线一区 | 亚洲精品国产精品乱码不99热 | 97成人在线| 黄色毛片在线观看 | 91免费网站在线观看 | 色婷婷成人 | 亚洲综合色婷婷 | 欧美精品久久久久性色 | 免费观看第二部31集 | 美女网站在线 | 少妇bbw揉bbb欧美 | 欧美日韩精品在线播放 | 国产精品精 | 正在播放一区 | 欧美日韩国产在线精品 | www178ccom视频在线| 欧美日韩视频在线观看免费 | 日韩av不卡在线观看 | 亚洲精品自拍视频在线观看 | 高清精品在线 | 亚洲最新av网站 | 日韩av免费一区 | 91桃色在线免费观看 | 国产精品一区二区av | 色婷婷在线播放 | 国产99亚洲 | 综合色爱 | 亚洲国产小视频在线观看 | 国产色区 | 黄色大片网 | 欧美国产日韩一区 | 又黄又爽又无遮挡的视频 | 欧美视频日韩视频 | 欧美日本日韩aⅴ在线视频 插插插色综合 | www.888av | 婷婷午夜激情 | 韩国在线一区二区 | 天天干天天草天天爽 | 又黄又爽的免费高潮视频 | 91亚洲欧美 | 亚洲国内精品在线 | 欧美日韩中字 | 久久精品视频播放 | 欧美无极色 | 日韩午夜在线观看 | 成人毛片在线观看视频 | 人人草在线视频 | 69精品视频在线观看 | 国产成人精品在线观看 | 欧美三级在线播放 | 国产精品一区二区三区久久 | 欧美日韩啪啪 | 日本黄色大片免费 | 久热久草在线 | 夜夜操狠狠操 | www在线观看视频 | 久久国产精品视频 | av高清影院 | 色狠狠综合 | 成人毛片a | 中文一区在线观看 | 欧美一级片免费观看 | 在线观看国产中文字幕 | 一级a性色生活片久久毛片波多野 | 成人午夜剧场在线观看 | 国产精品一区二区三区四 | 激情视频一区二区三区 | 丰满少妇在线观看资源站 | 国产主播大尺度精品福利免费 | 天海翼一区二区三区免费 | 亚洲人人精品 | 日韩理论片中文字幕 | 成人一级免费电影 | 久久一区二区免费视频 | 日韩大片在线看 | 99久久网站 | 国产日韩视频在线播放 | 91视频在线观看大全 | 精品亚洲午夜久久久久91 | 久精品视频免费观看2 | www亚洲国产 | 久久免费99精品久久久久久 | av久久在线| 麻豆国产在线视频 | 日韩一区二区三区高清在线观看 | 中文字幕在线观看视频免费 | 青春草免费在线视频 | 国产69精品久久久久久久久久 | 国产精品久久久久永久免费 | 国内精品久久久久影院日本资源 | 久久视频在线 | 成人丝袜 | 久草视频观看 | 亚洲少妇影院 | www激情网| 色999在线| www.夜夜夜 | 午夜久久久久久久 | 99免费精品 | 国产99精品在线观看 | 欧美精品乱码久久久久 | 国产精品对白一区二区三区 | 亚洲精品久久久久久久蜜桃 | 91高清免费看| 久久久久蜜桃 | 麻豆视频免费入口 | 在线观看中文字幕第一页 | 超碰97人人爱 | 91中文在线视频 | 欧美日韩中文字幕综合视频 | 91久久久久久久一区二区 | 一区电影 | 亚洲国产伊人 | 九色视频网站 | av片在线观看 | 国产精品99久久久久久宅男 | 久久久久国产一区二区 | 在线观看日韩国产 | 婷婷久久久久 | 日韩免费观看高清 | 国产精品一区二区吃奶在线观看 | 国产精品欧美一区二区三区不卡 | 成年人视频在线观看免费 | 五月天堂网 | 亚洲精品一区二区在线观看 | 91成人破解版 | 97香蕉久久国产在线观看 | 麻豆久久久 | 国产一级视频在线免费观看 | 天天干,天天插 | 免费性网站 | 国产录像在线观看 | 国产精品毛片久久久久久久久久99999999 | 久草在线中文视频 | 亚洲国产日韩精品 | 看av免费网站 | 亚洲婷婷综合色高清在线 | 久操中文字幕在线观看 | 国产一区二区高清视频 | 日韩欧美xx | 精品国产一区二区三区久久 | 日韩在线观看网址 | av电影一区二区三区 | 97超碰资源网 | 五月天激情视频 | 国产在线免费观看 | 国产精品一区二区三区久久 | 欧美日韩国产二区三区 | 国产精品久久二区 | 香蕉久久久久 | 日日夜夜天天射 | 精品欧美小视频在线观看 | 日精品| 少妇性bbb搡bbb爽爽爽欧美 | 久久久久亚洲国产 | 欧美久久久影院 | 国产精品视频 | 黄色小说免费在线观看 | 欧美一区影院 | 在线观看久久久久久 | 久久人人爽爽人人爽人人片av | 91人网站 | 九九九热精品免费视频观看 | 精品欧美乱码久久久久久 | 国产色资源 | 久久伦理视频 | 日韩av视屏在线观看 | 久久99中文字幕 | 夜夜夜夜猛噜噜噜噜噜初音未来 | 国产一级二级三级视频 | 日日夜夜骑| 欧美孕妇与黑人孕交 | 精品视频专区 | 亚洲一区天堂 | free,性欧美 九九交易行官网 | 奇人奇案qvod| 天天干天天草天天爽 | 久久一区二区三区日韩 | 在线视频成人 | 日日夜夜噜 | 国产视频在线观看一区 | 日本黄色免费看 | 五月婷婷av| 天天做天天爱天天爽综合网 |