日韩av黄I国产麻豆传媒I国产91av视频在线观看I日韩一区二区三区在线看I美女国产在线I麻豆视频国产在线观看I成人黄色短片

歡迎訪問 生活随笔!

生活随笔

當前位置: 首頁 > 编程资源 > 编程问答 >内容正文

编程问答

【OS】NMON的简介和使用

發布時間:2024/3/26 编程问答 64 豆豆
生活随笔 收集整理的這篇文章主要介紹了 【OS】NMON的简介和使用 小編覺得挺不錯的,現在分享給大家,幫大家做個參考.

【OS】NMON的簡介和使用





目前NMON已開源,以sourceforge為根據地,網址是http://nmon.sourceforge.net。







1.?目的

本文介紹操作系統監控工具Nmon的概念、使用方式及使用參數。指導運維人員通過nmon工具監視AIX/Linux操作系統資源使用情況,收集監控結果及產生的數據文件,制作相關系統性能分析報告。

2.?Nmon簡介

Nmon (Nigel’s Monitor)是由IBM 提供、免費監控 AIX 系統與 Linux 系統資源的工具。該工具可將服務器系統資源耗用情況收集起來并輸出一個特定的文件,并可利用 excel 分析工具(nmon analyser)進行數據的統計分析。

2.1??軟件特性

nmon 工具可以在一個屏幕上顯示所有重要的性能優化信息,并動態地對其進行更新。這個高效的工具可以工作于任何啞屏幕、telnet 會話、甚至撥號線路。另外,它不會消耗大量的 CPU 周期,通常低于百分之二(在更新的計算機上,其 CPU 使用率將低于百分之一)。nmon使用啞屏幕,在屏幕上對數據進行顯示,并且每隔兩秒鐘對其進行更新。用戶可以很容易地將這個時間間隔更改為更長或更短的時間段。如果拉伸窗口,并在 X Windows、VNC、PuTTY 或類似的窗口中顯示這些數據,nmon 工具可以同時輸出大量的信息。

nmon 工具還可以將相同的數據捕獲到一個文本文件,便于以后對報告進行分析和繪制圖形。輸出文件采用電子表格的格式 (.csv)。

目前nmon已開源,以sourceforge為根據地,網址是http://nmon.sourceforge.net

2.2??軟件組成

Nmon使用需要nmon工具和nmonanalyser分析程序兩者配合使用。nmon工具生成性能數據文件,然后monanalyser以nmon生成的數據文件作為輸入,輸出為Excel?電子表格,并自動地生成相應的圖形,使得我們能夠直觀地觀察OS性能(CPU、IO和內存等)的變化過程。

2.3??運行環境

??? nmon 工具運行于:

·?AIX? 4.1.5、4.2.0、4.3.2 和 4.3.3(nmon Version 9a:該版本的功能已經確定,并且不會對其進行進一步的開發。)

·?AIX 5.1、5.2 和 5.3(nmon Version 10:該版本現在支持 AIX 5.3 和基于 POWER5? 處理器的計算機,并且提供了 SMT 和共享 CPU 微分區的支持。)

·?pSeries? p5 和 OpenPower? 上的 Linux? SUSE SLES 9、Red Hat EL 3 和 4、Debian

·?Linux SUSE、Red Hat 和許多最新的 x86(32 位模式的 Intel 和 AMD)上的發布版

·?zSeries? 或 mainframe 上的 Linux SUSE 和 Red Hat

nmon 工具大約每六個月更新一次,或者在可用的新的操作系統發布版中對其進行更新。

2.4??軟件功能

nmon 工具可以為 AIX 和 Linux 性能專家提供監視和分析性能數據的功能,其中包括:

·?CPU 使用率

·?內存使用情況

·?內核統計信息和運行隊列信息

·?磁盤 I/O 速度、傳輸和讀/寫比率

·?文件系統中的可用空間

·?磁盤適配器

·?網絡 I/O 速度、傳輸和讀/寫比率

·?頁面空間和頁面速度

·?CPU 和 AIX 規范

·?消耗資源最多的進程

·?IBM HTTP Web 緩存

·?用戶自定義的磁盤組

·?計算機詳細信息和資源

·?異步 I/O,僅適用于 AIX

·?工作負載管理器 (WLM),僅適用于 AIX

·?IBM TotalStorage? Enterprise Storage Server? (ESS) 磁盤,僅適用于 AIX

·?網絡文件系統 (NFS)

·?動態 LPAR (DLPAR) 更改,僅適用于面向 AIX 或 Linux 的 pSeries p5 和 OpenPower

2.5??軟件獲取

nmon工具和nmonanalyser工具都可以在IBM的Wike頁面上下載到。

1)?Nmon下載:
位置:可從IBM Wiki上下載
http://www-941.haw.ibm.com/collaboration/wiki/display/WikiPtype/nmon
下載頁面如下:
下載 (31.17 KB)2009-2-22 23:49

例如測試的系統是AIX5.3,那么就可以下載nmon4aix12e.zip ,下載后可以看到壓縮包里是一些文件,如下:

下載 (8.54 KB)

2009-2-22 23:49

其實nmon就是shell腳本,nmon文件運行時調用其他的文件,生成性能數據,這個工具運行時也是通過執行nmon腳本接受參數。

2)?Nmonanalyser下載:
位置:可從可從IBM Wiki上下載
http://www-941.haw.ibm.com/collaboration/wiki/display/Wikiptype/nmonanalyser
下載頁面如下:

下載 (26.08 KB)

2009-2-22 23:49

例如下載V3.3版本的Nmonanalyser

3.?Nmon使用

3.1??下載軟件

1)?下載之前須確定操作系統內核版本,本文以公司服務器192.168.40.212為例。登錄服務器使用命令獲取操作系統版本信息:

?

查看可知為Enterprise Linux Server release 5.5。

2)?登錄IBM官方網站(http://nmon.sourceforge.net/pmwiki.php?n=Site.Download)下載相應版本的nmon工具:nmon_linux_14g.tar.gz?

3.2??安裝軟件

1)?用root用戶登錄系統,建立目錄:#mkdir? /nmon

2)?通過FTP將下載的nmon工具上傳至服務器 192.168.40.212目錄/nmon下。?

3)?修改tar包權限:#chmod +x??nmon_linux_14g.tar.gz?

4)?解壓文件:#tar xvfZ? nmon_linux_14g.tar.gz?

5)?執行授權命令:#chmod? +x? nmon_x86_rhel54

3.3??運行界面

1)?運行./nmon_x86_rhel54,便可進入nmon的監控界面,如下圖:

?

2)?鍵入“c”查看系統CPU使用情況

?

3)?鍵入“m”查看系統內存使用情況

?

4)? 鍵入“d”查看系統磁盤I/O情況

?

5)?鍵入“h”查看幫助信息

?

3.4??使用方法

3.4.1?實時監控

在完成配置后,只需要“#./nmon”命令即可運行程序,通過一些快捷鍵調取關心的系統資源指標進行顯示,如按鍵“C”可以查看CPU相 關信息;按鍵“D”可以查看磁盤信息;按鍵“T”可以查看系統的進程信息;“M”對應內存、“N”對應網絡等等,完整的快捷鍵對應內容可以通過幫助(按鍵 “H”)查看,可以顯示Linux系統CPU、內存、進程信息,包括了CPU的用戶、系統、等待和空閑狀態值,可用內存、緩存大小以及進程的CPU消耗等詳細指標。?

該種方式顯示信息實時性強,能夠及時掌握系統承受壓力下的運行情況,每顆CPU利用率是多少、內存使用多少、網絡流量多少、磁盤讀寫……這些數據均是實時刷新,一目了然。

3.4.2?后臺監控

為了配合性能測試,我們往往需要將一個時間段內系統資源消耗情況記錄下來,這時可以使用命令在遠程窗口執行命令:

./nmon/ nmon_x86_rhel5? -f -N -m /nmon/log? -s 30 -c 120
其中各參數表示:
? -f 按標準格式輸出文件:_YYYYMMDD_HHMM.nmon
? -N include NFS sections
? -m 切換到路徑去保存日志文件
? -s 每隔n秒抽樣一次,這里為30
? -c 取出多少個抽樣數量,這里為120,即監控=120*(30/60/60)=1小時
??? 根據小時計算這個數字的公式為:c=h*3600/s,比如要監控10小時,每隔30秒采樣一次,則c=10*3600/30=1200

該命令啟動后,會在nmon所在目錄下生成監控文件,并持續寫入資源數據,直至360個監控點收集完成——即監控1小時,這些操作均自動完成,無需手工干 預,測試人員可以繼續完成其他操作。如果想停止該監控,需要通過“#ps –ef|grep nmon”查詢進程號,然后殺掉該進程以停止監控。

3.4.3?定時任務

??? 除配合性能測試的短期監控,我們也可以實現對系統的定期監控,作為運營維護階段的參考。定期監控實現如下:

1)?執行命令:#crontab? –e

2)?在最后一行添加如下命令:
0 8 * * 1,2,3,4,5? /nmon/nmon_x86_rhel5? -f -N -m /nmon/log? -s 30 -c 1200
表示:
周一到周五,從早上08點開始,監控10個小時(到18:00整為止),輸出到/nmon/log

4.?Nmon監控結果介紹

4.1??生成結果文件

???? 通過后臺監控和定期監控,我們可以得到擴展名為nmon的監控文件,這些文件記錄著系統資源的數據,需要配合分析工具(nmon analyser)進行解讀。

1)?使用FTP工具從服務器上取下生成結果文件/nmon/log/sjfx212_120318_1723.nmon到本機。

2)?打開nmon_analyser.zip 包下的nmon analyser v33g.xls 文件,點擊Analyse nomn data按鈕,選擇之前get下來的sjfx212_120318_1723.nmon文件。

Excel可能禁止運行宏了,點“安全警告旁邊的選項”,允許運行宏:

??? ???

3)?生成分析結果文件sjfx212_120318_1723.nmon.xlsx,并生成統計圖,直觀顯示系統資源情況。

4.2??主要性能參數介紹

u?系統匯總(對應excel標簽的‘SYS_SUMM’):其中藍線為cpu占有率變化情況;粉線為磁盤IO的變化情況。

u?磁盤讀寫情況匯總(對應excel標簽的‘DISK_SUMM’):?其中藍色為磁盤讀的速率KB/sec;紫色為磁盤寫的速率KB/sec。

u?內存情況匯總(對應excel標簽的‘MEM’): 曲線表示內存剩余量(MB)

4.3??頁面介紹

簡單介紹生成結果的各個頁面:

1)?系統匯總頁面:對應標簽頁(SYS_SUMM)

頁面顯示項主要有主機名,執行日期,系統cpu使用情況(藍線),系統I/O情況(粉紅線),其中坐標左縱軸為系統cpu(user%+sys%)使用率,橫軸為運行時長(下圖為一個小時),右縱軸為系統磁盤傳輸(Disk xfers),坐標下側為統計信息:系統I/O情況(一個周期內的平均值、最大值、出現最大值的時間),系統CPU使用情況。

選項

說明

備注

User%

用戶進程時間在CPU 開銷時間百分比

如果一個CPU 被充分使用,利用率分類之間均衡的比例應該是:

65% - 70% User Time

30% - 35% System Time

0% - 5%?? Idle Time

Wait%

所有進程線程被阻塞等待完成一次IO 請求所占CPU 開銷idle的時間百分比

Sys%

線程和中斷在CPU 開銷時間百分比

Idle%

CPU 空閑時間的百分比

CPU%

CPU 利用率的百分比

?

注釋:

如果系統CPU有IO wait存在,說明可能IO或內存方面存在瓶頸,其中主要導致IO Wait的主要原因如下:

ü?內存不夠而引起頻繁的的數據交換,導致數據存取存在交換空間的 I/O瓶頸

ü?硬盤數據分布不合理

2)?系統信息:標簽頁(AAA)

頁面信息主要包括:執行命令,主機CPU數(4),操作系統內核版本信息,主機名等信息。

?

3)?系統詳細信息:標簽頁(BBBP)

?頁面信息主要包括:操作系統版本,主機磁盤信息,主機CPU型號、主頻信息,內存信息,網卡信息等。

?

4)?CPU使用情況:標簽頁(CPU_ALL,CPU_SUMM,CPU001,CPU002,CPU003,CPU004)

主機CPU使用情況匯總以及單顆CPU的運行情況。

?

?

5)?磁盤讀寫情況匯總:標簽頁(DISK_SUMM,DISKBSIZE,DISKBUSY,DISKREAD,DISKWRITE,DISKXFER)

?磁盤的讀、寫及I/O統計信息,系統各磁盤分區的讀寫情況。其中DISKBUSY頁主要反映系統本地磁盤使用,DISK_SUMM包括本地和存儲(ESS, EMC, FASt及HDS)上所有磁盤使用情況。

?

?

?

6)?內存使用情況:標簽頁(MEM)

系統內存空閑、使用,swap、cached等統計信息。

?

7)?系統網絡情況:標簽頁(NET,NETPACKET)

反映系統的網絡運行情況,系統各個網絡適配器讀寫的數據包數

?

8)?系統進程:標簽頁(PROC)

反映系統運行線程及等待切換的線程平均數。其中RunQueue - 每個處理器應該運行隊列不超過1-3個線程。

?

?

9)?活動虛擬內存總量:標簽頁(VM)

Linux操作系統特有指標,主要包括系統/proc/vmstat文件中信息,兩張圖片主要顯示系統分頁文件(pagefile)和swap分區運行情況。如果系統總是存在大量換出頁(pgpgout/s)KB數,說明系統需要更多內存。

?

?

10)?頁式調度:標簽頁(PAGE)

記錄系統(AIX)頁式調度的情況。這一欄主要記錄系統換頁(paging)情況和頁面掃描:自由比率(page scan:free ratio)。其中系統換頁頻率應該不大于5次/秒,而當頁面掃描:自由比率持續大于4時,需要重點關注下系統內存和分頁空間的使用情況。

?

說明:

大量的內存交換操作會極大地影響系統的性能,尤其是在當數據庫文件創建在文件系統上時(JFS and JFS2)。在這種情況下經常訪問的數據,即在SGA中存在,也同樣在文件的緩存中存在。這種相同的數據在內存中緩存兩次的情況,會降低內存的使用效率,從而使內存頻繁進行交換操作,造成系統的I/O瓶頸,降低整個系統的性能。

11)?采集時間:標簽頁(ZZZZ)

記錄nmon工具采集系統信息的時間點。

?

5.?Nmon監控案例介紹

本節介紹通過nmon工具發現系統性能下降問題的常見現象和處理流程。

5.1??常見現象和產生原因

性能下降(Performance Degradation),主要是指系統的性能隨時間而逐漸下降(這里假定在系統性能下降的過程中系統的負載狀況沒有明顯變化)。系統運行過程中占用的CPU或內存隨時間增加也屬于廣義的性能下降問題。

在生產環境中,通常由終端客戶最先感覺到并報告性能下降問題。所以狹義的性能下降問題主要是指系統運行指標隨時間變化,比如吞吐率隨時間下降或頁面響應時間隨時間上升,或者兩者兼而有之。

下面列舉一些引起性能下降問題的原因:

ü?應用程序資源使用問題。主要是內存使用問題,即由于應用服務器的內存碎片問題或內存泄漏問題,導致垃圾回收的開銷隨時間增大。也有可能是因為磁盤臨時文件積累造成磁盤訪問開銷增大。

ü?應用程序設計問題。由于應用程序的設計存在可擴展性或可靠性問題,導致運行開銷隨時間或業務對象的積累而增大。

ü?數據庫訪問問題。該問題又可以分為許多類型,如調優參數問題、表結構或索引設計問題、垃圾數據問題等。其共同特點是導致應用程序利用特定操作訪問數據庫的開銷隨時間而增大。

ü?服務器軟件資源使用問題。雖然可能性很小,但是應用服務器、數據庫服務器等服務器程序也是軟件程序,也有可能存在性能下降問題。這些服務器程序在自身測試過程中可能遺漏了某些性能問題,而在用戶特定的執行狀況下觸發了這些問題,結果導致這些服務器程序使用的操作系統資源泄漏而出現性能下降問題。

ü?測試用例設計問題。性能測試中有可能發現一些“假”的性能下降問題。比如測試用例設計時假設在測試執行過程中系統負載保持恒定,但實際的測試用例實現導致系統負載或特定頁面的處理內容隨時間增多,也可能導致測試工具的測試報告中出現性能下降問題。

5.2??實例介紹

5.2.1?示例一

?

5-1? nmon操作系統監視匯總信息圖

由此信息圖可發現在12小時測試進行過程中,系統的磁盤傳輸(Disk xfers)逐漸增大,與此同時系統CPU占用率逐漸下降。進一步檢查單個CPU的使用情況,發現1號CPU的Wait狀態占用率明顯增大,如圖5-2所示。這說明CPU占用率逐漸下降是由于等待磁盤I/O引起的。

?

5-2? nmon單個CPU使用情況圖

接下來分析磁盤傳輸匯總信息,如圖5-3所示,可以看出磁盤寫數據量沒有明顯增加,但是磁盤讀數據量明顯隨時間而增加。

?

5-3? nmon磁盤傳輸匯總情況圖

憑磁盤傳輸匯總信息,在排除由應用服務器讀取導致的性能下降問題后,基本可以肯定不斷增加的磁盤讀取操作是由數據庫引起的。

隨后,分析DB2的快照監視器的監視結果,可以發現DB2的緩沖池(Buffer pool)的數據和索引物理讀(physical read)的比例非常高。如下例所示:

Buffer pool data logical reads???????????? = 5502388

Buffer pool data physical reads??????????? = 430671

Buffer pool temporary data logical reads?? = 0

Buffer pool temporary data physical reads? = 0

……

可以看到緩沖池的物理讀比例(即緩沖池不命中率)高達7%,這遠遠大于1%的警戒線。而且物理讀比例有隨時間增加的趨勢(通過不同時間的快照信息對比發現)。

至此可以懷疑性能下降問題是由于DB2的緩沖池配置參數設置不當引起的。考察數據庫配置參數信息發現,該數據庫的BUFFPAGE參數值為10000。與該測試用例使用的數據規模相比,這個參數值明顯偏小。于是將BUFFPAGE參數值增大10倍,變為100000,重新運行性能測試,發現性能下降問題基本消失。

5.2.2?示例二

本實例為通過nmon監控河北稅源管理平臺數據倉庫收集的數據。對照nmonanalyser工具生成結果進行分析。

1)?收集服務器基本信息

?

?

?

?????? 通過以上數據可以知道河北數據倉庫服務器基本配置信息如下:

ü?主機名:HE_SSGLY_DB_01

ü?操作系統版本:AIX 5.3.0.44?build?5300-04

ü?操作系統內核:HW-type=CHRP=Common H/W Reference Platform Bus=PCI LPAR=Dynamic Multi-Processor 64 bit

ü?主機型號:IBM p5 595 (9119-595)

ü?網絡配置:IP Address: 75.16.16.191?Sub Netmask: 255.255.248.0?Gateway: 75.16.16.100

ü?主機存儲:EMC存儲

ü?主機物理內存:49152 MB

ü?網卡信息:2塊網卡 速率為1024M/S

2)?文件系統使用情況

?

?

?????? 通過以上數據可以知道河北數據倉庫服務器文件系統相關信息如下:

ü?文件系統分區大小及使用情況

ü?文件系統掛載點信息

ü?文件系統類型為JFS2(Journaled File System 2)文件系統

3)?系統資源使用情況

?

????? 觀察上圖,可以發現服務器cpu使用率較高的時段為8:00-11:30、14:00-17:20,而系統I/O開銷較高的時段均出現在21:00-次日5:50。以上信息基本符合服務器晚上進行數據ETL處理、白天工作時段多進行平臺一般事務處理的情況。進一步觀察服務器cpu使用情況:

?

?

?

???? 發現CPU利用率較高均為用戶進程,單獨查看CPU-11注意到22:00-0:00這個時段內CPU的Wait狀態占用率明顯增大, user狀態占用率卻很低,此時系統I/O明顯升高,說明這個時段內有大量磁盤I/O發生,CPU占用率逐漸下降是由于等待磁盤I/O引起的。實際情況中,這個時段服務器正在進行ETL數據處理,的確有大量數據傳輸和磁盤讀寫發生。

?

5.2.3?示例三

?? ? 介紹個利用nmon進行系統實時監控的例子。

???? 把nmon腳本上傳到服務中,直接運行即可,執行命令如下:#./nmon或者#/tmp/nmon/nmon

???? 以下是aix5.3下使用nmon的一些截圖:

?

?? 分別輸入c、t、n、m,可以了解系統cpu,內存,消耗資源最高的線程的使用情況。

?




Links:

NMON home page

http://www-941.haw.ibm.com/collaboration/wiki/display/Wikiptype/nmon

NMON_Analyser home page

http://www-941.haw.ibm.com/collaboration/wiki/display/Wikiptype/nmonanalyser

User Forum

http://www.ibm.com/developerworks/forums/dw_forum.jsp?forum=749&cat=56





一.nmon工具介紹

nmon 是一個分析aix和linux性能的免費工具(主要是ibm為自己的aix操作系統開發的,但是也可以用在linux操作系統),而nmon_analyser是nmon的一個工具可以把nmon生成的報告轉化成excel報表的形式供查看

nmon 工具可以在一個屏幕上顯示所有重要的性能優化信息,并動態地對其進行更新。它并不會消耗大量的 CPU ,通常低于百分之二

?

下載地址:

http://nmon.sourceforge.net/pmwiki.php?n=Site.Download

http://sourceforge.net/projects/nmon/files/?source=navbar

?

注意選擇和自己機器對應的版本:

linux查看系統版本號:

1.方法1

對于linux系統而已,有成百上千個發行版。對于發行版的版本號查看方法

如以centos為例。輸入lsb_release -a即可

該命令適用于所有的linux,包括Redhat、SuSE、Debian等發行版

?

?

2.

?

3.

?

查看系統位數

同樣對于Linux系統而言也有32和64位之分,用上述查看內核方法,輸入

uname -a ?或 more /proc/version

在內核版本后面會有一個X86_64就是表示64為系統啦

# uname -a
x86_64則說明你是64位內核, 跑的是64位的系統.
i386, i686說明你是32位的內核, 跑的是32位的系統

?

查看內核的幾種方法:

1. uname -a#顯示詳細的內核信息,

Linux localhost.localdomain 2.6.18-92.1.6.el5xen #1 SMP Wed Jun 25 12:56:52 EDT 2008 x86_64 x86_64 x86_64 GNU/Linux

2. cat /etc/issue#顯示版本類型

Scientific Linux SL release 5.2 (Boron)
Kernel \r on an \m

3. cat /proc/version

Linux version 2.6.18-92.1.6.el5xen (brewbuilder@norob.fnal.gov) (gcc version 4.1.2 20071124 (Red Hat 4.1.2-42)) #1 SMP Wed Jun 25 12:56:52 EDT 2008

?

?

查看操作系統位數:

1.ls /?#如果有lib64或這個目錄,那操作系統就是64位的
2.getconfig LONG_BIT?若輸出32即為32位系統,64為64位系統

32位的系統中int類型和long類型一般都是4字節,

64位的系統中int類型還是4字節的,但是long已變成了8字節。

inux系統中可 用"getconf WORD_BIT"和"getconf LONG_BIT"獲得word和long的位數。
64位系統中應該分別得到32和64。

3.?uname -a中若為X86示意為64位系統,i386等位32位系統

?

?

?

二.下載安裝

?

nmon默認下載之后只是一個可執行測的文件,但是下載時根據不同的系統來進行區分的

大家可以根據自己的系統選擇相應的版本。

執行命令 more /etc/issue 查看系統版本。

?

首先將下載的nmon_x86_64_centos6文件上傳到linux服務器上

?

[root@localhost source]#cp nmon_x86_64_centos6 /usr/bin

[root@localhost source]#cd /usr/bin

[root@localhost source]# chmod 777 nmon_x86_64_centos6

[root@localhost source]# ./nmon_x86_64_centos6(切換到nmon_x86_rhel45文件所在的目錄)

[root@localhost source]# env

[root@localhost source]# mv?nmon_x86_64_centos6 nmon

[root@localhost source]# nmon(此時nmon就是全局命令)

也有可能

?

監控設置

15分鐘=900s

#alias nmon15='nmon -f -s3 -c300 -m /root/qumf/'

?

?

nmon工具命令的使用:

#nmon -s 300 -c 288 -f -m /tmp

-s 300:表示每300秒采集一次數據,

-c 288 :表示采集288次,300*288=86400秒,剛好是1天的數據,這樣運行一次這個程序就會生成一個一天的數據文件,

-m /tmp: 表示生成的數據文件的路徑

-f :表示生成的數據文件名中有時間

也可以讓其自動監控,每天一個文件夾,每小時一個文件,每分鐘或5分采樣一次

?

?

?

參數解釋:
-s10 每 10 秒采集一次數據。
-c60 采集 60 次,即為采集十分鐘的數據。
-f 生成的數據文件名中包含文件創建的時間。
-m 生成的數據文件的存放目錄。

?

nmon -f -s 10 -c 60

-f 表示生成的數據文件名中有時間;

-s 10 表示每 10 秒采集一次數據;

-c 60 表示采集 60 次,10*60=600 秒,

剛好是 10分鐘的數據,這樣運行一次這個程序就會生成一個采集 10分鐘數據的文件。該行命令將在當前目錄中創建輸出文件,其名稱為:_date_time.nmon", hostname" 是這臺主機的主機名。

?

更多用法
crontab -e
0 0 * * * /usr/local/nmo -s300 -c288 -f -m /home/nmon/ > /dev/null 2>&1
表示:
300*288=86400 秒,正好是一天的數據。
0 8 * * 1,2,3,4,5 /usr/local/nmon -f -N -m /home/nmon/log -s 30 -c 1200
表示:
周一到周五,從早上08點開始,監控10個小時(到18:00整為止),輸出到/home/nmon/log。

?

?

范例1:

自動按天采集數據:
在 crontab 中增加一條記錄:
0 0 * * * nmon -s300 -c288 -f -m /home/ > /dev/null 2>&1:
300*288=86400 秒,正好是一天的數據。

范例2:
A、執行命令:#crontab -e
B、在最后一行添加如下命令:
?0 8 * * 1,2,3,4,5 /nmon/scriptn/mon_x86_rhel52 -f -N -m /nmon/log -s 30 -c 1200
表示:
?周一到周五,從早上08點開始,監控10個小時(到18:00整為止),輸出到/nmon/log

?

三、nmon數據采集

1、數據采集

為了實時監控系統在一段時間內的使用情況并將結果記錄下來,我們可以通過運行以下命令實現:

?

1 #./ nmon -f -t -s30 -c 180

參數說明:

l-f:按標準格式輸出文件:_YYYYMMDD_HHMM.nmon;

l-t:輸出中包括占用率較高的進程;

l-s30:每30秒進行一次數據采集

l-c180:一共采集180次

輸入命令回車后,將自動在當前目錄生成一個hostname_timeSeries.nmon的文件,nmon輸出文件的命名方式是服務器名_日期時間.nmon

特別說明:

該命令啟動后,會在nmon所在目錄下生成監控文件,并持續寫入資源數據,直至360個監控點收集完成——即監控1小時,這些操作均自動完成,無需手工干預,測試人員可以繼續完成其他操作。如果想停止該監控,需要通過“#ps–ef|grepnmon”查詢進程號,然后殺掉該進程以停止監控。

?

[root@localhost source]# hostname

linux_test

[root@localhost source]# ./nmon -f -s 10 -c 60

[root@localhost source]# ps -ef | grep nmon

root 17815 1 0 08:22 pts/1 00:00:00 ./nmon -f -s 10 -c 60

root 17888 6977 0 08:22 pts/1 00:00:00 grep nmon

[root@localhost source]# ls linux_test_120724_0822.nmon

linux_test_120724_0822.nmon

當我們執行相關命令后,就在當前目錄生成了一個以本主機名linux_test開頭,以執行日期為規則,nmon結尾的文件,并且我們通過ps命令會看到相關的nmon進程,這里我們當10分鐘過去以后我們會發現相關nmon進程會消失的。

linux_test_120724_0822.nmon 就是我們所生成的數據文件,所有的信息都在linux_test_120724_0822.nmon記錄,而且我們通過more命令后發現都是我們看不懂的一些文本,這就需要我們把其轉化成我們能看懂的excel格式的文件。首先我們把linux_test_120724_0822.nmon文件導出到我們的windows本地。然后我們從http://www.ibm.com/developerworks/wikis/display/Wikiptype/nmonanalyser

下載nmonanalyser軟件到windows本地。打開解壓縮文件后,我們會發現2個文件,一個是nmonanalyse的說明word格式的說明穩定,另一個是excel格式的nmonanalyse文件。

我們這里所需要的就是excel格式問文件,我們雙擊打開。

?

四.生成圖形化結果

下載 nmon analyser (生成性能報告的免費工具):

下載地址:

https://www.ibm.com/developerworks/community/wikis/form/anonymous/api/wiki/61ad9cf2-c6a3-4d2c-b779-61ff0266d32a/page/b7fc61a1-eef9-4756-8028-6e687997f176/attachment/721e9797-b5fc-41d7-9b2f-5bd2aa2c8f7d/media/nmon_analyser_34a.zip

?

解壓之后可以得到一個nmon analyser v34a.xls,

雙擊打開nmon analyser v34a.xls;

設置excel宏的安全級別調低
選中上圖中的選項,

打開nmon_analyser中的excel,然后載入nmon數據,保存輸出的excel即可。

?

?

動態顯示:







 1、概述

  監控,在檢查系統問題或優化系統性能工作上是一個不可缺少的部分。通過操作系統監控工具監視操作系統資源的使用情況,間接地反映了各服務器程序的運行情況。根據運行結果分析可以幫助我們快速定位系統問題范圍或者性能瓶頸點。

  nmon是一種在AIX與各種Linux操作系統上廣泛使用的監控與分析工具,相對于其它一些系統資源監控工具來說,nmon所記錄的信息是比較全面的,它能在系統運行過程中實時地捕捉系統資源的使用情況,并且能輸出結果到文件中,然后通過nmon_analyzer工具產生數據文件與圖形化結果。

  nmon所記錄的數據包含以下一些方面(也是我們在尋找問題過程中所關注的資源點):

  ● cpu占用率

  ● 內存使用情況

  ● 磁盤I/O速度、傳輸和讀寫比率

  ● 文件系統的使用率

  ● 網絡I/O速度、傳輸和讀寫比率、錯誤統計率與傳輸包的大小

  ● 消耗資源最多的進程

  ● 計算機詳細信息和資源

  ● 頁面空間和頁面I/O速度

  ● 用戶自定義的磁盤組

  ● 網絡文件系統

  另外在AIX操作系統上,nmon還能監控到其他的一些信息,如異步I/O等。

  2、下載安裝nmon

  如何獲取nmon呢?我們可以在IBM的官方網站上免費下載獲取,下載網址為:http://www.ibm.com/developerworks/wikis/display/WikiPtype/nmon。

  nmon的安裝步驟如下:

  1)用root用戶登錄到系統中;

  2)建目錄:#mkdir /test;

  3)把nmon用ftp上傳到/test,或者通過其他介質拷貝到/test目錄中;

  4)執行授權命令:#chmod +x nmon。

  3、nmon數據采集

  3.1 數據采集

  為了實時監控系統在一段時間內的使用情況并將結果記錄下來,我們可以通過運行以下命令實現:

  #./ nmon -f -t -s 30 -c 180

  n -f:按標準格式輸出文件:_YYYYMMDD_HHMM.nmon;

  n -t:輸出中包括占用率較高的進程;

  n -s 30:每30秒進行一次數據采集

  n -c 180:一共采集180次

  輸入命令回車后,將自動在當前目錄生成一個hostname_timeSeries.nmon的文件,如果hosname為test1,生產的文件為:test1_090308_1313.nmon。


  通過sort命令可以將nmon結果文件轉換為csv文件:

  # sort -A test1_090308_1313.nmon > test1_090308_1313.csv

  執行完sort命令后即可在當前目錄生產test1_090308_1313.csv文件。

  3.2 生成圖形化結果

  為了分析nmon監控獲得的結果,IBM還提供了相應的圖形化分析工具nmon_analyser,通過nmon analyser.xls工具可以把監控的結果文件轉換成excel文件,方便分析系統的各項資源占用情況。

  nmon analyser.xls工具的使用方法如下:

  (1)打開nmon analyser.xls工具;

  (2)調整excel宏安全性:工具-宏-安全性

  (修改安全級別與可靠發行商)

  (選擇)安全級別:低

  (勾上)信任所有安裝的加載項和模板

  (勾上)信任對于“Visual Baisc項目”的訪問

  (3)修改完后,確定-關閉nmon analyser.xls,重新打開;

  (4)點擊Analyse nmon data按鈕,加載之前下載的test1_090308_1313.csv文件。

  以下是分析結果的截圖:

  以上就是nmon的簡單描述與使用介紹,大家可以根據自己所采集到的結果分析系統的情況。






nmon for Linux?- nmon is short for?Nigel's performance?Monitor for Linux on?POWER, x86, x86_64, Mainframe & now ARM (Raspberry Pi)


STOP PRESS: nmon for Linux Hits 500,000 downloads July 2017

This systems administrator, tuner, benchmark tool gives you a huge amount of important performance information in one go. It can output the data in two ways

  • On screen?(console, telnet, VNC, putty or X Windows) using curses for low CPU impact which is updated once every two seconds. You hit single characters on you keyboard to enable/disable the various sorts of data.
    • You can display the CPU, memory, network, disks (mini graphs or numbers), file systems, NFS, top processes, resources (Linux version & processors) and on Power micro-partition information.
    • For lots of examples, see the "Screen shots" from the left menu.
    • As you can see on the left lmon12e now in colour
  • Save the data to a comma separated file?for analysis and longer term data capture.
    • Use nmonchart (from this website) to generate a Googlechart webpage.
    • Use this together with nmon Analyser Microsoft Excel spreadsheet, which loads the nmon output file and automatically creates dozens of graphs ready for you to study or write performance reports.
    • Filter this data, add it to a rrd database (using an excellent freely available utility called rrdtool). This graphs the data to .gif or .png files plus generates the webpage .html file and you can then put the graphs directly on a website automatically on AIX with no need of a Windows based machine.
    • Directly put the data into a rrd database or other database for your own analysis
  • Latest version nmon for Linux is?16f
    Download the?precompiled binaries?or?nmon source code

  • More details

    • nmon is a single binary for
      • each operating system (Red Hat, SUSE, Ubuntu, Fedora, OpenSUSE etc.) and
      • each platform (Power, Mainframe, arm, x86 or x86_64).
    • Installing is very easy - just start the right executable.
      • Or rename the version you need to?/usr/bin/nmon?and then type: nmon
    • Why use five or six tools when one free tool can give you everything you need!!
    • For the pre-compiled versions - click on?Download
    • For the source code & compiling - click on?Compiling nmon

    On-screen

    When using nmon via a terminal session you can see the performance data directly on the screen and updated every second. You should if possible, stretch the terminal window to be longer to see more stats at one time. Here is a sample example from a Raspberry Pi 2 running Ubuntu 15.10 and nmon v16b. I typed "cCUd" to display this data.

    For more screen shots take the left-hand side menu option?Screen shots?or click?Screen shots.


    Data Analysis

    Once you save the nmon data you have a number of options to analyse and graph the statistics:

    • nmonchart?tool/script - see left hand menu
      • Nigel's nmonchart tools is quick and simple to convert a nmon output file to a webpage file .html that you can open with a browser directly or add to a website to share.
      • It takes a second or too and generates very nice looking graphs.
      • It is implemented in Korn shell script so you can add features (please share your updates).
      • The Clever part is using the Google.com Charting Javascript Library and your browser to do the actual graphing.
      • This this works on your PC, tablet or even larger mobile phone regardless of operating system.
      • Click here to find out more?nmonchart
    • nmon Analyser Excel Spread-sheet Download
      • This is the original tool and been developed over many years by Stephen Atkins
      • You can request support via the?Performance Tools Forum
      • However, Linux users might not like the idea of using the Microsoft Excel Spreadsheet and automating the creation of graphs can be tricky.
      • Sample Graphs out of the many (see screen shots for more and larger examples:
      • CPU Compared to Disk I/O
      • Disk Read and Write with I/O per second
      • Hot Disk analysis with Average, Weighted Average and Peak values
      • Network Read (top half) and Write (bottom half) Transfer Rates
    • nmon Consolidator Excel Spread-sheet Download
      • This is a newer tool and can combine nmon output files. It is by Stephen Atkins
      • Again its Microsoft Excel Spreadsheet
    • nmon2rrd
      • Microsoft free tool
      • This tool uses the excellent rrdtool to generate all the graphs and a website .html file.
      • Download it from the?nmon for AIX Wiki
      • This allows the automated analysis on many machines and viewing via a Browser.

    Now - Open Source

    nmon for Linux is a single source code file of 5000 lines and single makefile. This will enable you to compile nmon for your precise Linux version (if you can't find what you want in the binaries) and open a few other possibilities:

    • Fixing my code - be gentle, please.
    • Removing magic numbers i.e. constants that can catch us out as machines get larger
    • Developing for some strange environments like machines with no disks, blades that boot from NFS, internal Linux based engines within disks subsystems, embedded machines.
    • Who knows we may get nmon for Linux within the Linux Distro's - any one know how to go about that?

    Thanks for your support, suggestions, testing and I hope this starts a whole new wave of development and interest.


    History

    • nmon for Linux?was an internal project at IBM for many years and was released to open source under GPL on 27th July 2009.
    • Sourceforge.net?is being used to host the project, see?http://sourceforge.net/projects/nmon
    • nmon for AIX?does has a similar online look, file format but was always complete different source code.
      • It is now integrated into AIX topas command from
        • AIX 5.3 TL09
        • AIX 6.1 TL02.
      • nmon for AIX is not open source.
      • For more information?nmon for AIX Wiki



    Documentation

    nmon of Linux Documentation?- Updated 21st Nov 2016

    Ha ha ha ha - you are joking right :-)


    This page contains the following sections:

  • Hardware and Linux Supported
  • Getting Started via YouTube Videos - Including nmon for Linux and nmon for AIX
  • Getting Started - If you prefer to read the absolute minimum
  • nmon Command help - nmon for Linux -? and -h command output for the full syntax
  • nmon Support
  • Other sources


  • Hardware and Linux Supported

    • Platforms = hardware
      • POWER
      • x86_64 = AMD64 - 63 bit
      • Mainframe
      • x86 = 32 bit dropping off rapidly
      • ARM Raspberry Pi 2+3
      • Others . . . include embedded processors running Linux
    • Linux Distro’s
      • Ubuntu
      • Debian
      • SUSE SLES
      • OpenSUSE
      • Red Hat RHEL
      • Fedora
      • Centos
      • Many others . . .


    YouTube?Videos about nmon for Linux

    Many People prefer to watch a YouTube Video to learn - here are the six videos on nmon for both Linux and AIX operating systems. These are all the details you need to know to use nmon well. Note: nmonchart creates a .html file of all the graphs - it is one of many nmon graphing tools.

    nmon for Linux

  • nmon for Linux Starter Pack?20 minutes
  • nmon for Linux Data Capture?15 minutes
  • nmonchart to graph your nmon data files?22 minutes
  • Many nmon users use both Linux and AIX so here are the AIX equivalent videos and many nmon for Linux user use the nmon Analyser (Microsoft Excel spreadsheet)

    nmon for AIX

  • nmon Starter Pack Monitoring Online?14 minutes
  • nmon Starter Pack for AIX Data Capture?15 minutes
  • nmon Starter Pack for AIX Analyser?10 minutes


  • nmon for Linux Getting Started - If you prefer to read the absolute minimum

    Below assumes that you are logged on your system, that you have renamed your nmon binary file to just "nmon", that the nmon file has execute permission (chmod ugo+x nmon) and it is in your PATH.

    Using nmon for Linux Online

    • Just start nmon for Linux with: nmon
    • To stop it, just type: q
    • To get on screen hints type: h
      • and h again to remove the hints
    • Most of the rest are toggled commands i.e. type c to see the CPU stats and type c again to remove CPU stats.
    • The various stats come out in a set order (you can't control this) starting with CPU then memory and finally top processes at the bottom as there can be many processes this tends to fill up the rest of the window
    • Note if you make the window larger you can see more lines of output - this works in X Windows, VMC and Putty.
    • For memory stats type m
    • For disk graphs type d and you will see a 50 column graph of the read and write busy percentages
    • For disk numbers type D and if you type D again you see different information eventually typing D will close this section
    • For top processes there are different modes for the order of displaying the processes and different information, See the top line of the Top Processes section for further details.

    Using nmon for Linux in data capture mode

    • Start by capturing a small sample file. Type: nmon -f -s2 -c 30
      • The -f means you want the data saved to a file and not displayed on the screen.
      • The -s 2 means you want to capture data every 2 seconds
      • The -c 30 means you want thirty data points or snap shots
      • This means that after a few seconds collecting the configuration nmon for Linux will run for 2 x 30 = 60 seconds and stop. At the end some further configuration data is collected.
    • As nmon for Linux start up briefly check you system and options and then disconnect from you terminal sessions.
    • It then runs like a daemon process in the background. The point is that if you log out or get disconnected then nmon will complete the data file capture - this is a good thing.
    • If you want to be sure nmon is still running you can't use a simple "ps" because it is not associated with your log on session. Use "ps -ef | grep nmon" instead.
    • It is a common mistake to try to start using nmon for Linux output file before nmon has finished - this results in a either incomplete data which messes up a later tool or if you asked to a longer time between snap shots then there may be no data actually in the file, which confuses all tools trying to analyse the file.
    • Once nmon for Linux has finished and to build confidence try:
    • Use: grep ZZZZ yourfile.nmon
      • This should output the one line for each snapshot and date and time it happened.
    • Edit the nmon file with vi. You will notice it is a simple text file The start of each line defines the content of the line and then the values are separated with commas. This means the file can be imported in to a spread sheet. If you want to manually import the file, make sure you sort the file first (with the um er "sort" command). This sort means all the lines of a particular type are together. A sort is not required by most of the nmon for Linux analysing tools as they perform the function themselves.



    Hint for nmon version 16d


    # ./nmon -h

    Hint for nmon version 16d

    Full Help Info : nmon -hOn-screen Stats: nmonData Collection: nmon -f [-s ] [-c ] [-t|-T]Capacity Plan : nmon -x

    Interactive-Mode:

    Read the Welcome screen & at any time type: "h" for more helpType "q" to exit nmon

    For Data-Collect-Mode

    -f Must be the first option on the line (switches off interactive mode)Saves data to a CSV Spreadsheet format .nmon file in then local directoryNote: -f sets a defaults -s300 -c288 which you can then modifyFurther Data Collection Options:-s time between data snapshots-c of snapshots before exiting-t Includes Top Processes stats (-T also collects command arguments)-x Capacity Planning=15 min snapshots for 1 day. (nmon -ft -s 900 -c 96)

    End of Hints


    Full Help Information for nmon 16d


    For Interactive and Data Collection Mode:

    User Defined Disk Groups (DG) - This works in both modesIt is a work around Linux issues, where disks & partitions are mixed up in /proc files& drive driver developers use bizarre device names, making it trick to separate them.-g Use this file to define the groups- On each line: group-name (space separated list)- Example line: database sdb sdc sdd sde- Up to 64 disk groups, 512 disks per line- Disks names can appear more than one group-g auto - Will generate a file called "auto" with just disks from "lsblk|grep disk" outputFor Interactive use define the groups then type: g or GFor Data Capture defining the groups switches on data collection

    Data-Collect-Mode = spreadsheet format (i.e. comma separated values)

    Note: Use only one of f, F, R, x, X or z to switch on Data Collection modeNote: Make it the first argument then use other options to modify the defaultsNote: Don't collect data that you don't want - it just makes the files too largeNote: Too many snapshots = too much data and crashes Analyser and other toolsNote: 500 to 800 snapshots make a good graph on a normal size screenRecommended normal minimal options: snapshots every 2 minutes all day:Simple capture: nmon -f -s 120 -c 720With Top Procs: nmon -fT -s 120 -c 720Set the directory: nmon -fT -s 120 -c 720 -m /home/nag/nmonCapture a busy hour: nmon -fT -s 5 -c 720 -m /home/nag/nmon

    For Data-Collect-Mode Options

    -f spreadsheet output format [note: default -s300 -c288]output file is _YYYYMMDD_HHMM.nmon-F same as -f but user supplied filenameNot recommended as the default file name is perfectThe other options in alphabetical order:-a Include Accelerator GPU stats-b Online only: for black and white mode (switch off colour)-c The number of snapshots before nmon stops-d To set the maximum number of disks [default 256]Ignores disks if the systems has 100's of disk or the config is odd!-D Use with -g to add the Disk Wait/Service Time & in-flight stats-f and -F See above-g User Defined Disk Groups (see above) - Data Capture: Generates BBBG & DG lines-g auto See above but makes the file "auto" for you of just the disks like sda etc.-h This help output-I Set the ignore process & disks busy threshold (default 0.1%)Don't save or show proc/disk using less than this percent-l Disks per line in data capture to avoid spreadsheet width issues. Default 150. EMC=64.-m nmon changes to this directory before saving to fileUseful when starting nmon via cron-M Adds MHz stats for each CPU thread. Some POWER8 model CPU cores can be different frequencies-N Include NFS Network File System for V2, V3 and V4-p nmon outputs the PID when it starts. Useful in scripts to capture the PID for a later safe stop.-r Use in a benchmark to record the run details for later analysis [default hostname]-R Old rrdtool format used by some - may be removed in the future. If you use this email Nigel-s Time between snap shots - with "-c count" decides duration of the data capture-t Include Top Processes in the output-T As -t plus it saves command line arguments in UARG section-U Include the Linux 10 CPU utilisation stats (CPUUTIL lines in the file)-V Print nmon version & exit immediatelyTo manually load nmon files into a spreadsheet:sort -A *nmon >stats.csvTransfer the stats.csv file to your PCStart spreadsheet & then Open with type=comma-separated-value ASCII fileThis puts every datum in a different cellNow select the data of one type (same 1st column) and graph itThe nmon Analyser & other tools do not need the file sorted.

    Capacity Planning mode - use cron to run each day

    -x Sensible spreadsheet output for one dayEvery 15 mins for 1 day ( i.e. -ft -s 900 -c 96)-X Sensible spreadsheet output for busy hourEvery 30 secs for 1 hour ( i.e. -ft -s 30 -c 120)-z Like -x but the output saved in /var/perf/tmp assuming root user

    Interactive Mode Keys in Alphabetical Order

    Start nmon then type the letters below to switch on & off particular statsThe stats are always in the same order on-screenTo see more stats: make the font smaller or use two windowsKey --- Toggles on off to control what is displayed ---b = Black and white mode (or use -b command line option)c = CPU Utilisation stats with bar graphs (CPU core threads)C = CPU Utilisation as above but concise wide view (up to 192 CPUs)d = Disk I/O Busy% & Graphs of Read and Write KB/sD = Disk I/O Numbers including Transfers, Average Block Size & Peaks (type: 0 to reset)g = User Defined Disk Groups (assumes -g when starting nmon)G = Change Disk stats (d) to just disks (assumes -g auto when starting nmon)h = This help informationj = File Systems including Journal File Systemsk = Kernel stats Run Queue, context-switch, fork, Load Average & Uptimel = Long term Total CPU (over 75 snapshots) via bar graphsL = Large and =Huge memory page statsm = Memory & Swap statsM = MHz for machines with variable frequency 1st=Threads 2nd=Cores 3=Graphsn = Network stats & errors (if no errors it disappears)N = NFS - Network File System1st NFS V2 & V3, 2nd=NFS4-Client & 3rd=NFS4-Servero = Disk I/O Map (one character per disk pixels showing how busy it is)Particularly good if you have 100's of disksp = PowerVM LPAR Stats from /proc/ppc64/lparcfgq = Quitr = Resources: Machine type, name, cache details & OS version & Distro + LPARt = Top Processes: select the data & order 1=Basic, 3=Perf 4=Size 5=I/O=root onlyu = Top Process with command line detailsU = CPU utilisation stats - all 10 Linux stats:user, user_nice, system, idle, iowait, irq, softirq, steal, guest, guest_nicev = Experimental Verbose mode - tries to make recommendationsV = Virtual Memory statsKey --- Other Interactive Controls ---+ = Double the screen refresh time- = Halves the screen refresh time0 = Reset peak counts to zero (peak highlight with ">")1 = Top Processes mode 1 Nice, Priority, Status3 = Top Processes mode 3 CPU, Memory, Faults4 = Top Processes mode 4 as 3 but order by memory5 = Top Processes mode 5 as 3 but order by I/O (if root user)6 = Highlights 60% row on Long Term CPU view7 = Highlights 70% row on Long Term CPU view8 = Highlights 80% row on Long Term CPU view9 = Highlights 90% row on Long Term CPU view. = Minimum mode i.e. only busy disks and processes shownspace = Refresh screen now

    Interactive Start-up Control

    If you find you always type the same toggles every time you startthen place them in the NMON shell variable. For example:export NMON=cmdrtn

    Other items for Interactive and Data Collection mode:

    a) To limit the processes nmon lists (online and to a file)either set NMONCMD0 to NMONCMD63 to the program namesor use -C cmd:cmd:cmd etc. example: -C ksh:vi:syncd

    Other items for Data Collection mode:

    b) To you want to stop nmon use: kill -USR2 c) Use -p and nmon outputs the background process pidd) If you want to pipe nmon output to other commands use a FIFO:mkfifo /tmp/mypipenmon -F /tmp/mypipe &tail -f /tmp/mypipee) If nmon fails please report it with:1) nmon version like: 16d2) the output of: cd /proc; cat cpuinfo meminfo partitions stat vmstat3) some clue of what you were doing4) I may ask you to run the debug version or collect data filesf) If box & line characters are letters then check: terminal emulator & $TERMg) External Data Collectors - nmon will execute a command or script at each snapshot timeThey must output to a different file which is merge afterwards with the nmon outputSet the following shell variables:NMON_START = script to generate CVS Header test line explaining the columnsGenerate: TabName,DataDescription,Column_name_and_units,Column_name_and_units ...NMON_SNAP = script for each snapshots data, the parameter is the T0000 snapshot numberGenerate: TabName,T00NN,Data,Data,Data ...NMON_END = script to clean up or finalise the dataNMON_ONE_IN = call NMON_START less often (if it is heavy in CPU terms)Once capture done: cat nmon-file data-file >merged-file ; ready for Analyser or other toolsThe nmon Analyser will automatically do its best to graph the data on a new Tab sheetDeveloper: Nigel Griffiths See http://nmon.sourceforge.net Feedback welcome - On the current release onlyNo warranty given or implied. Copyright GPLv3

    nmon for Linux Support

    If you:

    • need help running nmon or understanding the data saved
    • have a suggestion for improvements
    • have bug fixes or want to report errors
    • want to extend nmon for Linux on to other platforms
    • want to include nmon on your distro (not a problem, I would just like to know)

    getting in touch at the?nmon for Linux - Help Forum

    • Alternatively use the?Performance Tools Forum




    NMON_Analyser User Guide for V4.6


    Preface

    NMON_Analyser is designed to complement NMON (Nigel’s Monitor) in analysing and reporting performance problems; it produces graphs for virtually all sections of output created using the “spreadsheet output” mode of NMON as well as doing some additional analyses for ESS, EMC and FAStT subsystems. ?It will also work with files produced by topasout and with other tools that produce data in “NMON” format. It is written in VBA for Excel and will work with Excel 2007 or later. ???It may work also on Excel 2003 with the required Microsoft updates to support .xlsx files https://support.microsoft.com/kb/924074?wa=wsignin1.0

    ?

    NMON was originally written by Nigel Griffiths (nag@uk.ibm.com) and is now (since AIX 5.3 TL09 and AIX 6.1 TL02) part of topas. ?NMON_Analyser was originally written by Stephen Atkins with contributions from many people – including Ralf Schmidt-Dannert and Markus Fehling, both of IBM. ?Currently nmon Analyzer is maintained by Ron McCargar (mccargar@us.ibm.com.)

    ?

    Support for NMON_Analyser is provided on a best efforts basis. ?Please direct questions to the User Forum (see below) rather than contact the author direct. ???

    ?

    Links:

    NMON_Analyser home page https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/Power%20Systems/page/nmon_analyser

    User Forum https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000749&ps=25


    Collecting data using NMON

    Please make sure you have the latest versions of both NMON and NMON_Analyser before starting a new engagement. ?If you want automatic notification of a new release of the Analyser send a note to steve_atkins@uk.ibm.com?and I’ll add you to my distribution list. ?Nigel maintains a similar list for NMON.

    ?

    You will need to have root privileges in order to get a complete set of data on the BBBP sheet. ?In order to collect data for the DISKBUSY sheets you need to make sure that iostat data collection is enabled:

    chdev -l sys0 -a iostat=true

    ?

    For spreadsheet output mode (comma separated values) use the following flags when invoking nmon:

    -f ??????????spreadsheet output format [note: default -s300 -c288]

    ???????? Output file is _YYYYMMDD_HHMM.nmon

    -F?? same as -f but user supplied filename

    -c???? number of snapshots

    -d? ??? requests disk service and wait times (DISKSERV and DISKWAIT)

    -i ??Ignore processes using less than this amount of CPU when generating TOP section – useful for reducing data volumes

    -g file containing disk group definitions

    -l?? number of hdisks per sheet - defaults to 150, maximum 250. ?See notes

    -m?

    NMON changes to this directory before saving the file


    -r??? goes into spreadsheet file [default hostname]

    -s??? interval between snap shots

    -x? capacity planning (15 mins for 1 day = -fdt -s900 -c96)

    -t ????????include top processes in the output

    -T ????????as?–t plus saves command line arguments in UARG section

    -A ????????include data for async I/O (PROCAIO) sections

    -Dprevents DISK sections being produced (useful when Disk Groups are being used because there are too many hdisks to process)

    -E ????????stops ESS sections being produced (necessary when Disk Groups are being used because there are too many vpaths to process)

    -J prevents JFS sections being produced (prevents Excel errors when you have more than 255 filesystems)

    -Lincludes LARGEPAGE section

    -Ninclude NFS sections

    -Sinclude WLM sections with subclasses

    -Winclude WLM sections without subclasses

    -Yinclude SUMMARY section (very efficient alternative to –t?if PID level data is not required)

    ?

    ????????example: nmon_aix51 -F asterix.nmon -r Test1 -s6 -c12


    Notes:

    1.?The –f?(or –F) flag must appear first.

    2.?The value of the -l?flag controls the number of hdisks per sheet on the DISK sheets and per line on the BBBD sheet. ??There are two factors to consider when choosing this value. ?Excel has a limit of 256 columns per sheet; however, both NMON and NMON_Analyser use some columns, so the upper limit is really 250. ??The second factor is that Excel VBA has an upper limit of 2048 bytes for input line length. ??This particularly affects users of EMC systems that use long hdisk names (e.g. hdiskpower123). ??The default of 150 is safe for such systems. ??Other users may set the value to 250 in order to reduce the number of output sheets.

    3.?Consider the value of the -s?flag very carefully. ??The shorter the interval between snapshots, the more variable the values for each resource will be. ?If you use an interval of 1 second, don’t be surprised to see many of your disks hitting 100% busy for short periods. ?For normal monitoring, 10-minute intervals (-c 600) provide a good balance.

    4.?The graphs produced by NMON_Analyser look best when the number of snapshots (specified by the -c?flag) is 300 or less. ???

    5.?The TOP section (produced by specifying the -t?flag) can generate large amounts of output and the size of the output can grow exponentially if a large value is specified for the -c?flag. ??If you want the TOP section then specify no more than 250 snapshots - ideally less.

    Collecting data using topas (xmwlm)

    You need AIX V5.3 TL5 Service pack 4 with APAR IY87993 or later. ???Note that only the output using the –a?flag can be analysed. ?In particular, cross-partition statistics cannot be analysed; if you wish to get a report for the entire machine, collect data from each LPAR separately and then use NMON_Consolidator to merge the data. ?Commands like the following are required to collect the data.

    ?

    topasout -a /etc/perf/daily/xmwlm.yymmdd

    :

    cp /etc/perf/daily/xmwlm.yymmdd_01 hostname.ddmmyy.topasout.csv

    Using NMON_Analyser

    ·?FTP the input file to your PC – ideally using the ASCI or TEXT options to make sure that lines are terminated with the CRLF characters required by Windows applications.

    ·?Open the NMON_Analyser spreadsheet and specify the options you want on the “Analyser” and “Settings” sheets (see below). Save the spreadsheet if you want to make these options your personal defaults.

    ??Click on the "Analyse nmon data" button and find/select the .nmon file(s) to be processed. ?You may select several files in the same directory. ?If you wish to process several files in different directories you may wish to consider using the “FILELIST” option described below.

    ??You may see the message SORT command failed for “filename”?if the file has >65K lines and the filename (or directory name) contains blanks or special characters. ?Either rename the file/directory or just pre-sort the file before using the Analyser.

    Analyser options

    ?

    GRAPHSThe first option is?either ALL?or LIST. ??If the value is LIST?then only those sheets which appear in the LIST on the Settings sheet will have graphs drawn for them. ???This option is particularly useful if the graphs are to be printed/published or to reduce the amount of memory/fonts/disk space required when analysing files from large systems.

    The second is?either CHARTS, PICTURES, PRINT?or?WEB. ??The meaning of these are as follows:

    ·?CHARTS produce Excel charts in-place on the selected sheets

    ·?PICTURES graphs will be produced on a separate “Charts” sheets as pictures. ?Selecting this option can reduce the size of the output file by up to 90%.

    ·?PRINTimplies PICTURES. ?Pictures will be printed to the designated printer (see “Printing Options” below)

    ·?WEB implies PICTURES. ?Automated web publishing (see “Web Publishing” below)

    INTERVALSspecifies the first and last time interval to be processed. ??Intervals outside this range will be discarded after parsing. ??Note that these are numbers between 1-999999 and are not time values. ??Setting a value of 2 for the first interval is useful in discarding the very large numbers that often appear at the start of an NMON collection run with AIX. ??If you have used a splitter program on the input file, or if you are analysing data from a LINUX system, then you should leave this as 1.

    TIMESspecifies the first and last time/date to be processed. ??Samples outside this range will be discarded after parsing. ??They can be specified in any form recognised by Excel as time/date values; e.g. ?

    14:00:1016:15:30

    ?4-Aug-12 6-Aug-12

    18:00 28/6/1204:00

    ????Notes:?

    ·?Specifying a date without a time is the same as specifying a time of 00:00.

    ·?If the second time is less than the first then Analyser will assume the second time is in the next calendar day

    ·?If there are no qualifying intervals in the file a message will be issued:

    “Invalid values for FIRST/LAST - values reset to 1/999999”

    and the entire file will be processed.

    MERGEspecifying YES?here results in NMON_Analyser merging all of the input files to form a single file. ??The input files must be unsorted. By default the Analyser will delete the TOP and UARG sections during the merge process; specify TOP?to prevent this from happening but be aware that processing time will be increased and that if the TOP section exceeds the maximum number of lines per sheet (depending on the version of Excel) then data will be lost. Specify KEEP?to stop the merged file from being deleted at the end of the run. ?Specify ONLY?if you simply want to create a merged file for future analysis.

    PIVOTspecifying YES?here results in NMON_Analyser creating ?a Pivot Chart from the specified sheet after all other processing (including printing/publishing the other charts) has completed. ??See additional parameters on the “Settings” sheet.

    ESSspecifying NO?here results in NMON_Analyser bypassing the additional analysis performed for ESS subsystems. ??This will result in faster analysis and can allow larger files to be analysed successfully when “out of memory” errors occur.

    FILELISTthe name of a control file containing a list of nmon output files to be processed by the Analyser. ?Leave this field blank for normal operation. ?The name must be fully qualified (e.g. c:\nmon\testcases\filenames.txt). ??The names specified in the text file must contain full path information. ?Wildcard characters may be included in the filenames so long as they conform to Windows standards. For example:

    c:\nmon\testcases\*.nmon

    NB: if you save the spreadsheet with a value in this field, the Analyser will automatically begin execution the next time you open it. ?This is defined as “batch mode” (see Appendix). ?You can stop the execution by pressing Ctrl+Break or by deleting/renaming the Control File.

    Batch Processing Options

    The following fields can be found on the “Settings” sheet.

    ?

    REPROCChange this to NO?if you want to bypass processing of input files which may have been processed in a previous run. ??This is useful if you make use of wildcards in the batch control file. ?Note that the REPROC option only takes effect when you have specified more than one input file. ?????

    OUTDIRthe name of an existing directory in which output files will be saved by default. This is primarily intended for batch operation (see FILELIST?above) but also works to set the default directory for interactive sessions. ?If the directory does not exist (or OUTDIR is blank) then output files are saved back to the same directory as the corresponding input files.

    Example: C:\NMON\Analyser\Output\

    Formatting Options

    The following fields can be found on the “Settings” sheet.

    ?

    BBBFontEnter the name of a fixed pitch font to be used for formatting the BBBC and BBBP sheets. ?Courier?provides acceptable results.

    GWIDTH Change the values in this row to make the generated graphs bigger or smaller. ?The default value of 0 means that the Analyser will dynamically size the graphs according to your screen size, font settings or page size. ??Be careful not to set a value larger than your page width when printing.

    GHEIGHT? Change the values in this row to make the generated graphs bigger or smaller. ?If you specify a value here you must also specify a value for GWIDTH.

    LIST? A comma-separated list a sheets for which the Analyser is to draw graphs. ?Only used if the GRAPHS option is set to LIST. ??The list can contain any valid wildcard characters recognised by Excel, e.g. “EMC*” ???

    Note: graphs are always drawn for SYS_SUMM, CPU_SUMM and DISK_SUMM

    NOLISTThe default is?KEEP. ?If you change this to DELETE?then all sheets which do not appear in LIST will be deleted after analysis. ?This can dramatically reduce the size of files that are to be kept for long periods.?

    CPUmaxSpecifies the maximum number of CPUnnn, PCPUnnn and SCPUnnn sheets that will be generated. ???The default value of 0 will choose all sheets for an LPAR using dedicated processors and number equal to the SMT mode for an LPAR using shared processors. ???

    REORDERspecifying YES?here results in NMON_Analyser reordering the sheets to improve navigation to more relevant information.

    TOPDISKSthe maximum number of hdisks/vpaths to include on disk graphs. ?A value of 0 produces graphs containing all the hdisks on a sheet (up to 250). ??Graphs containing more than 50 hdisks will be automatically scaled to fit and may therefore exceed the size of the screen.

    xToDFormat to be used for timestamps on Time of Day graphs. ?Anything acceptable to Excel as a Number Format Code ?may be entered. ??Default is hh:mm. ?Note that ?the date is also available within the timestamp and you may therefore use something like dd-mmm-yy hh:mm if, for example, you have merged multiple NMON files together. ?If you use something other than the default string you may need to increase the value of GHEIGHT – experiment with different values if you don’t see what you expect.

    SORTDEFAULT ?This setting indicates if the 1st graph on 'default' sheets (ones not handled else where by the Analyzer) are sorted. ?Note:?If Yes, this will also reorder the columns.

    Pivot Chart

    These parameters are used to construct a pivot chart. ????The required parameters are: Sheetname, PageField, RowField, ColumnField, DataField and xlFunction (can be COUNT, SUM, MIN, AVG, MAX). ?This is primarily useful for the TOP and SUMMARY sheets but might prove useful for other, possibly user-supplied, data sheets. ??????

    Printing Options

    The following fields can be found on the “Settings” sheet. Note - these only take effect if you select PRINT?for the OUTPUT option on the Analyser sheet.

    LSCAPEChange to YES?if you want the Analyser to set the page orientation to Landscape. ??By default the Analyser will fit one chart per page when printing landscape. ?

    COPIESSet to the number of copies to be printed.

    PRINTERThe name of the printer. ?Specifying a value of PREVIEW will cause the Analyser to invoke the Excel print preview function - useful for testing. ???You may also specify DEFAULT to print to the system default printer or the name of network printer.

    The Analyser adds page headers and footers.

    Web Publishing Options

    The following fields can be found on the “Settings” sheet.

    PNGChange to NO?if your browser can’t handle the PNG graphics format. ?Graphics will be generated as GIF files.

    SUBDIRIf this is?YES then?all supporting files, such as background textures and graphics, are organized in a separate folder. If this is NO?then supporting files are saved in the same folder as the Web page.

    WEBDIRthe name of an existing directory in which HTML files will be saved by default. ?If the directory does not exist (or OUTDIR is blank) then output files are saved back to the same directory as the corresponding input file.?

    Example: C:\NMON\Analyser\HTML\??

    Interpreting the output sections

    Notes on the “Weighted Average” as used in the Analyser

    Several graphs produced by the Analyser show average, weighted average and maximum values. ??Although everyone understands averages and maximums, the concept of a weighted average is a little more difficult to grasp. ?

    ?

    One of the problems we are faced with in analysing sample data is that resources on the target system may be idle for long periods during the collection. ?For example, the NMON data collection may be started some time before the system reaches peak utilisation and may not be stopped until the workload being monitored has long since finished. ??Although this does not affect the maximums it can severely affect the accuracy of the averages.

    ?

    The idea of a weighted average is to apply a weighting factor to each snapshot to indicate how relevant that snapshot is to the average. ???In NMON_Analyser, we use the value of the measurement itself as the weighting factor. ??In effect, this produces a figure that shows how busy a resource is when it is active. ?For example: ?a database log disk is only active during the middle part of a benchmark. ??We record the following figures for %tm_act (DISKBUSY):

    ?

    Snapshot

    %tm_act

    1

    0

    2

    0

    3

    0

    4

    0

    5

    0

    6

    6.1

    7

    6.3

    8

    6.5

    9

    9.1

    10

    5.9

    11

    0

    12

    0

    13

    0

    14

    0

    15

    0

    ?

    The average for this set of data is 2.3 and the weighted average is 7.0. ??The weighted average gives a better picture of how busy the disk is while logging is taking place. ??NMON_Analyser uses the weighted average as a sort key when sorting the contents of disk sheets.

    ?

    Note that, occasionally, NMON generates very large numbers for the first interval and this heavily skews the weighted average values. ??If you see this problem occurring then change the ?value for the FIRST parameter to 2 in order to exclude the first interval completely. ??This is only a problem for AIX.

    SYS_SUMM

    This section is entirely generated by the Analyser and contains a useful summary of data taken from other sheets. ???Note the the avg/max values for User%, Sys%, Wait% and Idle% are independent and will not add up to 100%. ??The CPU% column shows User% + Sys % for each line. ??

    ?

    For non-partitioned or dedicated CPU partitions the graph shows the total CPU Utilisation (%usr + %sys) together with the Disk I/O rate (taken from the DISKXFER sheet) by time of day. ??For micro-partitions the graph shows the number of physical CPUs being used instead of CPU%.

    ?

    The value “Max:Avg” is simply the maximum value divided by the average. ?If monitored over a long period of time the value for CPU% can be useful in spotting a system reaching saturation level (the ratio will steadily decrease). ?If you have historical data then the value can also be useful in determining how much latent demand is present in a system running at saturation level.

    AAA

    The AAA section is generated by NMON at the start of the data collection and contains information about the system and NMON itself – contents vary by release, following is for 12e

    ?

    AIXthis is the release / maintenance level of AIX being used on the target system as reported by the lslpp?command.

    buildthe particular build of nmon used to collect this data

    commandthe command line used to invoke NMON and the date when it finished.

    cpus? the number of CPUs in the system and the number active at the start of data collection.

    datedate at the start of the collection.

    disks_per_linethe setting of the?-l?flag on the NMON command line or the default value.

    hardwarethe processor technology used in the target system.

    hostthe hostname of the target system.

    Intervalthe time (in seconds) between snapshots.

    kernelinformation from the kernel - useful in identifying the type of kernel (32-bit or 64-bit) and whether this is an LPAR.

    LPARNumberName the LPAR number followed by the name

    MachineTypemachine type and model of the system

    prognamethe name of the NMON executable.

    runnametaken from the NMON command line if specified using the -r?parameter, else defaults to hostname.

    SerialNumber the machine serial number

    snapshots?the number of snapshots - this is used by the Analyser. ??The Analyser will modify this value to match the number of snapshots actually found in the input file.

    stealthis value will be a 1 if running on Linux and Steal% CPU is provided in the CPUnnn and CPU_ALL data.

    subversion?detailed information about the nmon version used to collect this data – including the date and time it was created.

    timetime as shown by the system clock at the start of the collection. Also see date value.

    timestampsizethe number of characters used for timestamps in each record. ?The default is 5.

    TLthe Technology Level of the AIX release

    userthe name of the user executing the NMON command.

    versionthe version of NMON used to collect this data.

    VIOSthe release/maintenance level if ?this is a VIOS LPAR.

    analyser the version of NMON_Analyser used to generate the output file together with the elapsed time (in seconds) for processing this file.

    environmentthe version of Excel you are using.

    parms the values of ?most of the user options specified on the “Analyser” sheet.

    settings the values of ?most of the user options specified on the “Settings” sheet.

    elapsed the execution time of the Analyser. ?I use this for tuning.

    NodeNamethe name of the LPAR’s node name

    ?

    NMON_Analyser deletes the NOTES?lines generated by NMON.

    ?BBBB

    The BBBB sheet lists all of the disks listed in the ODM together with the capacity (in Gbytes) and the adapter type (SCSI/SSA/Fibre) as reported by lsdev. ??Note that some fibre-attached devices do not report their capacity to AIX.

    NMON_Analyser deletes the column containing the sort key generated by NMON on all BBB sheets.

    BBBC

    The BBBC sheet shows the output from the lspv?command for all local disks at the start of the data collection. The Analyser highlights the hdisk name using a bold font and sets the sheet to use the fixed-pitch font specified on the NLS sheet (default Courier) in order to improve readability.

    BBBD

    The BBBD sheet shows a list of all I/O adapters listed in the ODM together with the hdisks addressed through that adapter. ?

    BBBE

    The BBBE sheet contains data extracted from the lsdev?command and shows the mapping between vpaths and hdisks. ?NMON_Analyser uses this information to construct the ESSBUSY, ESSRSIZE and ESSWSIZE sheets.

    BBBG

    The BBBG sheet contains details of the NMON disk group mappings.

    BBBL

    The BBBL sheet is only produced if the operating system is running in a partitions and contains details of the configuration of the LPAR at the start of the collection run.

    BBBN

    The BBBN describes each network adapter in the system and shows the name, speed and MTU size.

    BBBP

    The BBBP sheet contains the un-interpreted output from the?emstat?and lsattr?commands. ?Note that to get output from these commands requires NMON to be running with root privileges.

    The Analyser sets the sheet to use the fixed-pitch font specified on the NLS sheet (default Courier) in order to improve readability. ?

    BBBR

    This sheet records dynamic LPAR reconfiguration events during the collection run.

    BBBV

    This sheet lists all of the volume groups present at the start of the collection run.

    CPUnnn

    These sheets show %usr, %sys, %wait?and %idle?by time of day for each logical processor. ?Note that for micropartitions the Idle%?and Wait% figures will include times when the physical processor was ceded to the shared pool.

    NMON_Analyser generates a graph and a column headed “CPU%” containing the sum of %usr?and %sys?for use on the CPU_SUMM sheet. ??The Analyser also adds blank intervals for CPUs that are varied online during the collection interval so that the graphs cover the entire collection period. ??

    ?

    If REORDER?is set to YES?on the Control sheet the Analyser will move all CPUnn sheets to the end of the file.

    CPU_ALL

    This sheet shows the average utilisation for all physical threads by time of day. ??Note that for micro-partitions CPU%?is a measure of utilisation vs the entitlement but is adjusted by libperfstat so that it never exceeds 100%; this makes the numbers virtually useless for analysis of uncapped partitions and you may choose to look at the charts on the LPAR sheet instead.??Note also that micro-partitions generally record very little Idle%?or Wait%?because they will normally cede their timeslice to other LPARs rather than waste CPU time waiting for work. ???If the CPUmax?value is lower than the number of physical threads in use then the graph title will show the number of threads excluded.

    ?

    The second graph shows the number of active CPUs by time of day and is useful in determining whether CPUs have been varied on/off during the collection period. ?If the system has SMT enabled then the Analyser shows logical CPUs rather than physical CPUs and the legend reports “SMT=ON”.

    CPU_SUMM

    The Analyser generates this sheet from data on the CPUnnn sheets. ??It gives a breakdown of CPU Utilisation by thread (logical processor) and by core over the collection period. ??The chart can be very useful in identifying situations in which the system is thread-starved (i.e. too few threads to fully utilise the logical processors) or where the workload is dominated by a small number of ?single-threaded processes.

    ?

    Note that if CPUs have been dynamically reconfigured during the collection period, these figures reflect only those intervals when the CPU was varied on.

    General notes for DISK, ESS, EMC, FASt and DG sheets

    The DISK sheets record device statistics for each hdisk in the system. ??If there are more than ?hdisks in the system (see "Collecting data using NMON" above) NMON generates multiple output sections. ??DISKBUSY will therefore contain device busy statistics for the first group of hdisks, DISKBUSY1 for the next, DISKBUSY2 for the next and so on. ?

    ?

    The Analyser will normally sort the contents of the sheet in ascending sequence using the weighted average values as a sort key. ?However, if a storage subsystem is detected then sorting of the DISK sheets is disabled and the ESS/EMC/FASt/DG sheet contents are sorted instead. ??

    ?

    If the number of hdisks (or vpaths) on the sheet exceeds the value specified for TOPDISKS?(see “Analyser options” above), the graph will only show information for the specified number of disks and a warning will appear in the graph title.

    ?

    If REORDER?is set to YES?on the Control sheet the Analyser will move all but the DISKBUSY, DISK_SUMM and DISKSERV sheets to the end of the file when a storage subsystem or Disk Group is detected. ??The rationale behind this being that the ESS/EMC/FASt/DG sheets contain the most useful data and that the DISK sheets merely replicate it. ??However, the DISKBUSY sheet is useful for checking the activity on system disks (normally local) and the DISK_SUMM sheet gives total data rates for the system (local + subsystem disks).

    ?Note that NMON and NMON_Analyser can only handle a maximum of 250 vpaths in a system. ?If you have more than this then you need to use the Disk Groups feature of NMON to select the vpaths that are of primary interest.

    EMC/PowerPath subsystems

    NMON_Analyser detects the presence of an EMC/PowerPath subsystem by scanning the input file for the string “hdiskpower” before starting the analysis. ??EMC PowerPath creates devices called “hdiskpowern” which each map to multiple hdisks. ??They are therefore comparable to the vpaths generated by ESS/SDD. ??However, unlike vpaths, hdiskpower devices appear to AIX (and, therefore NMON) as real disks and NMON therefore records their activity on the DISK sheets. ??

    ?

    NMON_Analyser removes all hdiskpower entries from DISK sheets and moves them to new sheets beginning with the letters EMC. ??For example, hdiskpower entries found on DISKBUSY1 are simply moved to a new sheet called EMCBUSY1. ?If a DISK sheet contains only hdiskpower devices, that sheet is simply renamed. ?

    ?

    All device statistics reported by the Analyser (e.g. System I/O rates) are correct and as NMON excludes hdiskpower activity from the IOADAPT statistics these figures are also correct. ?

    FAStT subsystems

    NMON_Analyser detects the presence of a FAStT subsystem by scanning the input file for the string “dac” before starting the analysis. ?They are handled in the same way as EMC/PowerPath subsystems except that the dac devices are moved to sheets with names beginning "FASt".

    Note that the Analyser is not able to correctly handle systems having both EMC and FASt subsystems.

    DGBUSY

    This sheet records the average value of device busy for each hdisk in the NMON Disk Group.

    DGREAD

    This sheet records the average data rate (Kbytes/sec) for read operations to each NMON Disk Group.

    DGSIZE

    This sheet records the average data transfer size (block size), in Kbytes, for read/write operations to each NMON Disk Group. ?

    DGWRITE

    This sheet records the average data rate (Kbytes/sec) for write operations to each NMON Disk Group.

    DGXFER

    This sheet records the total I/O operations per second to each NMON Disk Group.

    DISKBSIZE

    These sheets record the average data transfer size (block size), in Kbytes, for read/write operations on each hdisk in the system. ?If this number is not very close to the stripe size for the device there may be a problem that could be solved by increasing the value of numclust.

    DISKBUSY

    These sheets record device busy for each hdisk in the system. ?This is the same as the %tm_act?value recorded by iostat. Note that if this sheet contains all zero values then it means you forgot to enable iostat collection before starting nmon:

    chdev -l sys0 -a iostat=true

    DISKREAD

    These sheets record the data rate (Kbytes/sec) for read operations on each hdisk in the system.

    DISKSERV

    These sheets record the service times (in milliseconds) for read/write transfers to each hdisk in the system.

    DISKWAIT

    These sheets record the queue times (in milliseconds) for read/write transfers to each hdisk in the system.

    DISKWRITE

    These sheets record the data rate (Kbytes/sec) for write operations on each hdisk in the system.

    DISKXFER

    These sheets record the I/O operations per second for each hdisk in the system. ??This is the same as the tps?value recorded by iostat.

    DISK_SUMM

    The Analyser creates this sheet. ??It shows the total data rates (reads and writes) in Kbytes/sec plus total I/O rates for all hdisks in the system. ???The figures on this sheet are accurate for all systems including ESS, EMC, FASt and HDS configurations. ??

    ?

    These data are displayed on the chart; the IO/sec data are also graphically displayed on the AAA sheet.

    DONATE

    This sheet records physical processor usage and donation to the shared pool. ?Only present for dedicated LPARs running on ?POWER6 systems.

    EMCBSIZE/FAStBSIZE

    This sheet records the average data transfer size (blocksize), in Kbytes, for read/write operations to each esoteric device in a system using EMC/PowerPath or FAStT.

    EMCBUSY/FAStBUSY

    These sheets record device busy for each esoteric device in a system using EMC/PowerPath or FAStT. ?

    EMCREAD/FAStREAD

    These sheets record the data rate (Kbytes/sec) for read operations to each esoteric device in a system using EMC/PowerPath or FAStT.

    EMCWRITE/FAStWRITE

    This sheet records the data rate (Kbytes/sec) for write operations to each esoteric device in a system using EMC/PowerPath or FAStT.

    EMCXFER/FAStXFER

    These sheets record the I/O operations per second to each esoteric device in a system using EMC/PowerPath or FAStT.

    EMCSERV/FAStSERV

    The Analyser creates this sheet. ??It shows estimated services times (not response times) for each esoteric device over the collection interval. ??The service time is derived from the device busy and the transfer rate taken from the corresponding BUSY and XFER sheets. ??Intervals where the transfer rate is below SVCXLIM?are ignored in order to improve the accuracy of the estimate.

    ERROR

    This sheet shows all errors reported by nmon during the collection run. ??FIRST/LAST intervals are ignored for this sheet. ??Timestamps are not converted to time values.

    ESSBSIZE

    This sheet is only present if ESS?is set to YES?and records the average data transfer size (blocksize), in Kbytes, for read/write operations on each vpath in the system. The data on this sheet are calculated by NMON_Analyser as the average of the DISKBSIZE values for component hdisks as recorded on the BBBE sheet.

    ESSBUSY

    This sheet is only present if ESS?is set to YES?and records device busy for each vpath in a system using ESS. ?The data on this sheet are calculated by NMON_Analyser as the average of the DISKBUSY values for component hdisks as recorded on the BBBE sheet. ?

    ESSREAD

    This sheet records the data rate (Kbytes/sec) for read operations on each vpath in the system. ?This information is provided by NMON.

    ESSWRITE

    This sheet records the data rate (Kbytes/sec) for write operations on each vpath in the system. ?This information is provided by NMON.

    ESSXFER

    These sheets record the I/O operations per second for each vpath in the system. ??This information is provided by NMON.

    FCREAD

    This sheet records the data rate (MBytes/sec) for read operations on each Fibre Channel adapter in the system. ?

    FCWRITE

    This sheet records the data rate (MBytes/sec) for write operations on each Fibre Channel adapter in the system.

    FCXFERIN

    This sheet records the read operations per second for each Fibre Channel adapter in the system.

    FCXFEROUT

    This sheet records the write operations per second for each Fibre Channel adapter in the system. ??

    FILE

    This sheet contains a subset of the fields reported by NMON on the Kernel Internal Statistics panel. ??These are the same values as reported by the sar?command.

    ?

    All fields are rates/sec.

    igettranslations of i-node numbers to pointers to the i-node structure of a file or device.?This is reported as iget/s?by the sar -a?command. Calls to iget occur when a call to to namei has failed to find a pointer in the i-node cache. This figure should therefore be as close to 0 as possible. ?

    nameicalls to the directory search routine that finds the address of a v-node given a path name. This is reported as lookuppn/s?by the sar -a?command.

    dirblknumber of 512-byte blocks read by the directory search routine to locate a directory entry for a specific file. This is reported as dirblk/s?by the sar -a?command.

    readchcharacters transferred by read system call. This is reported as rchar/s?by the sar -c?command

    writechcharacters transferred by write system call. This is reported as wchar/s?by the sar -c?command.

    ttyrawchtty input queue characters. This is reported as rawch/s?by the sar -y?command.

    ttycanchtty canonical input queue characters. This field is always 0 (zero) for AIX Version 4 and later versions.

    ttyoutchtty output queue characters. This is reported as outch/s?by the sar -y?command

    ?

    NMON_Analyser produces two graphs - one showing rates/sec for readch?and writech?by time of day and one showing rates/sec for iget, namei?and dirblk. ?

    FRCA

    This sheet is only generated if FRCA is loaded on the target system. NMON_Analyser produces a graph showing the cache hit ratio (as a percentage). ?If FRCA is not loaded, NMON generates no data and the Analyser consigns the redundant header record to the StrayLines sheet.

    IOADAPT

    For each I/O adapter listed on the BBBC sheet, contains the data rates for both read and write operations (Kbytes/sec) and total number of I/O operations performed. On AIX 5.1 and later, this information is reported by the iostat -A?command. NMON_Analyser reorders the columns on the sheet for easier graphing.

    ?

    The Analyser generates three graphs. ??Note the area charts can be easily converted to line charts if required. ??Simply right click on the white space within the chart area, then select Chart Type>Line>OK.

    IP

    This sheet only appears for topasout. ?

    JFSFILE

    For each file system, this sheet shows what percentage of the space allocation is being used during each interval. ?These figures are the same as the %Used?value reported by the df?command. The column headings show the mount point; sheet BBBC can be used to cross-reference to the file system/LV. ?

    JFSINODE

    For each file system, this sheet shows what percentage of the Inode allocation is being used during each interval. ?These figures are the same as the %Iused?value reported by the df?command. The column headings show the mount point; sheet BBBC can be used to cross-reference to the file system/LV.

    LAN

    This sheet only appears for topasout. ?

    LARGEPAGE

    The graph shows Usedpages and Freepages over time.

    ?

    Columns on the sheet are as follows:

    ?

    Freepagesthe number of large pages on the free list.

    Usedpagesthe number of large pages currently in use.

    Pagesthe number of large pages in the pool.

    HighWaterthe maximum number of pages used since the last reboot.

    SizeMBthe size of a large page in Mbytes.

    ?

    LPAR

    The first ?graph shows the number of physical processors used by this partition vs the entitlement. ??For an uncapped partition the number of physical processors may exceed the entitlement but can never exceed the number of virtual processors allocated to the partition. ??For AIX the graph also shows the number of unfolded virtual processors (AIX will “fold” - stop dispatching work to - excess processors in order to minimise scheduling costs).

    ?

    Note that the ratio of physical processor to entitlement (shown as %entc in the output of the lparstat?command) will generally be higher than CPU%?on the CPU_ALL sheet. ??The reason for this is that a partition that is within its entitlement may wait for a short period of time before ceding a processor that enters an I/O wait or becomes idle. ??This can eliminate unnecessary context switches.

    ?

    The second graph shows CPU utilisation as a percentage of virtual processors – for AIX this is broken down in to usr%, sys% and wait%. ??This level of detail is not available for Linux or releases of NMON prior to version 12.

    ?

    The third graph is only present for AIX systems and shows cpu utilisation of the shared pool by this and other partitions. ??The area marked “UsedPoolCPU%” represents the percentage of the shared pool that has been used by this partition, while the area marked “other% “ represents the percentage used by all other partitions. ??Note that if the partition is not authorised to see utilisation of the shared pool then the pool will appear to be 100% utilised.

    ?

    Columns on the sheet are as follows:

    PhysicalCPUphysical cores consumed by AIX

    VirtualCPUsnumber of Virtual CPUs allocated to the LPAR

    logicalCPUsnumber of threads (i.e.?Virtual CPUs multiplied by SMT mode)

    poolCPUsnumber of cores in the pool that this LPAR occupies

    entitled the number of cores guaranteed to be available to this LPAR

    weightthe priority of this LPAR when competing for unused CPU cycles

    PoolIdle? the number of ?unused cores in the Pool

    usedAllCPU%? percentage of active cores in the machine that this LPAR is using

    usedPoolCPU? percentage of cores in the pool that this LPAR is using

    SharedCPU? 1 if the LPAR is not a dedicated CPU

    Capped? 1 if the LPAR is capped

    EC_User%? percentage of Entitlement used in User mode

    EC_Sys%? percentage of Entitlement used in System mode

    EC_Wait%? percentage of Entitlement waiting for I/O

    EC_Idle%? percentage of Entitlement used in Idle mode

    VP_User%? percentage of Virtual CPU used in User mode

    VP_Sys%? percentage of Virtual CPU used in System mode

    VP_Wait%? percentage of Virtual CPU waiting for I/O

    VP_Idle%? percentage of Virtual CPU spent in Idle mode

    Folded? no. of Virtual CPUs unused for efficiency reasons

    CPU_Pool_id? the Id of the CPU pool, useful if there are multiple pools

    MEM

    The main graph on this sheet shows the amount of Real Free memory in Mbytes by time of day. ?This would be the same as dividing the fre?values reported by vmstat?over the same interval by 256. The small graph shows the amount of real memory. This is useful in determining if dynamic reconfiguration has been used during the collection period. ?

    ?

    For AIX, other columns on the sheet are as follows:

    ?

    Real Freethe percentage of real pages on the free list.

    Virtual Freethe percentage of unallocated virtual slots on the paging spaces.

    Real Free (MB)the amount of memory on the free list in Mbytes.

    Virtual Free (MB)the amount of unallocated space on the paging spaces.

    Real Total (MB)the total amount of memory available to AIX.

    Virtual Total (MB)the total amount of space allocated for paging spaces. ?

    ?

    Note: you can calculate the amount of memory used during an interval simply by subtracting the Real Free (MB) value?from the?Real Total (MB) value. ???This will, however, include file pages. ??The graph on the MEMUSE sheet gives a more accurate assessment of memory used by programs (computational pages).

    MEMUSE

    Except for %comp, the values on this sheet are the same as would be reported by the vmtune?command. ?

    ?

    %numpermthe percentage of real memory allocated to file pages.

    %minpermvalue specified on the vmtune?command or system default of 20%. This will normally be constant for the run unless the vmtune?or rmss?commands are used during collection.

    %maxpermvalue specified on the vmtune?command or system default of 80%. This will normally be constant for the run unless the vmtune?or rmss?commands are used during collection.

    minfreethe minimum number of pages AIX is to keep on the free list. ?Specified on the vmtune?command or system default of maxfree - 8.

    maxfreethe maximum number of pages AIX is steal in order to replenish the free list. ?Specified on the vmtune?command or system default.

    %compThe percentage of real memory allocated to computational pages. NMON_Analyser calculates this value. ?Computational pages are those backed by page space and include working storage and program text segments. ??They exclude data, executable and shared library files.

    ?

    The Analyser generates two graphs. ?The first shows the split between computational and file pages by time of day. ?The second plots the values of %numperm, %minperm, %maxperm and?%comp.

    ??

    If %numperm?falls below %minperm?then computational pages will be stolen. ??If %numperm?rises above %maxperm?then computational pages cannot be stolen. ?Low values for both %minperm?and %maxperm?indicate that the system has been tuned for a database server. ??You may also want to check the setting of STRICT_MAXPERM?on the BBBP sheet (if present).

    MEMNEW

    The graph shows the allocation of memory split into the three major categories: pages used by user processes, file system cache, and pages used by the system (kernel).

    ?

    Process%the percentage of real memory allocated to user processes

    FSCache%the percentage of real memory allocated to file system cache

    System%the percentage of real memory used by system segments

    Free%the percentage of unallocated real memory

    User%the percentage of real memory used by non-system segments

    MEMPAGES4K/64K/16MB/16GB

    These sheets are only present for AIX. ?They show various statistics for different page sizes in use within the system. ???The Analyser will delete the sheets for pages sizes not currently in use.

    ?

    For the MEMPAGES64KB sheet a graph is drawn showing the use of ?both 4KB and 64KB pages within the system – starting with AIX V6.1 (and supporting hardware) these page sizes are selected dynamically based on memory access patterns.

    MEMREAL

    This sheet only appears for topasout. The Analyser adds a column showing Real Free memory in Mbytes. ??

    MEMVIRT

    This sheet only appears for topasout.

    NET?

    This sheet shows the data rates, in Kbytes/sec, for each network adapter in the system (including SP switch if present). ?This is the same as produced by the netpmon –O dd?command. ?NMON_Analyser adds one column for each adapter showing the total data rate (read + write) and two columns showing Total Read and Total Write. ?Note that the Total Write is calculated as a negative number for graphing. ?

    ?

    The Analyser generates three graphs. ??The first graph shows total network traffic broken down as Total-Read and Total-Write. ?The writes are shown below the X-Axis.

    ?

    Note the area chart can be easily converted to a line chart if required. ??Simply right click on the white space within the chart area, then select Chart Type>Line>OK.

    NETPACKET

    This sheet shows the number of read/write network packets for each adapter. This is the same as produced by the netpmon –O dd?command. ??

    NETSIZE

    This sheet shows the average packet size in bytes for each network adapter in the system.

    NFS sheets

    There are separate sheets for NFS2, NFS3 and NF4 client/server. ???The Analyser will delete empty sheets. ??

    PAGE

    This sheet has the paging statistics as recorded by NMON.

    ?

    faultsthe number of page faults per second. This is not a count of page faults that generate I/O, because some page faults can be resolved without I/O.

    pginthe total rate/sec of page-in operations to both paging space and file systems during the interval.

    pgoutthe total rate/sec of page-out operations to both paging space and file systems during the interval.?

    pgsinthe rate/sec of page-in operations from paging space during the interval. ?This is the same as the pi?value reported by vmstat. If pgsin?is consistently higher than pgsout?this may indicate thrashing. ??

    pgsoutthe rate/sec of page-out operations to paging space during the interval. This is the same as the po?value reported by vmstat.

    reclaimsfrom NMON 10 onwards this field is the same as the?fr?value reported by?vmstat and represents the number of pages/sec freed by the page-replacement routine.

    scansthe number of pages/sec examined by the page replacement routine. ?This is the same as the sr?value reported by vmstat. Page replacement is initiated when the number of free pages falls below minfree?and stops when the number of free pages exceeds maxfree.

    cyclesthe number of times/sec the page replacement routine had to scan the entire Page Frame Table in order to replenish the free list. ?This is the same as the cy?value reported by vmstat?but note that vmstat?reports this number as an integer whereas nmon?reports it as a real number.

    fsincalculated by the Analyser as pgin-pgsin?for graphing

    fsoutcalculated by the Analyser as pgout-pgsout?for graphing

    sr/frcalculated by the Analyser as scans/reclaims?for graphing

    ?

    NMON_Analyser produces two graphs. ??The first shows paging operations to/from paging space. ?The ideal here would be no more than 5 operations/sec per page space (see the BBBC sheet for details). ??The second graph shows the scan:free rate. ??Memory may be over-committed when this figure is >4 although you also need to examine the MEM and PAGE sheets as well.

    POOLS

    This sheet contains information about the shared pool in which the LPAR is running. ??Most of the data will only be present if ?“Allow performance information collection.” is set in the LPAR properties.

    ??

    shcpus_in_systhe number of cores allocated to the global shared pool

    max_pool_capacitythe maximum number of VPs defined for this pool

    entitled_pool_capacitythe entitlement for this pool (includes reserve entitlement)

    pool_max_timesame as max_pool_capacity?but may vary in value if the pool definition is changed during the collection run.

    pool_busy_timethe average number of cores in use by this shared pool during the interval

    shcpu_tot_timethe average number of cores available to the global shared pool (including shared dedicated resources?) during the interval

    shcpu_busy_timethe average number of core in use within the global shared pool

    pool_idthis identifier of this pool

    entitledthe entitlement of this LPAR

    PROC

    This sheet contains a subset of the fields reported by NMON on the Kernel Internal Statistics panel. ?The RunQueue?and Swap-in?fields are average values for the interval. All other fields are rates/sec:

    ??

    RunQueuethe average number of kernel threads in the run queue. This is reported as runq-sz?by the sar -q?command and is reported as RunQueue?on the nmon Kernel Internal Statistics panel. ??A value that exceeds 3x the number of CPUs may indicate CPU constraint.

    Swap-inthe average number of kernel threads waiting to be paged in. This is reported as swpq-sz?by the sar -q?command.

    pswitchthe number of context switches. This is reported as pswch/s?by the sar -w?command.

    syscallthe total number of system calls. This is reported as scall/s?by the sar -c?command.

    readthe number of read system calls. This is reported as sread/s?reported by the sar -c?command.

    writethe number of write system calls. This is reported as swrit/s?by the sar -c?command.

    forkthe number of fork system calls. This is reported as fork/s?by the sar -c?command.??

    execthe number of exec system calls. This is reported as exec/s?by the sar -c?command.

    rcvintthe number of tty receive interrupts. This is reported as revin/s?by the sar -y?command.

    xmtintthe number of tty transmit interrupts. This is reported as xmtin/s?by the sar -y?command.

    semthe number of IPC semaphore primitives (creating, using and destroying). This is reported as sema/s?by the sar -m?command.

    msgthe number of IPC message primitives (sending and receiving). This is reported as msg/s?by the sar -m?command.

    ?

    NMON_Analyser produces three graphs - one showing the average length of the RunQueue and the number of?swap-ins by time of day, another showing rates/sec for pswitch?and syscalls?by time of day and a third showing rates/sec for forks?and execs. ?

    ?

    The graph for forks/execs can be useful when monitoring web server systems. ?

    PROCAIO

    This sheet contains information about the number of asynchronous I/O processes available and active (i.e. using more than 0.1% of the CPU) . ????It also shows the amount of CPU being used by the AIO processes during the collection interval.

    ?

    Two graphs are produced. ??The second uses two y-axes. ??The number of running aio processes is shown against the first axis and the amount of cpu used is shown against the second.

    RAWCPUTOTAL, RAWLPAR

    These sheets contain a dump of various counters such as context switches and phantom interrupts. ?

    TCPUDP

    This sheet only appears for topasout.

    TOP

    This sheet is only generated if you specify the -t??flag on the NMON command line. ?The output is similar to that produced using the ps v?command. ??Note that, because of the limitation of having only 65,000 lines on a single sheet, some data may be omitted for very large files and this may mean that entire PIDs or even commands may be missing from the analysis.

    ?

    Note that data are only present for processes that consumed a significant amount of CPU during an interval. ?The TOP sheet does not represent a complete view of the system.

    ?

    NMON_Analyser does the following:

    ??reorders the columns for easier processing. ??

    ??Sorts the data on the sheet into COMMAND?name order - using TIME?as a minor sort key. ?

    ??Creates a table at the end of the sheet summarising the data by command name and used for graphing.

    ?

    You can see the detail section by scrolling to the top of the sheet. ??The summary table is largely obscured by the graphs and so you will need to move (or delete) them for easier viewing.

    ?

    PIDin the detail section this is the process ID of a specific invocation of a command. ?In the summary table this is the command name.

    %CPUin the detail section this is the utilisation of a single processor (rather than of the system) by that PID during the interval. In the summary table this is the average amount of CPU used by all invocations of the command during the collection period.

    %Usrin the detail section this is the average amount of User-mode CPU used by that PID during the interval.

    %Sysin the detail section this is the average amount of Kernel-mode CPU used by that PID during the interval.

    Threadsthe number of (software) threads being used by this command.

    Sizethe average amount of paging space (in Kbytes) allocated for the data section (private segment + shared library data pages) for one invocation of this command. ?This is the same as the SIZE?figure on the ps v?command. ?Note that if Size?is greater than ResData?it means some working segment pages are currently paged out.

    ResTextthe average amount of real memory (in Kbytes) used for the code segments of one invocation of this command. ??Note that multiple concurrent invocations will normally share these pages.

    ResDatathe average amount of real memory (in Kbytes) used for the data segments of one invocation of this command. ?A method of calculating real memory usage for a command is ResText?+ (ResData?* N).

    CharIOthis is the count of bytes/sec being passed via the read and write system calls. The bulk of this is reading and writing to disks but also includes data to/from terminals, sockets and pipes. Use this to work out which processes are doing the I/O.

    %RAMthis is an indication of what percentage of real memory this command is using. This is (ResText + ResData) / Real Mem; it is the same as the %MEM?value on the

    ps v?command. ???Due to rounding/truncation, and the large amounts of memory in modern systems, this is usually 0. ?

    Pagingsum of all page faults for this process. ?Use this to identify which process is causing paging but note that the figure includes asynchronous I/O and can be misleading.

    Commandname of the command

    WLMClassname of the Workload Partition or Workload Manager superclass to which this command has been allocated (64-bit kernel only). ??

    IntervalCPUgenerated by the Analyser. ??In the detail section this shows the total amount of CPU used by all invocations of a command in the time interval. ?It is calculated as the sum of CPU used by all PIDS running the same command divided by the number of active processors (physical cores) available during the interval. In the summary section this is broken down as Average, Weighted Average and Maximum and is used to generate the graph.

    WSetgenerated by the Analyser. ??In the detail section this shows the total amount of memory used by all invocations of a command recorded in the time interval. ?It is calculated as ResText?+ (ResData?* N) (where “N” is the number of copies of this command running concurrently during the interval). ?In the summary section this is broken down as Minimum, Average and Maximum and is used to generate the graph.

    Usergenerated by the Analyser if a UARG sheet is present. ?This contains the name of the user running the process. ?

    Arggenerated by the Analyser if a UARG sheet is present. ?This contains the complete argument string entered for the command.

    ?

    The Analyser generates four graphs using data in the generated table:

    ??A graph showing Average, Weighted Average and Maximum CPU Utilisation by command

    ??A graph showing Minimum, Average and Maximum Memory Utilisation by command

    ??A graph showing Average, Weighted Average and Maximum CHARIO by command

    ??A graph showing the CPU utilisation for each PID for each interval as a scatter chart. ?Note that this chart is only produced if there are fewer than 32,000 lines on the TOP sheet. See below for notes on interpreting this chart.

    Interpreting the %Processor by PID chart

    The purpose of the chart is to provide a link to the UARG sheet so that you can discover precisely which invocation of a command was responsible for using the CPU. It shows the processor utilisation (utilisation of a single CPU) by each PID captured on the TOP sheet. ?Note that a process can use more than 100% of a single CPU if it is multi-threaded. ??

    ?

    Active PIDs will create a cluster of points on the chart. ???The highest point will show the maximum amount of CPU used during any one snapshot. ??To find out which PID a point refers to, move the mouse to position the cursor above it and Excel will display a coordinate pair. ??The first coordinate is the PID – use this to refer to the UARG sheet to find precisely which command was being executed.

    UARG

    This sheet has the first 1,000 commands executed during the collection period. ??The commands are listed in time order. ??Note that commands appearing in the first interval may have been executing prior to the start of the collection.

    ?

    PIDthe process ID of a specific invocation of a command

    PPIDthe parent process ID

    COMMthe command being executed

    THCOUNTthe number of threads started by this process

    USERthe name of the user running this process

    GROUPthe name of the group to which the user belongs

    FullCommand the full command string entered by the user

    VM

    This sheet is only present for Linux systems and contains a dump of the /proc/vmstat file values.

    The two graphs shows file-backed paging (pgpgin/pgpgout) and swap space activity.

    WPAR sheets

    These sheets are only present for AIX V6 and record data for each Workload Partition in the system. ???They are not currently generated by the topas version of nmon.

    WLM sheets

    Contain details of CPU, Memory and I/O bandwidth used by each Superclass/Subclass defined to WLM during the collection run. ??The Analyser will extract subclasss data and create a new set of sheets for each class with more than one subclass. ?These sheets will be named “WLMCPU.class” etc. ?

    If this is a Micro-partition then the Analyser will also create a set of WLMPCPU sheets which will show the physical processor utilisation rather than %CPU utilisation.

    ZZZZ

    The Analyser uses the information on this sheet to automatically convert all NMON time stamps to actual time of day for easier analysis. For NMON10 or later a column is added which contains the date and time as a single value and this is used for the conversions. ?The number of rows on this sheet is used by the Analyser to reset the “snapshots” value on the AAA sheet in case the nmon run was terminated with SIGUSR2.

    ?Error Handling

    Error handling in NMON_Analyser is rudimentary. ???The Analyser can handle many input file errors, but occasionally the analysis will halt leaving you staring at a dialog box. ?Should this happen, please accept my apologies. ?However, before sending me a copy of the input file, please read the following:

    Common problems

    The most commonly reported problems arise from invalid input files. ??We also get problems reported where, for whatever reason, lines have been truncated, split or even duplicated. ??NMON_Analyser attempts to trap these errors and will report them on the “StrayLines” sheet. ?Check this sheet if the run stops unexpectedly.

    ?

    ??'No valid input! NMON run may have failed.'

    ?

    The most common cause of this message is that the NMON run failed and there really is no valid input. ??NMON initialises the output file by writing all of the section headings. ??If it subsequently fails, you will get an output file that consists purely of headings ?- with no data. ??Check the file by loading it into a word processor or, indeed, a spreadsheet (as a .csv file) before you send it to me.

    ?

    ??'Unexpected end of file.'

    ?

    This is only reported when processing files containing more ?lines than can be stored on one sheet and when SORTINP is set to NO. It is usually caused by the fact that lines ?are being terminated with a CR character instead of CRLF. ?Change your FTP option to ASCII ?or TEXT when sending the file to your PC. ?This problem only shows up with large files because the Analyser uses a different technique to read them than that used to read smaller files.

    ?

    Strangely, one of the most common problems I get is caused by the fact that the Analyser can’t handle files with a single data interval ?very well. ??If you want to test the package just let NMON run for a few minutes to get a reasonable data sample!

    Known bugs/problems (V3.4, topas_nmon)

    ??When analysing systems with a very large number of disks, Excel can stop with error “No more fonts can be added” or “Insufficient Resources”. ??Set GRAPHS?to LIST?and select only those sheets you are interested in using the LIST?option on the Settings sheet.

    ??The data can be sorted incorrectly with some versions of nmon (notably 14g) which generate variable length timestamps. ???Process each file separately and specify MERGE=YES to get the data sorted correctly.

    ??The PIVOT option does not seem to work with Excel 2007 or later

    ??The Analyser will crash if you edit the .csv file using Excel prior to the run and the file contains a TOP section. ??If you need to edit the input file, use a word processor.

    ??There are some issues with processing files from systems having both ESS and EMC subsystems attached. ?

    ??When analysing ESS subsystems with more than 253 vpaths some vpath data will be missing from the output. ?Use NMON Disk Groups to combine several vpaths into a single unit for reporting and use the –E?flag to prevent the ESS sections from being produced. ???Alternatively, set ESS?to NO?on the Control sheet and ignore warning messages about data truncation – only the first 253 vpaths will appear in the output.

    How to report a problem?

    Post the relevant information on the nmon forum (see the link in the introduction). ?It can help to include a copy of the original, unmodified?.nmon input file, plus the incomplete output spreadsheet, as a compressed (zipped) file but please consider your system security before doing so.

    ?

    If you have the ability to capture a screenshot then a copy of any dialog boxes also proves useful on occasion.

    ?

    Note:?Development is currently done on Microsoft Excel 2003 (11.8320.8221) SP3. ?It may not be possible to fix problems arising from the use of different releases.

    Excel/VBA Resources/Links

    This is a good source for Excel tips and it has some VBA examples as well:

    http://exceltips.vitalnews.com/

    ?

    This is the home of an excellent reference book:

    http://www.exceltip.com/

    ?

    And of course

    http://www.microsoft.com


    Appendix: Notes on Batch Operation

    If you regularly process large numbers of files, the operation of NMON_Analyser can be completely automated. ???Simply create a text file containing a list of nmon file names (using wild card characters as appropriate) and enter the name of this file into the FILELIST field of the Analyser control sheet. ???Specify the name of an existing directory in the OUTDIR field if you want all of the output files to end up in one place. ??Save the NMON_Analyser spreadsheet under a new name (this is recommended so that you can still use NMON_Analyser for interactive sessions). ?

    Now create Windows .bat files to invoke Excel (see the samples below). ??

    ?

    After processing the last input file, the Analyser will automatically close down Excel. ??Note, however, that this only happens if you load a copy of the Analyser that has a saved FILELIST name and?if there are no other open spreadsheets. ??This allows you to use the FILELIST option safely during an interactive session.

    Sample .bat files

    These sample batch control files are designed to use the pscp file from the PuTTY suite written and maintained by Simon Tatham at http://www.chiark.greenend.org.uk/~sgtatham/putty/

    My thanks to Jamie Dennis for providing them.

    getcsv.bat

    cd \NMON\RawData

    del *.csv

    d:\progra~1\putty\pscp -p -r -l userid host://Performance/NMON/Rawdata/*.csv .

    analyser.bat

    D:

    cd \NMON\FinishedData

    del d:\NMON\FinishedData\*.xls

    "D:\NMON\nmon analyser batch.xls"

    putxls.bat

    D:

    cd \NMON\FinishedData

    d:\progra~1\putty\pscp -p -r -l userid *.xls host:/Performance/NMON/

    Control.txt

    d:\NMON\RawData\*.csv

    NMON Analyser Batch.xls

    OUTDIR

    d:\NMON\FinishedData\

    FILELIST

    d:\NMON\FinishedData\Control.txt







    About Me

    ...............................................................................................................................

    ● 本文整理自網絡

    ● 本文在itpub(http://blog.itpub.net/26736162)、博客園(http://www.cnblogs.com/lhrbest)和個人微信公眾號(xiaomaimiaolhr)上有同步更新

    ● 本文itpub地址:http://blog.itpub.net/26736162/abstract/1/

    ● 本文博客園地址:http://www.cnblogs.com/lhrbest

    ● 本文pdf版及小麥苗云盤地址:http://blog.itpub.net/26736162/viewspace-1624453/

    ● 數據庫筆試面試題庫及解答:http://blog.itpub.net/26736162/viewspace-2134706/

    ● QQ群:230161599???? 微信群:私聊

    ● 聯系我請加QQ好友(646634621),注明添加緣由

    ● 于 2017-07-01 09:00 ~ 2017-07-31 22:00 在魔都完成

    ● 文章內容來源于小麥苗的學習筆記,部分整理自網絡,若有侵權或不當之處還請諒解

    ● 版權所有,歡迎分享本文,轉載請保留出處

    ...............................................................................................................................

    拿起手機使用微信客戶端掃描下邊的左邊圖片來關注小麥苗的微信公眾號:xiaomaimiaolhr,掃描右邊的二維碼加入小麥苗的QQ群,學習最實用的數據庫技術。





    總結

    以上是生活随笔為你收集整理的【OS】NMON的简介和使用的全部內容,希望文章能夠幫你解決所遇到的問題。

    如果覺得生活随笔網站內容還不錯,歡迎將生活随笔推薦給好友。

    av福利超碰网站 | 超碰人人草| 久久综合五月天婷婷伊人 | 超碰人人在线 | 噜噜色官网 | а天堂中文最新一区二区三区 | 国产精品二区在线 | 91视频在线看 | 日韩资源在线播放 | 久久一区精品 | 69夜色精品国产69乱 | 国产成人黄色在线 | 久久影视精品 | 日韩影视在线 | 国产成人精品国内自产拍免费看 | 欧美日一级片 | 麻豆成人精品视频 | 青青草国产精品 | 999久久久国产精品 高清av免费观看 | 91在线你懂的 | 日本精品在线视频 | 精品国产免费人成在线观看 | 国产+日韩欧美 | 亚洲一级特黄 | 99精品在线免费视频 | 成 人 黄 色 免费播放 | 国产高清视频在线播放 | 99久e精品热线免费 99国产精品久久久久久久久久 | 日日夜夜中文字幕 | 亚洲91中文字幕无线码三区 | 欧美夫妻生活视频 | 欧美极品少妇xxxxⅹ欧美极品少妇xxxx亚洲精品 | 国产黄在线播放 | 色黄www小说 | 亚洲成人xxx | 免费av黄色 | 日本深夜福利视频 | 在线看av的网址 | 欧美综合国产 | 亚洲综合激情 | 日韩精品一二三 | 国产一区二区手机在线观看 | 狠狠狠色丁香综合久久天下网 | 97色se | 亚洲精品美女 | 超碰在线公开免费 | 久久久福利视频 | 国产在线播放观看 | 国产精品永久久久久久久久久 | 亚洲免费观看在线视频 | 看毛片网站 | 日韩r级在线 | 成人av av在线| 国产无套精品久久久久久 | 97精品视频在线播放 | 国产精品成人自产拍在线观看 | 免费视频久久久 | 日狠狠 | 国内精品久久久久影院一蜜桃 | 五月天.com | 狠狠色丁婷婷日日 | 亚洲成人av一区二区 | av电影av在线 | 超碰在线1 | 日韩久久久久久久久久久久 | 国内视频在线 | 涩涩网站在线看 | 国产破处在线播放 | 一区二区三区免费在线 | 国产成人一区二区三区 | 国产精品久久久久久久久婷婷 | 97国产精品一区二区 | 欧美男女爱爱视频 | 欧美日韩国产在线精品 | 欧美成人69av| 婷婷色在线播放 | 91伊人久久大香线蕉蜜芽人口 | 国产精品剧情在线亚洲 | 久久国精品 | 欧美极品少妇xxxx | 在线观看av网站 | 四虎精品成人免费网站 | 在线观看国产一区二区 | 欧美另类高潮 | 天天夜操 | 天天综合导航 | 欧美精品九九99久久 | 国产一区免费在线 | 中文字幕婷婷 | 成人h在线| 九九国产精品视频 | 欧美aaa一级 | 三级视频片 | 超碰97久久 | 五月婷网站 | 新版资源中文在线观看 | 一区二区在线影院 | 91精品久久久久久久久久久久久 | 狠狠狠色狠狠色综合 | 最近免费在线观看 | 99精品免费久久久久久日本 | 国产亚洲精品久久久久秋 | 人人玩人人爽 | 超碰97人人爱 | 五月天最新网址 | 亚洲成av人片在线观看无 | 一区二区三区在线视频111 | 不卡视频在线看 | 91在线麻豆 | 中文字幕在线观看av | 九九久久精品视频 | 99精品欧美一区二区 | 亚洲国产欧美在线看片xxoo | 日本三级国产 | 激情九九| 狠狠操狠狠操 | 伊人首页| 涩涩伊人| 国产字幕在线看 | 91视频麻豆视频 | 久久伊人国产精品 | 日韩av成人在线观看 | 日韩av一区二区在线 | 天天草网站 | 久久久久免费精品国产 | 美女免费av| 日本资源中文字幕在线 | 国产精品久久久久久久久久三级 | 精品久久久久久久久久久久久久久久 | 丁香在线| 日韩系列在线 | 国产高清视频色在线www | 成人三级黄色 | 免费国产在线观看 | 98久9在线 | 免费 | 激情网综合| 亚洲国产高清在线观看视频 | 久久一区精品 | 亚洲精品国偷拍自产在线观看蜜桃 | 国产一区二区三区视频在线 | 在线视频观看国产 | 麻豆视频国产 | 在线不卡的av | www视频在线播放 | a色视频| 97色婷婷人人爽人人 | 狠狠色丁香久久综合网 | 国产精品18久久久久久首页狼 | 射射色 | 国产成人在线一区 | 992tv人人网tv亚洲精品 | 国产香蕉久久精品综合网 | 黄色成人免费电影 | 91久久奴性调教 | 久久99精品久久久久久清纯直播 | 色婷婷色 | 成人网444ppp | 五月婷婷在线观看 | 国产va饥渴难耐女保洁员在线观看 | 天天综合网 天天综合色 | 狠狠色噜噜狠狠狠 | 国产精品久久久久9999吃药 | 久久精品在线视频 | 精品久久一级片 | 久久免费一级片 | 成年人在线看片 | 日韩在线中文字幕视频 | 国产不卡免费av | 免费观看成人网 | 色婷婷狠狠五月综合天色拍 | 欧美另类老妇 | 国产精品亚洲片在线播放 | 欧美一级电影免费观看 | 久久网站最新地址 | 久久艹精品 | 激情五月婷婷综合网 | 亚洲精品中文字幕视频 | 日韩午夜剧场 | 国产精品黄色在线观看 | 久草在线观看视频免费 | 欧美坐爱视频 | 狠狠狠干狠狠 | 日韩在线高清 | 成人久久久久久久久久 | 色欧美成人精品a∨在线观看 | 91精品伦理 | 久久精品老司机 | 人人爽人人做 | 操老逼免费视频 | 免费在线看v | 特级毛片爽www免费版 | 亚洲影音先锋 | 成人精品福利 | japanesexxxxfreehd乱熟 | 亚洲视频精品在线 | 麻豆va一区二区三区久久浪 | 欧美日韩国产在线一区 | 色99之美女主播在线视频 | 欧美资源在线观看 | 日韩视频在线不卡 | 中文字幕在线观看你懂的 | 亚洲高清91 | 99精品在这里 | 久久久影视 | 亚洲精品日韩av | 99免费观看视频 | 91色综合 | 99精品在线播放 | 日本中文字幕在线观看 | 热久精品 | 欧美坐爱视频 | 一级全黄毛片 | 人人看人人做人人澡 | 黄色三级免费看 | 亚洲综合成人在线 | 国产尤物视频在线 | 午夜国产一区二区 | 亚洲a资源 | 婷婷日韩| 国产视频手机在线 | 五月天激情婷婷 | 99久高清在线观看视频99精品热在线观看视频 | 天天草天天干天天 | www.黄色在线 | 日韩爱爱片| 手机看片中文字幕 | 国产日韩欧美自拍 | 在线观看黄网站 | 四虎免费av| 亚洲黄a| 99热精品国产一区二区在线观看 | 婷婷在线网站 | 99精彩视频在线观看免费 | 黄色国产大片 | 九九久久久久久久久激情 | 色国产精品 | 久久国产精品久久久久 | 免费在线观看黄 | 中文字幕一区二区三区在线视频 | 97热久久免费频精品99 | 日韩一区正在播放 | 日韩电影中文字幕在线 | 国产精品videossex国产高清 | 亚洲日本韩国一区二区 | 久久久影院一区二区三区 | 久久夜色精品国产欧美乱极品 | 视频在线99re | 午夜视频一区二区三区 | 中文字幕乱码日本亚洲一区二区 | 亚洲视频在线观看免费 | 国产一级二级三级在线观看 | 欧美日韩免费观看一区二区三区 | 五月天婷婷视频 | 国产在线视频一区 | 综合国产在线 | 一性一交视频 | 91亚洲精品久久久久图片蜜桃 | 男女啪啪视屏 | 狠狠色狠狠色综合系列 | 成人理论在线观看 | 欧美一级欧美一级 | www日韩视频 | 一级黄视频 | 中文字幕日韩在线播放 | 91精品国产高清自在线观看 | 国产精品999久久久 久产久精国产品 | 99热精品国产一区二区在线观看 | 在线观看免费91 | 91九色最新地址 | 亚洲精品国产精品乱码在线观看 | 天天骚夜夜操 | 亚洲在线视频免费 | 久久xxxx| 四虎国产精品成人免费影视 | 欧美激情第八页 | 91视频最新网址 | 亚洲精品视频免费在线 | 91av电影| 国产一级在线视频 | 亚洲免费成人 | 中文字幕电影高清在线观看 | 最新日韩在线观看视频 | 欧美中文字幕第一页 | 亚洲欧美视频在线播放 | 国产一区成人 | 国产一卡二卡四卡国 | 91精品视频观看 | 欧美日韩国产综合一区二区 | 欧美日韩高清在线一区 | 五月天婷婷视频 | 五月婷影院 | 欧美国产三区 | 国产美女免费观看 | 99视屏| 日本中文字幕在线电影 | 在线黄av| 麻豆视频免费入口 | 99久久精品午夜一区二区小说 | 亚洲伊人网在线观看 | 日韩精品久久久久 | 成人在线视频论坛 | 亚洲欧美国产精品va在线观看 | 在线观看免费一级片 | a天堂免费| 亚洲作爱| 最新色视频 | 欧美日韩国产精品爽爽 | 久久久久久久久免费视频 | 国产黄av| 国产成人精品亚洲精品 | 国产欧美日韩一区 | 六月色 | 美女黄视频免费 | av电影在线免费观看 | 久久视频热 | www.亚洲精品在线 | 日本久久综合网 | 午夜精品一区二区三区视频免费看 | 日韩欧美精品一区二区三区经典 | 波多野结衣在线观看一区二区三区 | 欧美性大战久久久久 | 亚洲午夜久久久久久久久 | 久草网视频 | 亚洲成色777777在线观看影院 | 日韩免费网址 | 国产精品成人一区二区 | 91| 日韩精品一区二区在线视频 | 中文字幕av在线电影 | 在线 高清 中文字幕 | 日韩免费中文字幕 | 97视频网站 | 国产成人精品免费在线观看 | 九九热精 | 久久久久欠精品国产毛片国产毛生 | 成人免费一区二区三区在线观看 | 日韩av网站在线播放 | 四虎国产精品成人免费4hu | 日韩电影一区二区三区在线观看 | avav99| 自拍超碰在线 | 国产精品情侣视频 | 在线免费观看欧美日韩 | 91成人精品视频 | 人人舔人人 | 91色国产在线 | 免费av在线播放 | 人人超碰人人 | 成人在线电影观看 | 精品国产一区二区在线 | 99免费看片| 久久五月婷婷丁香 | 国产精品免费久久久久久 | av先锋影音少妇 | 免费男女羞羞的视频网站中文字幕 | 欧美久久电影 | 国产网红在线观看 | 五月婷婷电影网 | 亚洲日本欧美在线 | 国产一二区视频 | 91热这里只有精品 | 亚洲精品在线观看网站 | 久久avav| 中文字幕超清在线免费 | 在线亚洲观看 | 成人一区影院 | 亚洲 欧美 日韩 综合 | 日韩av一区二区在线播放 | 奇米影视8888在线观看大全免费 | 国产一区二区精品 | 欧美a性| 国产永久免费高清在线观看视频 | 美国三级黄色大片 | 久久歪歪 | 国产视频日韩视频欧美视频 | 人人讲 | 久久精品99 | 精品在线免费观看 | 亚洲最新精品 | 中文字幕在线观看完整 | 日本久久成人 | 久久影视网 | 国产裸体永久免费视频网站 | 成人黄色在线电影 | 国产不卡在线观看视频 | 在线观看一区 | 久久午夜视频 | 在线成人小视频 | 99在线免费观看 | 亚洲精品99久久久久中文字幕 | 99免费观看视频 | 一区二区成人国产精品 | 国产亚洲欧美在线视频 | 久久精品视频日本 | 综合色站导航 | 久久人人爽视频 | 97国产小视频 | 久久伊99综合婷婷久久伊 | 国产精品免费视频一区二区 | 97视频人人 | 91成人精品视频 | 97免费在线观看视频 | 欧美巨大荫蒂茸毛毛人妖 | 亚洲国产中文字幕在线 | 国产一级片免费观看 | av网址aaa | 久久免费电影 | 久久人人添人人爽添人人88v | 午夜精品久久久久久久99水蜜桃 | 国产高清网站 | 中文在线免费看视频 | 久久综合色影院 | 国产精品久久久久久高潮 | 亚洲第二色 | 中文字幕亚洲五码 | 天天综合网久久综合网 | 黄色小说免费在线观看 | 精品免费在线视频 | 久久久久久久久久久久国产精品 | 久久久亚洲麻豆日韩精品一区三区 | 国产中年夫妇高潮精品视频 | 国产免费高清 | 午夜国产一区 | 蜜桃视频精品 | 在线观看免费av网站 | 91最新地址永久入口 | 韩日成人av | 99视频在线精品 | 丁香五月缴情综合网 | 综合网成人| 久久久久久久99 | 国产精品第一页在线观看 | 久精品视频免费观看2 | 国产精品国产亚洲精品看不卡 | 黄色小视频在线观看免费 | 国产精品黄网站在线观看 | 九九久久精品视频 | 日韩成人黄色 | 亚洲精品高清一区二区三区四区 | a久久久久久 | 国产精品完整版 | 日日干夜夜操视频 | 色婷婷综合五月 | 91在线看视频免费 | 亚洲综合在线发布 | 一区二区成人国产精品 | 天天碰天天操 | 日日摸日日添夜夜爽97 | 午夜av电影院 | 国产人成在线观看 | 国产视频97 | 五月激情综合婷婷 | 伊人狠狠色 | 在线观看日韩精品视频 | 日韩精品久久久久 | 91成人网页版 | 亚洲精品国产高清 | 国产香蕉久久精品综合网 | 天天操天天爱天天干 | 亚洲精品动漫在线 | 久久久国产精品网站 | 夜夜骑首页 | 久久久蜜桃 | 人人干人人模 | 国产精品视频久久久 | 91探花视频 | 日日麻批40分钟视频免费观看 | 热精品| 99精品国产99久久久久久97 | 91亚洲精品乱码久久久久久蜜桃 | 国产午夜精品免费一区二区三区视频 | 亚a在线| 中文字幕久久网 | 国产在线小视频 | 色橹橹欧美在线观看视频高清 | 六月色丁 | 视频国产在线 | 亚洲va欧美va国产va黑人 | av免费片 | 日韩大片在线 | wwwav视频| 999色视频 | 亚洲一级黄色 | 不卡国产在线 | 国产精品视频永久免费播放 | 亚洲精品男女 | 久久综合色天天久久综合图片 | 亚洲日本va中文字幕 | 韩日精品中文字幕 | 久久精品在线 | av女优中文字幕在线观看 | 国产丝袜在线 | 视频在线观看入口黄最新永久免费国产 | 欧美电影在线观看 | 成年人视频免费在线 | 国产在线日韩 | 日本在线观看一区二区三区 | 国产精品激情偷乱一区二区∴ | 国产精品女教师 | 草久久久久 | 欧美中文字幕第一页 | 国产剧情在线一区 | 国产精品福利在线 | 亚洲精品乱码久久久久久蜜桃动漫 | 成人精品一区二区三区中文字幕 | 久草网视频 | 最近最新mv字幕免费观看 | 射射射综合网 | 欧美日韩高清一区二区三区 | 婷婷在线免费观看 | 欧美一级黄色网 | 欧美国产不卡 | 欧美aⅴ在线观看 | 二区三区在线视频 | 色婷五月天 | 久久手机看片 | 免费精品视频在线观看 | 超碰在线成人 | 久久精品一二三区 | 国产精品乱码久久 | 五月婷av | 欧美国产一区二区 | 国产精品一区二区视频 | 少妇精品久久久一区二区免费 | 日韩免费不卡av | 久久国产精品免费一区 | 欧美日韩二区三区 | 成人av资源 | 在线亚洲人成电影网站色www | 九九视频一区 | 免费日韩 精品中文字幕视频在线 | 国产精品一区二区三区久久久 | 贫乳av女优大全 | 婷婷网在线 | 国产艹b视频 | 人人干人人干人人干 | 最新日韩在线观看视频 | 免费开视频 | 国产一级做a爱片久久毛片a | 日韩在线大片 | 日韩黄在线观看 | 中文av在线天堂 | 久久精品一区二区三区视频 | 99精品欧美一区二区 | 四虎在线永久免费观看 | 在线观看国产永久免费视频 | 欧美在线视频不卡 | 免费久久久久久久 | 欧美日本高清视频 | 天天干天天摸 | 亚洲精选视频在线 | 中文字幕在线观看国产 | 91高清免费观看 | 亚洲国产精品一区二区久久,亚洲午夜 | 久久高清 | 亚洲六月丁香色婷婷综合久久 | 久久久这里有精品 | 丁香六月欧美 | 午夜性色 | 国产精品久久99综合免费观看尤物 | 在线影院av | 一 级 黄 色 片免费看的 | 久久在现 | 精品国产理论片 | 福利网址在线观看 | 免费av 在线| 免费三级黄色片 | 三上悠亚一区二区在线观看 | 青青河边草观看完整版高清 | 伊人婷婷久久 | 97在线看 | 亚洲 精品在线视频 | 国产又粗又猛又色又黄网站 | 亚洲综合在线发布 | 久久久久久久久黄色 | 一级久久久| 狠狠色狠狠色综合系列 | 人人超在线公开视频 | 激情综合网五月激情 | 在线看成人 | 日韩在线三级 | www.久久久| 一区二区三区免费在线观看视频 | 日韩一二三 | 伊人久久av | 免费看特级毛片 | 欧美激情h| 欧美日韩3p | 免费在线观看的av网站 | 成人性生交视频 | 综合色站导航 | 高清精品视频 | 日韩欧美网址 | 在线观av | 午夜视频在线观看欧美 | 在线黄网站| 国产精品久久久久久久久久久免费看 | 欧美激情综合色综合啪啪五月 | 亚洲日本一区二区在线 | 丁香国产视频 | 五月婷婷丁香六月 | 三级在线视频观看 | 国产免费三级在线观看 | a色视频| 五月亚洲 | 在线观看黄 | 在线观看国产成人av片 | 在线观看黄色大片 | 国产精品1区2区在线观看 | 超碰在线中文字幕 | 成年人视频免费在线 | 国产精品剧情在线亚洲 | 激情深爱五月 | 亚洲首页 | 天天综合区 | 中文字幕亚洲精品日韩 | 天天干天天干天天 | 久久av影视 | 亚洲一区二区精品视频 | 亚洲www天堂com | 爱爱av网| 国产亚洲精品久久久久久久久久 | 日韩激情中文字幕 | 成人国产精品久久久春色 | 亚洲精品国产第一综合99久久 | 久草视频在线免费播放 | 国产一级在线视频 | 国产成人a亚洲精品v | 欧美日韩视频 | 正在播放国产91 | 成人毛片在线视频 | 亚洲永久精品视频 | 久久经典视频 | 婷婷久月 | 亚洲精品在线观看的 | 久久久官网 | 亚洲人人射 | 久草在线观看资源 | 久久你懂的 | 国产精品免费视频网站 | 亚洲另类视频 | 天天干天天操天天操 | 欧美国产高清 | 91成人免费在线 | 国产青草视频在线观看 | 婷婷www| 国产精品成人久久久久久久 | 麻豆高清免费国产一区 | 一区二区三区日韩视频在线观看 | 亚洲九九爱 | 天天操夜夜操夜夜操 | 国产极品尤物在线 | 国产一级在线免费观看 | 综合激情网... | 国产美女在线精品免费观看 | 日本二区三区在线 | 免费日韩在线 | 99999精品视频 | 99爱这里只有精品 | 91亚洲精品国偷拍自产在线观看 | 麻花豆传媒mv在线观看网站 | 国产精品久久久久影视 | 国产精品黄色影片导航在线观看 | 99热在线国产精品 | 国产精品99久久久久 | 亚洲精品视频在线播放 | 中国一级片在线观看 | 欧美一区二区免费在线观看 | 国产专区视频在线观看 | 国产无遮挡又黄又爽在线观看 | 国产三级精品三级在线观看 | 天天草天天爽 | 在线黄色国产电影 | 欧美一区二区三区在线视频观看 | 国产二级视频 | 中文字幕在线观看一区 | 欧美日产一区 | 婷婷亚洲最大 | 日韩精品一区二区免费 | 国产精品久久久久久久久久久不卡 | 亚洲精品色视频 | 中文av不卡 | 天天操天天操天天操天天操 | 国产福利在线免费 | 日日干干夜夜 | 伊人久操| 97超碰人人澡人人爱学生 | 激情综合中文娱乐网 | 亚洲精品一区二区精华 | 麻豆影视在线播放 | 日本中文字幕高清 | www.黄色网.com | av日韩在线网站 | av成年人电影 | 蜜臀av性久久久久蜜臀aⅴ流畅 | 免费成人在线电影 | 日韩电影中文,亚洲精品乱码 | 国产成人av一区二区三区在线观看 | 日本中文一区二区 | 超碰激情在线 | 玖草在线观看 | 久久天天躁夜夜躁狠狠85麻豆 | 人人爽人人爽人人爽 | 激情文学丁香 | 国产精品麻豆果冻传媒在线播放 | 婷婷激情在线 | 国产精品成人一区二区 | 午夜av在线电影 | 久久久999免费视频 日韩网站在线 | 草久在线播放 | 奇米影音四色 | 亚洲成人精品久久久 | 欧美日韩一区二区三区在线免费观看 | 久久一区二区三区国产精品 | 麻豆精品视频在线观看免费 | av短片在线| 国产精品国产三级国产不产一地 | 少妇搡bbbb搡bbb搡69 | 国产精品久免费的黄网站 | 天天射天天舔天天干 | 久久精品欧美一 | 午夜在线看片 | 久久免费国产视频 | 91精品国产乱码在线观看 | 999久久国精品免费观看网站 | 在线中文字幕av观看 | 一区二区三区精品在线 | 美女亚洲精品 | 久久免费看毛片 | 国产成人三级一区二区在线观看一 | 国产精品99久久久久久有的能看 | 91久色蝌蚪 | 波多野结衣电影一区二区三区 | 中文字幕久久精品一区 | 91高清在线看 | 国产精品永久 | 色婷婷免费 | 黄色av播放| 黄色片网站免费 | 国产资源av | 午夜精品久久一牛影视 | 亚洲国产精品久久久久久 | 欧美日韩在线精品一区二区 | 欧洲精品视频一区二区 | 久久成人精品视频 | 一区二区三区在线免费观看 | 日韩精品亚洲专区在线观看 | 亚洲电影av在线 | 色资源网免费观看视频 | 麻豆你懂的 | 亚洲精品tv久久久久久久久久 | 国产福利91精品一区二区三区 | 久久你懂得| 亚洲天天摸日日摸天天欢 | 天天操天天色天天射 | 国产精品午夜在线 | av在线一级 | 国产经典三级 | 国产手机在线精品 | 奇米影视8888在线观看大全免费 | 在线91观看 | 国产欧美精品一区二区三区四区 | 丁香视频五月 | 69精品| 久久调教视频 | 久草网视频在线观看 | 日本公乱妇视频 | 91九色精品女同系列 | 国产网站av | 成年人在线观看网站 | 一本之道乱码区 | 五月视频 | 国产在线va | 免费看国产精品 | 国产成人中文字幕 | 精品国产成人在线 | 狠狠狠色丁香婷婷综合激情 | 欧美a级成人淫片免费看 | 国产精品观看视频 | 亚洲成av人片一区二区梦乃 | 亚洲精品在 | 久久乱码卡一卡2卡三卡四 五月婷婷久 | 综合激情| 99久久日韩精品免费热麻豆美女 | 亚洲精品午夜国产va久久成人 | 亚洲精品国产日韩 | 在线观看免费福利 | 九色91在线视频 | 91精品国产高清自在线观看 | 天天鲁天天干天天射 | 欧美日韩一二三四区 | av福利在线| 超碰人人在线观看 | 人人澡人人澡人人 | 午夜国产一区 | 九草在线视频 | 精品主播网红福利资源观看 | 亚洲春色成人 | 久久最新视频 | 美女在线观看网站 | 国产99久久九九精品 | 伊人久久国产精品 | 国产精品久久久久久久久久久久午夜片 | 亚洲精品在线观看中文字幕 | 99综合电影在线视频 | 四虎免费在线观看视频 | 久久久久免费 | 高清美女视频 | 伊人久久在线观看 | 一区二区三区高清 | 国产精品日韩欧美一区二区 | 日韩精品久久久久久久电影99爱 | 91在线免费播放视频 | 国产精品普通话 | 婷五月激情 | 欧美日韩国产免费视频 | 不卡的av在线 | 久久久国产日韩 | 精品国模一区二区 | 亚洲日本色 | 国产精品福利在线观看 | 欧美一区二区视频97 | 国产一区免费在线观看 | 午夜国产一区二区三区四区 | 精品国产乱码久久久久久久 | 夜夜爽88888免费视频4848 | 操操操操网 | 91丨九色丨国产在线观看 | 国产精品欧美久久久久三级 | ,午夜性刺激免费看视频 | 成人一区影院 | 日韩影视大全 | 久久久久久伊人 | 国产精品99久久久精品 | 天天色婷婷 | 成人毛片在线视频 | 五月天激情综合 | 99成人精品 | www.日本色 | 丁香婷婷在线观看 | 日韩啪啪小视频 | 九九免费在线看完整版 | 久草在线免费资源站 | aaa毛片视频 | 97超碰人人 | 91日韩在线专区 | 欧美日本日韩aⅴ在线视频 插插插色综合 | 99精品在线视频观看 | 久久婷婷网 | 日韩av网站在线播放 | av色影院 | 2020天天干夜夜爽 | 99久久久成人国产精品 | 色婷婷99| 成人午夜在线电影 | 日韩欧美xxx | av高清一区二区三区 | 毛片网站在线 | 亚洲一区二区视频在线 | 激情欧美xxxx | 99久久99久久精品国产片 | 亚洲男模gay裸体gay | 狠狠色丁婷婷日日 | 丝袜足交在线 | 国产视频精品网 | 日韩欧美一区二区三区在线观看 | 免费看日韩片 | 亚洲精品国产高清 | 欧美成人xxxxxxxx| 免费观看国产视频 | 婷婷干五月 | 亚洲美女免费精品视频在线观看 | 黄色免费观看 | 四虎影视成人永久免费观看亚洲欧美 | 亚洲精品中文字幕在线观看 | 国产精品国产毛片 | 少妇资源站 | 国产中年夫妇高潮精品视频 | 五月婷网站 | av免费网站在线观看 | 丝袜美腿亚洲 | 97国产在线| 国产成人99久久亚洲综合精品 | 又大又硬又黄又爽视频在线观看 | 久久久久伦理电影 | 夜色成人av | 成人va在线观看 | 国产精品久久久久久电影 | 不卡日韩av | 丁香花在线观看免费完整版视频 | 久久久久久国产精品美女 | 欧美日韩视频在线观看一区二区 | 国产综合小视频 | 五月婷婷激情六月 | 久久99久久99精品 | 国产日韩欧美视频在线观看 | 久久久片| 在线观看中文字幕2021 | 国产一区 在线播放 | 国产日产高清dvd碟片 | 一区二区不卡高清 | 色资源中文字幕 | 久久综合婷婷综合 | 精品久久一区二区三区 | 国产精品成人久久 | 久久久久久综合网天天 | 久久免费电影网 | 亚洲 精品在线视频 | 亚洲国产中文字幕在线观看 | 久草视频免费在线播放 | 久久久九色精品国产一区二区三区 | 韩国av电影在线观看 | 三级动图| 视频成人永久免费视频 | 日韩亚洲国产中文字幕 | 免费久久99精品国产婷婷六月 | 天天射天天做 | 日本女人在线观看 | 国产麻豆视频免费观看 | 91视频网址入口 | 毛片的网址 | 日韩在线视频免费看 | 亚洲精品免费播放 | 欧美日韩精品在线观看视频 | 夜夜夜夜猛噜噜噜噜噜初音未来 | 中文国产成人精品久久一 | 日韩av线观看 | 国产资源在线免费观看 | 欧洲一区二区三区精品 | 人人网av | 国产精品99久久久久人中文网介绍 | 久草国产视频 | 在线观看网站黄 | 91高清视频免费 | 成人免费观看视频网站 | 国产男男gay做爰 | 美女在线免费观看视频 | 欧美日韩中文国产一区发布 | 色婷婷免费视频 | 在线观看成年人 | 在线看v片 | 久久99精品国产99久久 | 日韩激情在线视频 | www.综合网.com | 天天操综合网站 | 国产国产人免费人成免费视频 | 久久久久久久久久久久影院 | 免费在线观看不卡av | 亚洲精品观看 | 91麻豆精品国产91久久久久 | 91色偷偷 | 黄色视屏免费在线观看 | 综合网久久 | 久久噜噜少妇网站 | 中文字幕一区二区三区在线视频 | 国产综合激情 | 日韩亚洲在线视频 | 中文字幕丝袜美腿 | 丁香花在线观看免费完整版视频 | 精品久久久久久久久中文字幕 | avav片| 欧美91精品国产自产 | 色av婷婷 | 欧美视频二区 | 五月综合色婷婷 | 国产精品影音先锋 | 美腿丝袜av | 国产一级一片免费播放放 | 激情久久一区二区三区 | 婷婷色网址 | 国产精品久久久久久久久毛片 | 91久久国产综合精品女同国语 | 天天干天天玩天天操 | 久久96国产精品久久99漫画 | 亚洲精品字幕在线观看 | 国产字幕在线看 | 免费大片黄在线 | 国产精品久久久久久五月尺 | 欧美日韩一区二区视频在线观看 | 亚洲精品视频一 | 色就色,综合激情 | 日韩mv欧美mv国产精品 | 手机看片国产 | 久久精品日产第一区二区三区乱码 | 国产精品二区三区 | 操综合| 色av婷婷| 久久综合中文字幕 | 最近久乱中文字幕 | 在线观看免费日韩 | 337p欧美 | 97超碰超碰 | 99久久99久久 | 国产精品精品国产色婷婷 | 在线观看午夜av | 国产成人精品久久二区二区 | 四虎影视成人精品国库在线观看 | 国产另类av |