本文主要講述使用 Prometheus監控ES，梳理核心監控指標并構建 Dashboard ，當集群有異常或者節點發生故障時，可以根據性能圖表以高效率的方式進行問題診斷，再對核心指標篩選添加告警

通常需要監控的幾個關鍵領域是：

1. 查詢和索引（indexing）性能
2. 內存分配和垃圾回收
3. 主機級別的系統和網絡指標
4. 集群健康狀態和節點可用性
5. 資源飽和度和相關錯誤

promethues官方的提供的ElasticSearch exporter提供的核心metrics監控以上關鍵領域

官網exporer地址：https://prometheus.io/docs/instrumenting/exporters/

elasticsearch_exporter　github地址:https://github.com/justwatchcom/elasticsearch_exporter

根據說明文檔安裝啟動即可：

[root@elk-log-srv01 elasticsearch_exporter-1.1.0rc1.linux-amd64]# ls
CHANGELOG.md  dashboard.json  deployment.yml  elasticsearch_exporter  elasticsearch.rules  LICENSE  nohup.out  README.md

　　支持kubernetes，可以看到提供的有相應的yml文件

直接啟動：

nohup ./elasticsearch_exporter --es.uri="http://172.16.0.xx:9200" 2>&1 &

　監聽在:9114端口

?

修改prometheus.yml

  - job_name: 'elasticsearch'static_configs:- targets:- 172.16.0.xx:9114#- 172.16.0.xx:9114#- 172.16.0.1xx:9114

　　

在grafana中導入elasticsearch模板文件：

　

{"__inputs": [{"name": "DS_PROMETHEUS","label": "Prometheus","description": "","type": "datasource","pluginId": "prometheus","pluginName": "Prometheus"}],"__requires": [{"type": "grafana","id": "grafana","name": "Grafana","version": "5.0.4"},{"type": "panel","id": "graph","name": "Graph","version": "5.0.0"},{"type": "datasource","id": "prometheus","name": "Prometheus","version": "5.0.0"},{"type": "panel","id": "singlestat","name": "Singlestat","version": "5.0.0"}],"annotations": {"list": [{"builtIn": 1,"datasource": "-- Grafana --","enable": true,"hide": true,"iconColor": "rgba(0, 211, 255, 1)","name": "Annotations & Alerts","type": "dashboard"}]},"description": "ElasticSearch cluster stats","editable": true,"gnetId": 2322,"graphTooltip": 1,"id": null,"iteration": 1544776125598,"links": [{"asDropdown": true,"icon": "external link","includeVars": false,"keepTime": true,"tags": ["OS"],"targetBlank": true,"title": "OS","type": "dashboards"},{"asDropdown": true,"icon": "external link","keepTime": true,"tags": ["MySQL"],"targetBlank": true,"title": "MySQL","type": "dashboards"},{"asDropdown": true,"icon": "external link","keepTime": true,"tags": ["MongoDB"],"targetBlank": true,"title": "MongoDB","type": "dashboards"},{"asDropdown": true,"icon": "external link","keepTime": true,"tags": ["App"],"targetBlank": true,"title": "App","type": "dashboards"}],"panels": [{"collapsed": false,"gridPos": {"h": 1,"w": 24,"x": 0,"y": 0},"id": 90,"panels": [],"repeat": null,"title": "KPI","type": "row"},{"cacheTimeout": null,"colorBackground": false,"colorValue": true,"colors": ["#508642","#bf1b00","#e5ac0e"],"datasource": "${DS_PROMETHEUS}","format": "none","gauge": {"maxValue": 100,"minValue": 0,"show": false,"thresholdLabels": false,"thresholdMarkers": true},"gridPos": {"h": 3,"w": 4,"x": 0,"y": 1},"height": "50","id": 53,"interval": null,"links": [],"mappingType": 1,"mappingTypes": [{"name": "value to text","value": 1},{"name": "range to text","value": 2}],"maxDataPoints": 100,"nullPointMode": "connected","nullText": null,"postfix": "","postfixFontSize": "50%","prefix": "","prefixFontSize": "50%","rangeMaps": [{"from": "null","text": "N/A","to": "null"}],"sparkline": {"fillColor": "rgba(31, 118, 189, 0.18)","full": true,"lineColor": "rgb(31, 120, 193)","show": false},"tableColumn": "__name__","targets": [{"expr": "(elasticsearch_cluster_health_status{cluster=\"$cluster\",color=\"red\",instance=\"$instance\"}==1)  or (elasticsearch_cluster_health_status{cluster=\"$cluster\",color=\"yellow\",instance=\"$instance\"}==1) +1 or (elasticsearch_cluster_health_status{cluster=\"$cluster\",color=\"green\",instance=\"$instance\"}==1) + 2","format": "time_series","instant": true,"intervalFactor": 2,"legendFormat": "","refId": "A"}],"thresholds": "1,2,3","title": "Cluster health","type": "singlestat","valueFontSize": "80%","valueMaps": [{"op": "=","text": "N/A","value": "null"},{"op": "=","text": "Green","value": "3"},{"op": "=","text": "Yellow","value": "2"},{"op": "=","text": "Red","value": "1"}],"valueName": "avg"},{"cacheTimeout": null,"colorBackground": false,"colorValue": true,"colors": ["#299c46","rgba(237, 129, 40, 0.89)","#d44a3a"],"datasource": "${DS_PROMETHEUS}","decimals": null,"format": "none","gauge": {"maxValue": 100,"minValue": 0,"show": false,"thresholdLabels": false,"thresholdMarkers": true},"gridPos": {"h": 3,"w": 2,"x": 4,"y": 1},"height": "50","id": 81,"interval": null,"links": [],"mappingType": 1,"mappingTypes": [{"name": "value to text","value": 1},{"name": "range to text","value": 2}],"maxDataPoints": 100,"nullPointMode": "connected","nullText": null,"postfix": "","postfixFontSize": "50%","prefix": "","prefixFontSize": "50%","rangeMaps": [{"from": "null","text": "N/A","to": "null"}],"sparkline": {"fillColor": "rgba(31, 118, 189, 0.18)","full": true,"lineColor": "rgb(31, 120, 193)","show": true},"tableColumn": "__name__","targets": [{"expr": "count(elasticsearch_breakers_tripped{cluster=\"$cluster\",name=~\"$name\"}>0)","format": "time_series","instant": true,"intervalFactor": 2,"legendFormat": "","refId": "A"}],"thresholds": "1,2","title": "Tripped for breakers","type": "singlestat","valueFontSize": "80%","valueMaps": [{"op": "=","text": "0","value": "N/A"},{"op": "=","text": "0","value": "no value"},{"op": "=","text": "0","value": "null"}],"valueName": "avg"},{"cacheTimeout": null,"colorBackground": false,"colorValue": true,"colors": ["rgba(50, 172, 45, 0.97)","rgba(237, 129, 40, 0.89)","rgba(245, 54, 54, 0.9)"],"datasource": "${DS_PROMETHEUS}","editable": true,"error": false,"format": "percent","gauge": {"maxValue": 100,"minValue": 0,"show": false,"thresholdLabels": false,"thresholdMarkers": true},"gridPos": {"h": 3,"w": 4,"x": 6,"y": 1},"height": "50","id": 51,"interval": null,"links": [],"mappingType": 1,"mappingTypes": [{"name": "value to text","value": 1},{"name": "range to text","value": 2}],"maxDataPoints": 100,"nullPointMode": "connected","nullText": null,"postfix": "","postfixFontSize": "50%","prefix": "","prefixFontSize": "50%","rangeMaps": [{"from": "null","text": "N/A","to": "null"}],"sparkline": {"fillColor": "rgba(31, 118, 189, 0.18)","full": false,"lineColor": "rgb(31, 120, 193)","show": false},"tableColumn": "","targets": [{"expr": "sum (elasticsearch_process_cpu_percent{cluster=\"$cluster\",name=~\"$name\"} ) / count (elasticsearch_process_cpu_percent{cluster=\"$cluster\",name=~\"$name\"} )","format": "time_series","instant": true,"interval": "","intervalFactor": 2,"legendFormat": "","metric": "","refId": "A","step": 60}],"thresholds": "70,80","title": "CPU usage Avg.","transparent": false,"type": "singlestat","valueFontSize": "80%","valueMaps": [{"op": "=","text": "N/A","value": "null"}],"valueName": "current"},{"cacheTimeout": null,"colorBackground": false,"colorValue": true,"colors": ["rgba(50, 172, 45, 0.97)","rgba(237, 129, 40, 0.89)","rgba(245, 54, 54, 0.9)"],"datasource": "${DS_PROMETHEUS}","editable": true,"error": false,"format": "percent","gauge": {"maxValue": 100,"minValue": 0,"show": false,"thresholdLabels": false,"thresholdMarkers": true},"gridPos": {"h": 3,"w": 4,"x": 10,"y": 1},"height": "50","id": 50,"interval": null,"links": [],"mappingType": 1,"mappingTypes": [{"name": "value to text","value": 1},{"name": "range to text","value": 2}],"maxDataPoints": 100,"nullPointMode": "connected","nullText": null,"postfix": "","postfixFontSize": "50%","prefix": "","prefixFontSize": "50%","rangeMaps": [{"from": "null","text": "N/A","to": "null"}],"sparkline": {"fillColor": "rgba(31, 118, 189, 0.18)","full": false,"lineColor": "rgb(31, 120, 193)","show": false},"tableColumn": "","targets": [{"expr": "sum (elasticsearch_jvm_memory_used_bytes{cluster=\"$cluster\",name=~\"$name\"}) / sum (elasticsearch_jvm_memory_max_bytes{cluster=\"$cluster\",name=~\"$name\"}) * 100","format": "time_series","instant": true,"interval": "","intervalFactor": 2,"legendFormat": "","metric": "","refId": "A","step": 60}],"thresholds": "70,80","title": "JVM memory used Avg.","transparent": false,"type": "singlestat","valueFontSize": "80%","valueMaps": [{"op": "=","text": "N/A","value": "null"}],"valueName": "current"},{"cacheTimeout": null,"colorBackground": false,"colorValue": true,"colors": ["#bf1b00","#508642","#508642"],"datasource": "${DS_PROMETHEUS}","decimals": null,"description": "Number of nodes in the cluster","editable": true,"error": false,"format": "none","gauge": {"maxValue": 100,"minValue": 0,"show": false,"thresholdLabels": false,"thresholdMarkers": true},"gridPos": {"h": 3,"w": 2,"x": 14,"y": 1},"height": "50","id": 10,"interval": null,"links": [],"mappingType": 1,"mappingTypes": [{"name": "value to text","value": 1},{"name": "range to text","value": 2}],"maxDataPoints": 100,"nullPointMode": "connected","nullText": null,"postfix": "","postfixFontSize": "50%","prefix": "","prefixFontSize": "50%","rangeMaps": [{"from": "null","text": "N/A","to": "null"}],"sparkline": {"fillColor": "rgba(31, 118, 189, 0.18)","full": false,"lineColor": "rgb(31, 120, 193)","show": false},"tableColumn": "","targets": [{"expr": "sum(elasticsearch_cluster_health_number_of_nodes{cluster=\"$cluster\",instance=\"$instance\"})","format": "time_series","instant": true,"interval": "","intervalFactor": 2,"legendFormat": "","metric": "","refId": "A","step": 60}],"thresholds": "13,14","title": "Nodes","transparent": false,"type": "singlestat","valueFontSize": "80%","valueMaps": [{"op": "=","text": "N/A","value": "null"}],"valueName": "total"},{"cacheTimeout": null,"colorBackground": false,"colorValue": true,"colors": ["#bf1b00","#508642","#3f6833"],"datasource": "${DS_PROMETHEUS}","description": "Number of data nodes in the cluster","editable": true,"error": false,"format": "none","gauge": {"maxValue": 100,"minValue": 0,"show": false,"thresholdLabels": false,"thresholdMarkers": true},"gridPos": {"h": 3,"w": 2,"x": 16,"y": 1},"height": "50","id": 9,"interval": null,"links": [],"mappingType": 1,"mappingTypes": [{"name": "value to text","value": 1},{"name": "range to text","value": 2}],"maxDataPoints": 100,"nullPointMode": "connected","nullText": null,"postfix": "","postfixFontSize": "50%","prefix": "","prefixFontSize": "50%","rangeMaps": [{"from": "null","text": "N/A","to": "null"}],"sparkline": {"fillColor": "rgba(31, 118, 189, 0.18)","full": false,"lineColor": "rgb(31, 120, 193)","show": false},"tableColumn": "","targets": [{"expr": "sum(elasticsearch_cluster_health_number_of_data_nodes{cluster=\"$cluster\",instance=\"$instance\"})","format": "time_series","instant": true,"interval": "","intervalFactor": 2,"legendFormat": "","metric": "","refId": "A","step": 60}],"thresholds": "9,>10","title": "Data nodes","transparent": false,"type": "singlestat","valueFontSize": "80%","valueMaps": [{"op": "=","text": "N/A","value": "null"}],"valueName": "total"},{"cacheTimeout": null,"colorBackground": false,"colorValue": true,"colors": ["rgba(50, 172, 45, 0.97)","rgba(237, 129, 40, 0.89)","rgba(245, 54, 54, 0.9)"],"datasource": "${DS_PROMETHEUS}","description": "Cluster level changes which have not yet been executed","editable": true,"error": false,"format": "none","gauge": {"maxValue": 100,"minValue": 0,"show": false,"thresholdLabels": false,"thresholdMarkers": true},"gridPos": {"h": 3,"w": 2,"x": 18,"y": 1},"height": "50","hideTimeOverride": true,"id": 16,"interval": null,"links": [],"mappingType": 1,"mappingTypes": [{"name": "value to text","value": 1},{"name": "range to text","value": 2}],"maxDataPoints": 100,"nullPointMode": "connected","nullText": null,"postfix": "","postfixFontSize": "50%","prefix": "","prefixFontSize": "50%","rangeMaps": [{"from": "null","text": "N/A","to": "null"}],"sparkline": {"fillColor": "rgba(31, 118, 189, 0.18)","full": false,"lineColor": "rgb(31, 120, 193)","show": true},"tableColumn": "","targets": [{"expr": "elasticsearch_cluster_health_number_of_pending_tasks{cluster=\"$cluster\",instance=\"$instance\"}","format": "time_series","instant": true,"interval": "","intervalFactor": 2,"legendFormat": "","metric": "","refId": "A","step": 60}],"thresholds": "1,5","title": "Pending tasks","transparent": false,"type": "singlestat","valueFontSize": "80%","valueMaps": [{"op": "=","text": "N/A","value": "null"}],"valueName": "current"},{"cacheTimeout": null,"colorBackground": false,"colorValue": false,"colors": ["rgba(50, 172, 45, 0.97)","rgba(237, 129, 40, 0.89)","rgba(245, 54, 54, 0.9)"],"datasource": "${DS_PROMETHEUS}","editable": true,"error": false,"format": "short","gauge": {"maxValue": 100,"minValue": 0,"show": false,"thresholdLabels": false,"thresholdMarkers": true},"gridPos": {"h": 3,"w": 4,"x": 20,"y": 1},"height": "50","id": 89,"interval": null,"links": [],"mappingType": 1,"mappingTypes": [{"name": "value to text","value": 1},{"name": "range to text","value": 2}],"maxDataPoints": 100,"nullPointMode": "connected","nullText": null,"postfix": "","postfixFontSize": "50%","prefix": "","prefixFontSize": "50%","rangeMaps": [{"from": "null","text": "N/A","to": "null"}],"sparkline": {"fillColor": "rgba(31, 118, 189, 0.18)","full": true,"lineColor": "rgb(31, 120, 193)","show": false},"tableColumn": "","targets": [{"expr": "sum (elasticsearch_process_open_files_count{cluster=\"$cluster\",name=~\"$name\"})","format": "time_series","instant": true,"interval": "","intervalFactor": 2,"legendFormat": "","metric": "","refId": "A","step": 60}],"thresholds": "","title": "Open file descriptors per cluster","transparent": false,"type": "singlestat","valueFontSize": "80%","valueMaps": [],"valueName": "current"},{"collapsed": false,"gridPos": {"h": 1,"w": 24,"x": 0,"y": 4},"id": 91,"panels": [],"repeat": null,"title": "Shards","type": "row"},{"cacheTimeout": null,"colorBackground": false,"colorValue": false,"colors": ["rgba(245, 54, 54, 0.9)","rgba(237, 129, 40, 0.89)","rgba(50, 172, 45, 0.97)"],"datasource": "${DS_PROMETHEUS}","description": "The number of primary shards in your cluster. This is an aggregate total across all indices.","editable": true,"error": false,"format": "none","gauge": {"maxValue": 100,"minValue": 0,"show": false,"thresholdLabels": false,"thresholdMarkers": true},"gridPos": {"h": 3,"w": 4,"x": 0,"y": 5},"height": "50","id": 11,"interval": null,"links": [],"mappingType": 1,"mappingTypes": [{"name": "value to text","value": 1},{"name": "range to text","value": 2}],"maxDataPoints": 100,"minSpan": 4,"nullPointMode": "connected","nullText": null,"postfix": "","postfixFontSize": "50%","prefix": "","prefixFontSize": "50%","rangeMaps": [{"from": "null","text": "N/A","to": "null"}],"repeat": "shard_type","sparkline": {"fillColor": "rgba(31, 118, 189, 0.18)","full": true,"lineColor": "rgb(31, 120, 193)","show": true},"tableColumn": "","targets": [{"expr": "sum(elasticsearch_cluster_health_active_primary_shards{cluster=\"$cluster\",instance=\"$instance\"})","format": "time_series","instant": true,"intervalFactor": 2,"legendFormat": "","refId": "A","step": 60}],"thresholds": "","title": "Active primary shards","type": "singlestat","valueFontSize": "80%","valueMaps": [{"op": "=","text": "N/A","value": "null"}],"valueName": "current"},{"cacheTimeout": null,"colorBackground": false,"colorValue": false,"colors": ["rgba(245, 54, 54, 0.9)","rgba(237, 129, 40, 0.89)","rgba(50, 172, 45, 0.97)"],"datasource": "${DS_PROMETHEUS}","description": "Aggregate total of all shards across all indices, which includes replica shards","editable": true,"error": false,"format": "none","gauge": {"maxValue": 100,"minValue": 0,"show": false,"thresholdLabels": false,"thresholdMarkers": true},"gridPos": {"h": 3,"w": 4,"x": 4,"y": 5},"height": "50","id": 39,"interval": null,"links": [],"mappingType": 1,"mappingTypes": [{"name": "value to text","value": 1},{"name": "range to text","value": 2}],"maxDataPoints": 100,"minSpan": 4,"nullPointMode": "connected","nullText": null,"postfix": "","postfixFontSize": "50%","prefix": "","prefixFontSize": "50%","rangeMaps": [{"from": "null","text": "N/A","to": "null"}],"sparkline": {"fillColor": "rgba(31, 118, 189, 0.18)","full": true,"lineColor": "rgb(31, 120, 193)","show": true},"tableColumn": "","targets": [{"expr": "sum(elasticsearch_cluster_health_active_shards{cluster=\"$cluster\",instance=\"$instance\"})","format": "time_series","instant": true,"intervalFactor": 2,"legendFormat": "","refId": "A","step": 60}],"thresholds": "","title": "Active shards","type": "singlestat","valueFontSize": "80%","valueMaps": [{"op": "=","text": "N/A","value": "null"}],"valueName": "current"},{"cacheTimeout": null,"colorBackground": false,"colorValue": false,"colors": ["rgba(245, 54, 54, 0.9)","rgba(237, 129, 40, 0.89)","rgba(50, 172, 45, 0.97)"],"datasource": "${DS_PROMETHEUS}","description": "Count of shards that are being freshly created","editable": true,"error": false,"format": "none","gauge": {"maxValue": 100,"minValue": 0,"show": false,"thresholdLabels": false,"thresholdMarkers": true},"gridPos": {"h": 3,"w": 4,"x": 8,"y": 5},"height": "50","id": 40,"interval": null,"links": [],"mappingType": 1,"mappingTypes": [{"name": "value to text","value": 1},{"name": "range to text","value": 2}],"maxDataPoints": 100,"minSpan": 4,"nullPointMode": "connected","nullText": null,"postfix": "","postfixFontSize": "50%","prefix": "","prefixFontSize": "50%","rangeMaps": [{"from": "null","text": "N/A","to": "null"}],"sparkline": {"fillColor": "rgba(31, 118, 189, 0.18)","full": true,"lineColor": "rgb(31, 120, 193)","show": true},"tableColumn": "","targets": [{"expr": "sum(elasticsearch_cluster_health_initializing_shards{cluster=\"$cluster\",instance=\"$instance\"})","format": "time_series","instant": true,"intervalFactor": 2,"legendFormat": "","refId": "A","step": 60}],"thresholds": "","title": "Initializing shards","type": "singlestat","valueFontSize": "80%","valueMaps": [{"op": "=","text": "N/A","value": "null"}],"valueName": "current"},{"cacheTimeout": null,"colorBackground": false,"colorValue": false,"colors": ["rgba(245, 54, 54, 0.9)","rgba(237, 129, 40, 0.89)","rgba(50, 172, 45, 0.97)"],"datasource": "${DS_PROMETHEUS}","description": "The number of shards that are currently moving from one node to another node.","editable": true,"error": false,"format": "none","gauge": {"maxValue": 100,"minValue": 0,"show": false,"thresholdLabels": false,"thresholdMarkers": true},"gridPos": {"h": 3,"w": 4,"x": 12,"y": 5},"height": "50","id": 41,"interval": null,"links": [],"mappingType": 1,"mappingTypes": [{"name": "value to text","value": 1},{"name": "range to text","value": 2}],"maxDataPoints": 100,"minSpan": 4,"nullPointMode": "connected","nullText": null,"postfix": "","postfixFontSize": "50%","prefix": "","prefixFontSize": "50%","rangeMaps": [{"from": "null","text": "N/A","to": "null"}],"sparkline": {"fillColor": "rgba(31, 118, 189, 0.18)","full": true,"lineColor": "rgb(31, 120, 193)","show": true},"tableColumn": "","targets": [{"expr": "sum(elasticsearch_cluster_health_relocating_shards{cluster=\"$cluster\",instance=\"$instance\"})","format": "time_series","instant": true,"intervalFactor": 2,"legendFormat": "","refId": "A","step": 60}],"thresholds": "","title": "Relocating shards","type": "singlestat","valueFontSize": "80%","valueMaps": [{"op": "=","text": "N/A","value": "null"}],"valueName": "current"},{"cacheTimeout": null,"colorBackground": false,"colorValue": false,"colors": ["rgba(245, 54, 54, 0.9)","rgba(237, 129, 40, 0.89)","rgba(50, 172, 45, 0.97)"],"datasource": "${DS_PROMETHEUS}","description": "Shards delayed to reduce reallocation overhead","editable": true,"error": false,"format": "none","gauge": {"maxValue": 100,"minValue": 0,"show": false,"thresholdLabels": false,"thresholdMarkers": true},"gridPos": {"h": 3,"w": 4,"x": 16,"y": 5},"height": "50","id": 42,"interval": null,"links": [],"mappingType": 1,"mappingTypes": [{"name": "value to text","value": 1},{"name": "range to text","value": 2}],"maxDataPoints": 100,"minSpan": 4,"nullPointMode": "connected","nullText": null,"postfix": "","postfixFontSize": "50%","prefix": "","prefixFontSize": "50%","rangeMaps": [{"from": "null","text": "N/A","to": "null"}],"sparkline": {"fillColor": "rgba(31, 118, 189, 0.18)","full": true,"lineColor": "rgb(31, 120, 193)","show": true},"tableColumn": "","targets": [{"expr": "sum(elasticsearch_cluster_health_delayed_unassigned_shards{cluster=\"$cluster\",instance=\"$instance\"} )","format": "time_series","instant": true,"intervalFactor": 2,"legendFormat": "","refId": "A","step": 60}],"thresholds": "","title": "Delayed shards","type": "singlestat","valueFontSize": "80%","valueMaps": [{"op": "=","text": "N/A","value": "null"}],"valueName": "current"},{"cacheTimeout": null,"colorBackground": false,"colorValue": false,"colors": ["rgba(245, 54, 54, 0.9)","rgba(237, 129, 40, 0.89)","rgba(50, 172, 45, 0.97)"],"datasource": "${DS_PROMETHEUS}","description": "The number of shards that exist in the cluster state, but cannot be found in the cluster itself","editable": true,"error": false,"format": "none","gauge": {"maxValue": 100,"minValue": 0,"show": false,"thresholdLabels": false,"thresholdMarkers": true},"gridPos": {"h": 3,"w": 4,"x": 20,"y": 5},"height": "50","id": 82,"interval": null,"links": [],"mappingType": 1,"mappingTypes": [{"name": "value to text","value": 1},{"name": "range to text","value": 2}],"maxDataPoints": 100,"minSpan": 4,"nullPointMode": "connected","nullText": null,"postfix": "","postfixFontSize": "50%","prefix": "","prefixFontSize": "50%","rangeMaps": [{"from": "null","text": "N/A","to": "null"}],"sparkline": {"fillColor": "rgba(31, 118, 189, 0.18)","full": true,"lineColor": "rgb(31, 120, 193)","show": true},"tableColumn": "","targets": [{"expr": "sum(elasticsearch_cluster_health_unassigned_shards{cluster=\"$cluster\",instance=\"$instance\"} )","format": "time_series","instant": true,"intervalFactor": 2,"legendFormat": "","refId": "A","step": 60}],"thresholds": "","title": "Unassigned shards","type": "singlestat","valueFontSize": "80%","valueMaps": [{"op": "=","text": "N/A","value": "null"}],"valueName": "current"},{"collapsed": false,"gridPos": {"h": 1,"w": 24,"x": 0,"y": 8},"id": 92,"panels": [],"repeat": null,"title": "JVM Garbage Collection","type": "row"},{"aliasColors": {},"bars": false,"dashLength": 10,"dashes": false,"datasource": "${DS_PROMETHEUS}","editable": true,"error": false,"fill": 1,"grid": {},"gridPos": {"h": 10,"w": 12,"x": 0,"y": 9},"height": "400","id": 7,"legend": {"alignAsTable": true,"avg": true,"current": true,"hideEmpty": false,"hideZero": false,"max": true,"min": true,"rightSide": false,"show": true,"sort": "avg","sortDesc": true,"total": false,"values": true},"lines": true,"linewidth": 1,"links": [],"nullPointMode": "connected","percentage": false,"pointradius": 5,"points": false,"renderer": "flot","seriesOverrides": [],"spaceLength": 10,"stack": false,"steppedLine": false,"targets": [{"expr": "irate(elasticsearch_jvm_gc_collection_seconds_count{cluster=\"$cluster\",name=~\"$name\"}[$interval])","format": "time_series","interval": "","intervalFactor": 2,"legendFormat": "{{name}} - {{gc}}","metric": "","refId": "A","step": 10}],"thresholds": [],"timeFrom": null,"timeShift": null,"title": "GC count","tooltip": {"msResolution": false,"shared": true,"sort": 2,"value_type": "cumulative"},"transparent": false,"type": "graph","xaxis": {"buckets": null,"mode": "time","name": null,"show": true,"values": []},"yaxes": [{"format": "short","label": "GCs","logBase": 1,"max": null,"min": null,"show": true},{"format": "short","label": null,"logBase": 1,"max": null,"min": null,"show": false}]},{"aliasColors": {},"bars": false,"dashLength": 10,"dashes": false,"datasource": "${DS_PROMETHEUS}","editable": true,"error": false,"fill": 1,"grid": {},"gridPos": {"h": 10,"w": 12,"x": 12,"y": 9},"height": "400","id": 27,"legend": {"alignAsTable": true,"avg": true,"current": true,"hideEmpty": false,"hideZero": false,"max": true,"min": true,"rightSide": false,"show": true,"sort": "avg","sortDesc": true,"total": false,"values": true},"lines": true,"linewidth": 1,"links": [],"nullPointMode": "connected","percentage": false,"pointradius": 5,"points": false,"renderer": "flot","seriesOverrides": [],"spaceLength": 10,"stack": false,"steppedLine": false,"targets": [{"expr": "irate(elasticsearch_jvm_gc_collection_seconds_sum{cluster=\"$cluster\",name=~\"$name\"}[$interval])","format": "time_series","interval": "","intervalFactor": 2,"legendFormat": "{{name}} - {{gc}}","metric": "","refId": "A","step": 10}],"thresholds": [],"timeFrom": null,"timeShift": null,"title": "GC time","tooltip": {"msResolution": false,"shared": true,"sort": 2,"value_type": "cumulative"},"transparent": false,"type": "graph","xaxis": {"buckets": null,"mode": "time","name": null,"show": true,"values": []},"yaxes": [{"format": "s","label": "Time","logBase": 1,"max": null,"min": null,"show": true},{"format": "short","label": null,"logBase": 1,"max": null,"min": null,"show": false}]},{"collapsed": true,"gridPos": {"h": 1,"w": 24,"x": 0,"y": 19},"id": 93,"panels": [{"aliasColors": {},"bars": false,"dashLength": 10,"dashes": false,"datasource": "Prometheus","fill": 1,"gridPos": {"h": 7,"w": 12,"x": 0,"y": 10},"id": 77,"legend": {"alignAsTable": true,"avg": true,"current": false,"max": true,"min": true,"rightSide": true,"show": true,"total": true,"values": true},"lines": true,"linewidth": 1,"links": [],"nullPointMode": "null","percentage": false,"pointradius": 5,"points": false,"renderer": "flot","seriesOverrides": [],"spaceLength": 10,"stack": false,"steppedLine": false,"targets": [{"expr": "irate(elasticsearch_indices_translog_operations{cluster=\"$cluster\",name=~\"$name\"}[$interval])","format": "time_series","intervalFactor": 2,"legendFormat": "{{name}}","refId": "A"}],"thresholds": [],"timeFrom": null,"timeShift": null,"title": "Total translog operations","tooltip": {"shared": true,"sort": 2,"value_type": "individual"},"type": "graph","xaxis": {"buckets": null,"mode": "time","name": null,"show": true,"values": []},"yaxes": [{"format": "short","label": null,"logBase": 1,"max": null,"min": null,"show": true},{"format": "short","label": null,"logBase": 1,"max": null,"min": null,"show": false}]},{"aliasColors": {},"bars": false,"dashLength": 10,"dashes": false,"datasource": "Prometheus","fill": 1,"gridPos": {"h": 7,"w": 12,"x": 12,"y": 10},"id": 78,"legend": {"alignAsTable": true,"avg": true,"current": true,"max": true,"min": true,"rightSide": true,"show": true,"total": false,"values": true},"lines": true,"linewidth": 1,"links": [],"nullPointMode": "null","percentage": false,"pointradius": 5,"points": false,"renderer": "flot","seriesOverrides": [],"spaceLength": 10,"stack": false,"steppedLine": false,"targets": [{"expr": "irate(elasticsearch_indices_translog_size_in_bytes{cluster=\"$cluster\",name=~\"$name\"}[$interval])","format": "time_series","intervalFactor": 2,"legendFormat": "{{name}}","refId": "A"}],"thresholds": [],"timeFrom": null,"timeShift": null,"title": "Total translog size in bytes","tooltip": {"shared": true,"sort": 2,"value_type": "individual"},"type": "graph","xaxis": {"buckets": null,"mode": "time","name": null,"show": true,"values": []},"yaxes": [{"format": "bytes","label": null,"logBase": 1,"max": null,"min": null,"show": true},{"format": "short","label": null,"logBase": 1,"max": null,"min": null,"show": false}]}],"repeat": null,"title": "Translog","type": "row"},{"collapsed": true,"gridPos": {"h": 1,"w": 24,"x": 0,"y": 20},"id": 94,"panels": [{"aliasColors": {},"bars": false,"dashLength": 10,"dashes": false,"datasource": "Prometheus","fill": 1,"gridPos": {"h": 7,"w": 24,"x": 0,"y": 11},"id": 79,"legend": {"alignAsTable": true,"avg": true,"current": false,"max": true,"min": true,"rightSide": true,"show": true,"sort": null,"sortDesc": null,"total": false,"values": true},"lines": true,"linewidth": 1,"links": [],"nullPointMode": "null","percentage": false,"pointradius": 5,"points": false,"renderer": "flot","seriesOverrides": [],"spaceLength": 10,"stack": false,"steppedLine": false,"targets": [{"expr": "elasticsearch_breakers_tripped{cluster=\"$cluster\",name=~\"$name\"}","format": "time_series","intervalFactor": 2,"legendFormat": "{{name}}: {{breaker}}","refId": "A"}],"thresholds": [],"timeFrom": null,"timeShift": null,"title": "Tripped for breakers","tooltip": {"shared": true,"sort": 2,"value_type": "individual"},"type": "graph","xaxis": {"buckets": null,"mode": "time","name": null,"show": true,"values": []},"yaxes": [{"format": "short","label": null,"logBase": 1,"max": null,"min": null,"show": true},{"format": "short","label": null,"logBase": 1,"max": null,"min": null,"show": false}]},{"aliasColors": {},"bars": false,"dashLength": 10,"dashes": false,"datasource": "Prometheus","fill": 1,"gridPos": {"h": 7,"w": 24,"x": 0,"y": 18},"id": 80,"legend": {"alignAsTable": true,"avg": true,"current": false,"max": true,"min": true,"rightSide": true,"show": true,"sort": null,"sortDesc": null,"total": false,"values": true},"lines": true,"linewidth": 1,"links": [],"nullPointMode": "null","percentage": false,"pointradius": 5,"points": false,"renderer": "flot","seriesOverrides": [],"spaceLength": 10,"stack": false,"steppedLine": false,"targets": [{"expr": "elasticsearch_breakers_estimated_size_bytes{cluster=\"$cluster\",name=~\"$name\"}","format": "time_series","intervalFactor": 2,"legendFormat": "{{name}}: {{breaker}}","refId": "A"},{"expr": "elasticsearch_breakers_limit_size_bytes{cluster=\"$cluster\",name=~\"$name\"}","format": "time_series","intervalFactor": 2,"legendFormat": "{{name}}: limit for {{breaker}}","refId": "B"}],"thresholds": [],"timeFrom": null,"timeShift": null,"title": "Estimated size in bytes of breaker","tooltip": {"shared": true,"sort": 2,"value_type": "individual"},"type": "graph","xaxis": {"buckets": null,"mode": "time","name": null,"show": true,"values": []},"yaxes": [{"format": "bytes","label": null,"logBase": 1,"max": null,"min": null,"show": true},{"format": "short","label": null,"logBase": 1,"max": null,"min": null,"show": false}]}],"repeat": null,"title": "Breakers","type": "row"},{"collapsed": true,"gridPos": {"h": 1,"w": 24,"x": 0,"y": 21},"id": 95,"panels": [{"aliasColors": {},"bars": false,"dashLength": 10,"dashes": false,"datasource": "Prometheus","editable": true,"error": false,"fill": 1,"grid": {},"gridPos": {"h": 10,"w": 12,"x": 0,"y": 12},"height": "400","id": 30,"legend": {"alignAsTable": true,"avg": true,"current": true,"hideEmpty": false,"hideZero": false,"max": true,"min": true,"rightSide": false,"show": true,"sort": "avg","sortDesc": true,"total": false,"values": true},"lines": true,"linewidth": 1,"links": [],"nullPointMode": "connected","percentage": false,"pointradius": 5,"points": false,"renderer": "flot","seriesOverrides": [],"spaceLength": 10,"stack": false,"steppedLine": false,"targets": [{"expr": "elasticsearch_os_load1{cluster=\"$cluster\",name=~\"$name\"}","format": "time_series","instant": false,"interval": "","intervalFactor": 2,"legendFormat": "load1: {{name}}","metric": "","refId": "A","step": 20},{"expr": "elasticsearch_os_load5{cluster=\"$cluster\",name=~\"$name\"}","format": "time_series","instant": false,"interval": "","intervalFactor": 2,"legendFormat": "load5: {{name}}","metric": "","refId": "B","step": 20},{"expr": "elasticsearch_os_load15{cluster=\"$cluster\",name=~\"$name\"}","format": "time_series","instant": false,"interval": "","intervalFactor": 2,"legendFormat": "load15: {{name}}","metric": "","refId": "C","step": 20}],"thresholds": [],"timeFrom": null,"timeShift": null,"title": "Load average","tooltip": {"msResolution": false,"shared": true,"sort": 2,"value_type": "cumulative"},"transparent": false,"type": "graph","xaxis": {"buckets": null,"mode": "time","name": null,"show": true,"values": []},"yaxes": [{"format": "short","label": "CPU usage","logBase": 1,"max": 100,"min": 0,"show": true},{"format": "short","label": null,"logBase": 1,"max": null,"min": null,"show": false}]},{"aliasColors": {},"bars": false,"dashLength": 10,"dashes": false,"datasource": "Prometheus","editable": true,"error": false,"fill": 1,"grid": {},"gridPos": {"h": 10,"w": 12,"x": 12,"y": 12},"height": "400","id": 88,"legend": {"alignAsTable": true,"avg": true,"current": true,"hideEmpty": false,"hideZero": false,"max": true,"min": true,"rightSide": false,"show": true,"sort": "avg","sortDesc": true,"total": false,"values": true},"lines": true,"linewidth": 1,"links": [],"nullPointMode": "connected","percentage": false,"pointradius": 5,"points": false,"renderer": "flot","seriesOverrides": [],"spaceLength": 10,"stack": false,"steppedLine": false,"targets": [{"expr": "elasticsearch_process_cpu_percent{cluster=\"$cluster\",name=~\"$name\"}","format": "time_series","instant": false,"interval": "","intervalFactor": 2,"legendFormat": "{{name}}","metric": "","refId": "A","step": 20}],"thresholds": [],"timeFrom": null,"timeShift": null,"title": "CPU usage","tooltip": {"msResolution": false,"shared": true,"sort": 2,"value_type": "cumulative"},"transparent": false,"type": "graph","xaxis": {"buckets": null,"mode": "time","name": null,"show": true,"values": []},"yaxes": [{"format": "percent","label": "CPU usage","logBase": 1,"max": 100,"min": 0,"show": true},{"format": "short","label": null,"logBase": 1,"max": null,"min": null,"show": false}]},{"aliasColors": {},"bars": false,"dashLength": 10,"dashes": false,"datasource": "Prometheus","editable": true,"error": false,"fill": 0,"grid": {},"gridPos": {"h": 10,"w": 12,"x": 0,"y": 22},"height": "400","id": 31,"legend": {"alignAsTable": true,"avg": true,"current": true,"hideEmpty": false,"hideZero": false,"max": true,"min": true,"rightSide": false,"show": true,"sort": "avg","sortDesc": true,"total": false,"values": true},"lines": true,"linewidth": 1,"links": [],"nullPointMode": "connected","percentage": false,"pointradius": 5,"points": false,"renderer": "flot","seriesOverrides": [],"spaceLength": 10,"stack": false,"steppedLine": false,"targets": [{"expr": "elasticsearch_jvm_memory_used_bytes{cluster=\"$cluster\",name=~\"$name\"}","format": "time_series","interval": "","intervalFactor": 2,"legendFormat": "{{name}} used: {{area}}","metric": "","refId": "A","step": 20},{"expr": "elasticsearch_jvm_memory_max_bytes{cluster=\"$cluster\",name=~\"$name\"}","format": "time_series","intervalFactor": 2,"legendFormat": "{{name}} max: {{area}}","refId": "C","step": 20},{"expr": "elasticsearch_jvm_memory_pool_peak_used_bytes{cluster=\"$cluster\",name=~\"$name\"}","format": "time_series","intervalFactor": 2,"legendFormat": "{{name}} peak used pool: {{pool}}","refId": "D","step": 20}],"thresholds": [],"timeFrom": null,"timeShift": null,"title": "JVM memory usage","tooltip": {"msResolution": false,"shared": true,"sort": 2,"value_type": "cumulative"},"transparent": false,"type": "graph","xaxis": {"buckets": null,"mode": "time","name": null,"show": true,"values": []},"yaxes": [{"format": "bytes","label": "Memory","logBase": 1,"max": null,"min": 0,"show": true},{"format": "short","label": null,"logBase": 1,"max": null,"min": null,"show": false}]},{"aliasColors": {},"bars": false,"dashLength": 10,"dashes": false,"datasource": "Prometheus","editable": true,"error": false,"fill": 0,"grid": {},"gridPos": {"h": 10,"w": 12,"x": 12,"y": 22},"height": "400","id": 54,"legend": {"alignAsTable": true,"avg": true,"current": true,"hideEmpty": false,"hideZero": false,"max": true,"min": true,"rightSide": false,"show": true,"sort": "avg","sortDesc": true,"total": false,"values": true},"lines": true,"linewidth": 1,"links": [],"nullPointMode": "connected","percentage": false,"pointradius": 5,"points": false,"renderer": "flot","seriesOverrides": [],"spaceLength": 10,"stack": false,"steppedLine": false,"targets": [{"expr": "elasticsearch_jvm_memory_committed_bytes{cluster=\"$cluster\",name=~\"$name\"}","format": "time_series","intervalFactor": 2,"legendFormat": "{{name}} committed: {{area}}","refId": "B","step": 20},{"expr": "elasticsearch_jvm_memory_max_bytes{cluster=\"$cluster\",name=~\"$name\"}","format": "time_series","intervalFactor": 2,"legendFormat": "{{name}} max: {{area}}","refId": "C","step": 20}],"thresholds": [],"timeFrom": null,"timeShift": null,"title": "JVM memory committed","tooltip": {"msResolution": false,"shared": true,"sort": 2,"value_type": "cumulative"},"transparent": false,"type": "graph","xaxis": {"buckets": null,"mode": "time","name": null,"show": true,"values": []},"yaxes": [{"format": "bytes","label": "Memory","logBase": 1,"max": null,"min": 0,"show": true},{"format": "short","label": null,"logBase": 1,"max": null,"min": null,"show": false}]}],"repeat": null,"title": "CPU and Memory","type": "row"},{"collapsed": true,"gridPos": {"h": 1,"w": 24,"x": 0,"y": 22},"id": 96,"panels": [{"aliasColors": {},"bars": false,"dashLength": 10,"dashes": false,"datasource": "Prometheus","editable": true,"error": false,"fill": 1,"grid": {},"gridPos": {"h": 10,"w": 12,"x": 0,"y": 13},"height": "400","id": 32,"legend": {"alignAsTable": true,"avg": true,"current": true,"hideEmpty": false,"hideZero": false,"max": true,"min": true,"rightSide": false,"show": true,"sort": "current","sortDesc": true,"total": false,"values": true},"lines": true,"linewidth": 1,"links": [],"nullPointMode": "connected","percentage": false,"pointradius": 5,"points": false,"renderer": "flot","seriesOverrides": [],"spaceLength": 10,"stack": false,"steppedLine": false,"targets": [{"expr": "1-(elasticsearch_filesystem_data_available_bytes{cluster=\"$cluster\",name=~\"$name\"}/elasticsearch_filesystem_data_size_bytes{cluster=\"$cluster\",name=~\"$name\"})","format": "time_series","interval": "","intervalFactor": 2,"legendFormat": "{{name}}: {{path}}","metric": "","refId": "A","step": 20}],"thresholds": [{"colorMode": "custom","fill": true,"fillColor": "rgba(216, 200, 27, 0.27)","op": "gt","value": 0.8},{"colorMode": "custom","fill": true,"fillColor": "rgba(234, 112, 112, 0.22)","op": "gt","value": 0.9}],"timeFrom": null,"timeShift": null,"title": "Disk usage","tooltip": {"msResolution": false,"shared": true,"sort": 2,"value_type": "cumulative"},"transparent": false,"type": "graph","xaxis": {"buckets": null,"mode": "time","name": null,"show": true,"values": []},"yaxes": [{"format": "percentunit","label": "Disk Usage %","logBase": 1,"max": 1,"min": 0,"show": true},{"format": "short","label": null,"logBase": 1,"max": null,"min": null,"show": false}]},{"aliasColors": {},"bars": false,"dashLength": 10,"dashes": false,"datasource": "Prometheus","editable": true,"error": false,"fill": 1,"grid": {},"gridPos": {"h": 10,"w": 12,"x": 12,"y": 13},"height": "400","id": 47,"legend": {"alignAsTable": true,"avg": true,"current": true,"hideEmpty": false,"hideZero": false,"max": true,"min": true,"rightSide": false,"show": true,"sort": "avg","sortDesc": true,"total": false,"values": true},"lines": true,"linewidth": 1,"links": [],"nullPointMode": "connected","percentage": false,"pointradius": 5,"points": false,"renderer": "flot","seriesOverrides": [{"alias": "sent","transform": "negative-Y"}],"spaceLength": 10,"stack": false,"steppedLine": false,"targets": [{"expr": "irate(elasticsearch_transport_tx_size_bytes_total{cluster=\"$cluster\",name=~\"$name\"}[$interval])","format": "time_series","intervalFactor": 2,"legendFormat": "{{name}}: sent ","refId": "D","step": 20},{"expr": "-irate(elasticsearch_transport_rx_size_bytes_total{cluster=\"$cluster\",name=~\"$name\"}[$interval])","format": "time_series","intervalFactor": 2,"legendFormat": "{{name}}: received","refId": "C","step": 20}],"thresholds": [],"timeFrom": null,"timeShift": null,"title": "Network usage","tooltip": {"msResolution": false,"shared": true,"sort": 2,"value_type": "cumulative"},"transparent": false,"type": "graph","xaxis": {"buckets": null,"mode": "time","name": null,"show": true,"values": []},"yaxes": [{"format": "Bps","label": "Bytes/sec","logBase": 1,"max": null,"min": null,"show": true},{"format": "pps","label": "","logBase": 1,"max": null,"min": null,"show": false}]}],"repeat": null,"title": "Disk and Network","type": "row"},{"collapsed": true,"gridPos": {"h": 1,"w": 24,"x": 0,"y": 23},"id": 97,"panels": [{"aliasColors": {},"bars": false,"dashLength": 10,"dashes": false,"datasource": "Prometheus","decimals": 2,"editable": true,"error": false,"fill": 1,"grid": {},"gridPos": {"h": 10,"w": 12,"x": 0,"y": 14},"height": "400","id": 1,"legend": {"alignAsTable": true,"avg": true,"current": true,"hideEmpty": false,"hideZero": false,"max": true,"min": true,"rightSide": false,"show": true,"sort": "avg","sortDesc": true,"total": false,"values": true},"lines": true,"linewidth": 1,"links": [],"nullPointMode": "connected","percentage": false,"pointradius": 5,"points": false,"renderer": "flot","seriesOverrides": [],"spaceLength": 10,"stack": true,"steppedLine": false,"targets": [{"expr": "elasticsearch_indices_docs{cluster=\"$cluster\",name=~\"$name\"}","format": "time_series","interval": "","intervalFactor": 2,"legendFormat": "{{name}}","metric": "","refId": "A","step": 20}],"thresholds": [],"timeFrom": null,"timeShift": null,"title": "Documents count on node","tooltip": {"msResolution": false,"shared": true,"sort": 0,"value_type": "cumulative"},"transparent": false,"type": "graph","xaxis": {"buckets": null,"mode": "time","name": null,"show": true,"values": []},"yaxes": [{"decimals": 2,"format": "short","label": "","logBase": 1,"max": null,"min": null,"show": true},{"format": "short","label": null,"logBase": 1,"max": null,"min": null,"show": false}]},{"aliasColors": {},"bars": false,"dashLength": 10,"dashes": false,"datasource": "Prometheus","editable": true,"error": false,"fill": 1,"grid": {},"gridPos": {"h": 10,"w": 12,"x": 12,"y": 14},"height": "400","id": 24,"legend": {"alignAsTable": true,"avg": true,"current": true,"hideEmpty": false,"hideZero": false,"max": true,"min": true,"rightSide": false,"show": true,"total": false,"values": true},"lines": true,"linewidth": 1,"links": [],"nullPointMode": "connected","percentage": false,"pointradius": 5,"points": false,"renderer": "flot","seriesOverrides": [],"spaceLength": 10,"stack": true,"steppedLine": false,"targets": [{"expr": "irate(elasticsearch_indices_indexing_index_total{cluster=\"$cluster\",name=~\"$name\"}[$interval])","format": "time_series","interval": "","intervalFactor": 2,"legendFormat": "{{name}}","metric": "","refId": "A","step": 20}],"thresholds": [],"timeFrom": null,"timeShift": null,"title": "Documents indexed rate","tooltip": {"msResolution": false,"shared": true,"sort": 0,"value_type": "cumulative"},"transparent": false,"type": "graph","xaxis": {"buckets": null,"mode": "time","name": null,"show": true,"values": []},"yaxes": [{"format": "short","label": "index calls/s","logBase": 1,"max": null,"min": null,"show": true},{"format": "short","label": null,"logBase": 1,"max": null,"min": null,"show": false}]},{"aliasColors": {},"bars": false,"dashLength": 10,"dashes": false,"datasource": "Prometheus","description": "Count of deleted documents on this node","editable": true,"error": false,"fill": 1,"grid": {},"gridPos": {"h": 10,"w": 8,"x": 0,"y": 24},"height": "400","id": 25,"legend": {"alignAsTable": true,"avg": true,"current": true,"hideEmpty": false,"hideZero": false,"max": true,"min": true,"rightSide": false,"show": true,"sort": "avg","sortDesc": true,"total": false,"values": true},"lines": true,"linewidth": 1,"links": [],"nullPointMode": "connected","percentage": false,"pointradius": 5,"points": false,"renderer": "flot","seriesOverrides": [],"spaceLength": 10,"stack": true,"steppedLine": false,"targets": [{"expr": "irate(elasticsearch_indices_docs_deleted{cluster=\"$cluster\",name=~\"$name\"}[$interval])","format": "time_series","interval": "","intervalFactor": 2,"legendFormat": "{{name}}","metric": "","refId": "A","step": 20}],"thresholds": [],"timeFrom": null,"timeShift": null,"title": "Documents deleted rate","tooltip": {"msResolution": false,"shared": true,"sort": 0,"value_type": "cumulative"},"transparent": false,"type": "graph","xaxis": {"buckets": null,"mode": "time","name": null,"show": true,"values": []},"yaxes": [{"format": "short","label": "Documents/s","logBase": 1,"max": null,"min": null,"show": true},{"format": "short","label": null,"logBase": 1,"max": null,"min": null,"show": false}]},{"aliasColors": {},"bars": false,"dashLength": 10,"dashes": false,"datasource": "Prometheus","decimals": 2,"editable": true,"error": false,"fill": 1,"grid": {},"gridPos": {"h": 10,"w": 8,"x": 8,"y": 24},"height": "400","id": 26,"legend": {"alignAsTable": true,"avg": true,"current": true,"hideEmpty": false,"hideZero": false,"max": true,"min": true,"rightSide": false,"show": true,"total": false,"values": true},"lines": true,"linewidth": 1,"links": [],"nullPointMode": "connected","percentage": false,"pointradius": 5,"points": false,"renderer": "flot","seriesOverrides": [],"spaceLength": 10,"stack": true,"steppedLine": false,"targets": [{"expr": "rate(elasticsearch_indices_merges_docs_total{cluster=\"$cluster\",name=~\"$name\"}[$interval])","format": "time_series","interval": "","intervalFactor": 2,"legendFormat": "{{name}}","metric": "","refId": "A","step": 20}],"thresholds": [],"timeFrom": null,"timeShift": null,"title": "Documents merged rate","tooltip": {"msResolution": false,"shared": true,"sort": 0,"value_type": "cumulative"},"transparent": false,"type": "graph","xaxis": {"buckets": null,"mode": "time","name": null,"show": true,"values": []},"yaxes": [{"decimals": 2,"format": "short","label": "Documents/s","logBase": 1,"max": null,"min": null,"show": true},{"format": "short","label": null,"logBase": 1,"max": null,"min": null,"show": false}]},{"aliasColors": {},"bars": false,"dashLength": 10,"dashes": false,"datasource": "Prometheus","editable": true,"error": false,"fill": 1,"grid": {},"gridPos": {"h": 10,"w": 8,"x": 16,"y": 24},"height": "400","id": 52,"legend": {"alignAsTable": true,"avg": true,"current": true,"hideEmpty": false,"hideZero": false,"max": true,"min": true,"rightSide": false,"show": true,"total": false,"values": true},"lines": true,"linewidth": 1,"links": [],"nullPointMode": "connected","percentage": false,"pointradius": 5,"points": false,"renderer": "flot","seriesOverrides": [],"spaceLength": 10,"stack": true,"steppedLine": false,"targets": [{"expr": "irate(elasticsearch_indices_merges_total_size_bytes_total{cluster=\"$cluster\",name=~\"$name\"}[$interval])","format": "time_series","interval": "","intervalFactor": 2,"legendFormat": "{{name}}","metric": "","refId": "A","step": 20}],"thresholds": [],"timeFrom": null,"timeShift": null,"title": "Documents merged bytes","tooltip": {"msResolution": false,"shared": true,"sort": 0,"value_type": "cumulative"},"transparent": false,"type": "graph","xaxis": {"buckets": null,"mode": "time","name": null,"show": true,"values": []},"yaxes": [{"decimals": null,"format": "decbytes","label": "Bytes/s","logBase": 1,"max": null,"min": null,"show": true},{"format": "short","label": null,"logBase": 1,"max": null,"min": null,"show": false}]}],"repeat": null,"title": "Documents","type": "row"},{"collapsed": true,"gridPos": {"h": 1,"w": 24,"x": 0,"y": 24},"id": 98,"panels": [{"aliasColors": {},"bars": false,"dashLength": 10,"dashes": false,"datasource": "Prometheus","editable": true,"error": false,"fill": 1,"grid": {},"gridPos": {"h": 10,"w": 12,"x": 0,"y": 15},"height": "400","id": 33,"legend": {"alignAsTable": true,"avg": true,"current": true,"hideEmpty": false,"hideZero": false,"max": true,"min": true,"rightSide": true,"show": true,"total": false,"values": true},"lines": true,"linewidth": 1,"links": [],"nullPointMode": "connected","percentage": false,"pointradius": 5,"points": false,"renderer": "flot","seriesOverrides": [],"spaceLength": 10,"stack": false,"steppedLine": false,"targets": [{"expr": "irate(elasticsearch_indices_search_query_time_seconds{cluster=\"$cluster\",name=~\"$name\"}[$interval]) ","format": "time_series","interval": "","intervalFactor": 2,"legendFormat": "{{name}}","metric": "","refId": "A","step": 10}],"thresholds": [],"timeFrom": null,"timeShift": null,"title": "Query time","tooltip": {"msResolution": false,"shared": true,"sort": 0,"value_type": "cumulative"},"transparent": false,"type": "graph","xaxis": {"buckets": null,"mode": "time","name": null,"show": true,"values": []},"yaxes": [{"format": "s","label": "Time","logBase": 1,"max": null,"min": null,"show": true},{"format": "short","label": null,"logBase": 1,"max": null,"min": null,"show": false}]},{"aliasColors": {},"bars": false,"dashLength": 10,"dashes": false,"datasource": "Prometheus","editable": true,"error": false,"fill": 1,"grid": {},"gridPos": {"h": 10,"w": 12,"x": 12,"y": 15},"height": "400","id": 5,"legend": {"alignAsTable": true,"avg": true,"current": true,"hideEmpty": false,"hideZero": false,"max": true,"min": true,"rightSide": true,"show": true,"total": false,"values": true},"lines": true,"linewidth": 1,"links": [],"nullPointMode": "connected","percentage": false,"pointradius": 5,"points": false,"renderer": "flot","seriesOverrides": [],"spaceLength": 10,"stack": false,"steppedLine": false,"targets": [{"expr": "irate(elasticsearch_indices_indexing_index_time_seconds_total{cluster=\"$cluster\",name=~\"$name\"}[$interval])","format": "time_series","interval": "","intervalFactor": 2,"legendFormat": "{{name}}","metric": "","refId": "A","step": 10}],"thresholds": [],"timeFrom": null,"timeShift": null,"title": "Indexing time","tooltip": {"msResolution": false,"shared": true,"sort": 0,"value_type": "cumulative"},"transparent": false,"type": "graph","xaxis": {"buckets": null,"mode": "time","name": null,"show": true,"values": []},"yaxes": [{"format": "s","label": "Time","logBase": 1,"max": null,"min": null,"show": true},{"format": "short","label": null,"logBase": 1,"max": null,"min": null,"show": false}]},{"aliasColors": {},"bars": false,"dashLength": 10,"dashes": false,"datasource": "Prometheus","editable": true,"error": false,"fill": 1,"grid": {},"gridPos": {"h": 10,"w": 12,"x": 0,"y": 25},"height": "400","id": 3,"legend": {"alignAsTable": true,"avg": true,"current": true,"hideEmpty": false,"hideZero": false,"max": true,"min": true,"rightSide": true,"show": true,"sort": "avg","sortDesc": true,"total": false,"values": true},"lines": true,"linewidth": 1,"links": [],"nullPointMode": "connected","percentage": false,"pointradius": 5,"points": false,"renderer": "flot","seriesOverrides": [],"spaceLength": 10,"stack": false,"steppedLine": false,"targets": [{"expr": "irate(elasticsearch_indices_merges_total_time_seconds_total{cluster=\"$cluster\",name=~\"$name\"}[$interval])","format": "time_series","interval": "","intervalFactor": 2,"legendFormat": "{{name}}","metric": "","refId": "A","step": 10}],"thresholds": [],"timeFrom": null,"timeShift": null,"title": "Merging time","tooltip": {"msResolution": false,"shared": true,"sort": 2,"value_type": "cumulative"},"transparent": false,"type": "graph","xaxis": {"buckets": null,"mode": "time","name": null,"show": true,"values": []},"yaxes": [{"format": "s","label": "Time","logBase": 1,"max": null,"min": null,"show": true},{"format": "short","label": null,"logBase": 1,"max": null,"min": null,"show": false}]},{"aliasColors": {},"bars": false,"dashLength": 10,"dashes": false,"datasource": "Prometheus","editable": true,"error": false,"fill": 1,"grid": {},"gridPos": {"h": 10,"w": 12,"x": 12,"y": 25},"height": "400","id": 87,"legend": {"alignAsTable": true,"avg": true,"current": true,"hideEmpty": false,"hideZero": false,"max": true,"min": true,"rightSide": true,"show": true,"sort": "avg","sortDesc": true,"total": false,"values": true},"lines": true,"linewidth": 1,"links": [],"nullPointMode": "connected","percentage": false,"pointradius": 5,"points": false,"renderer": "flot","seriesOverrides": [],"spaceLength": 10,"stack": false,"steppedLine": false,"targets": [{"expr": "irate(elasticsearch_indices_store_throttle_time_seconds_total{cluster=\"$cluster\",name=~\"$name\"}[$interval])","format": "time_series","interval": "","intervalFactor": 2,"legendFormat": "{{name}}","metric": "","refId": "A","step": 10}],"thresholds": [],"timeFrom": null,"timeShift": null,"title": "Throttle time for index store","tooltip": {"msResolution": false,"shared": true,"sort": 2,"value_type": "cumulative"},"transparent": false,"type": "graph","xaxis": {"buckets": null,"mode": "time","name": null,"show": true,"values": []},"yaxes": [{"format": "s","label": "Time","logBase": 1,"max": null,"min": null,"show": true},{"format": "short","label": null,"logBase": 1,"max": null,"min": null,"show": false}]}],"repeat": null,"title": "Times","type": "row"},{"collapsed": true,"gridPos": {"h": 1,"w": 24,"x": 0,"y": 25},"id": 99,"panels": [{"aliasColors": {},"bars": false,"dashLength": 10,"dashes": false,"datasource": "Prometheus","editable": true,"error": false,"fill": 1,"grid": {},"gridPos": {"h": 10,"w": 24,"x": 0,"y": 16},"height": "400","id": 48,"legend": {"alignAsTable": true,"avg": true,"current": true,"hideEmpty": false,"hideZero": true,"max": true,"min": true,"rightSide": true,"show": true,"sort": "avg","sortDesc": true,"total": false,"values": true},"lines": true,"linewidth": 1,"links": [],"nullPointMode": "connected","percentage": false,"pointradius": 5,"points": false,"renderer": "flot","seriesOverrides": [],"spaceLength": 10,"stack": true,"steppedLine": false,"targets": [{"expr": "rate(elasticsearch_indices_indexing_index_total{cluster=\"$cluster\",name=~\"$name\"}[$interval])","format": "time_series","interval": "","intervalFactor": 2,"legendFormat": "{{name}}: indexing","metric": "","refId": "A","step": 10},{"expr": "rate(elasticsearch_indices_search_query_total{cluster=\"$cluster\",name=~\"$name\"}[$interval])","format": "time_series","intervalFactor": 2,"legendFormat": "{{name}}: query","refId": "B","step": 10},{"expr": "rate(elasticsearch_indices_search_fetch_total{cluster=\"$cluster\",name=~\"$name\"}[$interval])","format": "time_series","intervalFactor": 2,"legendFormat": "{{name}}: fetch","refId": "C","step": 10},{"expr": "rate(elasticsearch_indices_merges_total{cluster=\"$cluster\",name=~\"$name\"}[$interval])","format": "time_series","intervalFactor": 2,"legendFormat": "{{name}}: merges","refId": "D","step": 10},{"expr": "rate(elasticsearch_indices_refresh_total{cluster=\"$cluster\",name=~\"$name\"}[$interval])","format": "time_series","intervalFactor": 2,"legendFormat": "{{name}}: refresh","refId": "E","step": 10},{"expr": "rate(elasticsearch_indices_flush_total{cluster=\"$cluster\",name=~\"$name\"}[$interval])","format": "time_series","intervalFactor": 2,"legendFormat": "{{name}}: flush","refId": "F","step": 10},{"expr": "rate(elasticsearch_indices_get_exists_total{cluster=\"$cluster\",name=~\"$name\"}[$interval])","format": "time_series","intervalFactor": 2,"legendFormat": "{{name}}: get_exists","refId": "G","step": 10},{"expr": "rate(elasticsearch_indices_get_missing_total{cluster=\"$cluster\",name=~\"$name\"}[$interval])","format": "time_series","intervalFactor": 2,"legendFormat": "{{name}}: get_missing","refId": "H","step": 10},{"expr": "rate(elasticsearch_indices_get_tota{cluster=\"$cluster\",name=~\"$name\"}[$interval])","format": "time_series","intervalFactor": 2,"legendFormat": "{{name}}: get","refId": "I","step": 10},{"expr": "rate(elasticsearch_indices_indexing_delete_total{cluster=\"$cluster\",name=~\"$name\"}[$interval])","format": "time_series","intervalFactor": 2,"legendFormat": "{{name}}: indexing_delete","refId": "J","step": 10}],"thresholds": [],"timeFrom": null,"timeShift": null,"title": "Total Operations  rate","tooltip": {"msResolution": false,"shared": true,"sort": 2,"value_type": "cumulative"},"transparent": false,"type": "graph","xaxis": {"buckets": null,"mode": "time","name": null,"show": true,"values": []},"yaxes": [{"format": "short","label": "Operations/s","logBase": 1,"max": null,"min": null,"show": true},{"format": "short","label": null,"logBase": 1,"max": null,"min": null,"show": false}]},{"aliasColors": {},"bars": false,"dashLength": 10,"dashes": false,"datasource": "Prometheus","editable": true,"error": false,"fill": 1,"grid": {},"gridPos": {"h": 10,"w": 24,"x": 0,"y": 26},"height": "400","id": 49,"legend": {"alignAsTable": true,"avg": true,"current": true,"hideEmpty": false,"hideZero": true,"max": true,"min": true,"rightSide": true,"show": true,"sort": "avg","sortDesc": true,"total": false,"values": true},"lines": true,"linewidth": 1,"links": [],"nullPointMode": "connected","percentage": false,"pointradius": 5,"points": false,"renderer": "flot","seriesOverrides": [],"spaceLength": 10,"stack": false,"steppedLine": false,"targets": [{"expr": "irate(elasticsearch_indices_indexing_index_time_seconds_total{cluster=\"$cluster\",name=~\"$name\"}[$interval])","format": "time_series","interval": "","intervalFactor": 2,"legendFormat": "{{name}}: indexing","metric": "","refId": "A","step": 10},{"expr": "irate(elasticsearch_indices_search_query_time_seconds{cluster=\"$cluster\",name=~\"$name\"}[$interval])","format": "time_series","intervalFactor": 2,"legendFormat": "{{name}}: query","refId": "B","step": 10},{"expr": "irate(elasticsearch_indices_search_fetch_time_seconds{cluster=\"$cluster\",name=~\"$name\"}[$interval])","format": "time_series","intervalFactor": 2,"legendFormat": "{{name}}: fetch","refId": "C","step": 10},{"expr": "irate(elasticsearch_indices_merges_total_time_seconds_total{cluster=\"$cluster\",name=~\"$name\"}[$interval])","format": "time_series","intervalFactor": 2,"legendFormat": "{{name}}: merges","refId": "D","step": 10},{"expr": "irate(elasticsearch_indices_refresh_time_seconds_total{cluster=\"$cluster\",name=~\"$name\"}[$interval])","format": "time_series","intervalFactor": 2,"legendFormat": "{{name}}: refresh","refId": "E","step": 10},{"expr": "irate(elasticsearch_indices_flush_time_seconds{cluster=\"$cluster\",name=~\"$name\"}[$interval])","format": "time_series","intervalFactor": 2,"legendFormat": "{{name}}: flush","refId": "F","step": 10},{"expr": "irate(elasticsearch_indices_get_exists_time_seconds{cluster=\"$cluster\",name=~\"$name\"}[$interval])","format": "time_series","intervalFactor": 2,"legendFormat": "{{name}}: get_exists","refId": "G","step": 10},{"expr": "irate(elasticsearch_indices_get_time_seconds{cluster=\"$cluster\",name=~\"$name\"}[$interval])","format": "time_series","intervalFactor": 2,"legendFormat": "{{name}}: get_time","refId": "H","step": 10},{"expr": "irate(elasticsearch_indices_get_missing_time_seconds{cluster=\"$cluster\",name=~\"$name\"}[$interval])","format": "time_series","intervalFactor": 2,"legendFormat": "{{name}}: get_missing","refId": "I","step": 10},{"expr": "irate(elasticsearch_indices_indexing_delete_time_seconds_total{cluster=\"$cluster\",name=~\"$name\"}[$interval])","format": "time_series","intervalFactor": 2,"legendFormat": "{{name}}: indexing_delete","refId": "J","step": 10},{"expr": "irate(elasticsearch_indices_get_time_seconds{cluster=\"$cluster\",name=~\"$name\"}[$interval])","format": "time_series","intervalFactor": 2,"legendFormat": "{{name}}: get","refId": "K","step": 10}],"thresholds": [],"timeFrom": null,"timeShift": null,"title": "Total Operations time","tooltip": {"msResolution": false,"shared": true,"sort": 2,"value_type": "cumulative"},"transparent": false,"type": "graph","xaxis": {"buckets": null,"mode": "time","name": null,"show": true,"values": []},"yaxes": [{"format": "s","label": "Time","logBase": 1,"max": null,"min": null,"show": true},{"format": "short","label": null,"logBase": 1,"max": null,"min": null,"show": false}]}],"repeat": null,"title": "Total Operations stats","type": "row"},{"collapsed": true,"gridPos": {"h": 1,"w": 24,"x": 0,"y": 26},"id": 100,"panels": [{"aliasColors": {},"bars": false,"dashLength": 10,"dashes": false,"datasource": "Prometheus","editable": true,"error": false,"fill": 1,"gridPos": {"h": 19,"w": 6,"x": 0,"y": 17},"id": 45,"legend": {"alignAsTable": true,"avg": true,"current": false,"hideZero": true,"max": true,"min": true,"show": true,"sort": null,"sortDesc": null,"total": false,"values": true},"lines": true,"linewidth": 1,"links": [],"nullPointMode": "connected","percentage": false,"pointradius": 5,"points": false,"renderer": "flot","seriesOverrides": [],"spaceLength": 10,"stack": false,"steppedLine": false,"targets": [{"expr": "irate(elasticsearch_thread_pool_rejected_count{cluster=\"$cluster\",name=~\"$name\"}[$interval])","format": "time_series","intervalFactor": 2,"legendFormat": "{{name}}: {{ type }}","refId": "A","step": 20}],"thresholds": [],"timeFrom": null,"timeShift": null,"title": "Thread Pool operations rejected","tooltip": {"msResolution": false,"shared": true,"sort": 2,"value_type": "individual"},"type": "graph","xaxis": {"buckets": null,"mode": "time","name": null,"show": true,"values": []},"yaxes": [{"format": "short","label": null,"logBase": 1,"max": null,"min": null,"show": true},{"format": "short","label": null,"logBase": 1,"max": null,"min": null,"show": true}]},{"aliasColors": {},"bars": false,"dashLength": 10,"dashes": false,"datasource": "Prometheus","editable": true,"error": false,"fill": 1,"gridPos": {"h": 19,"w": 6,"x": 6,"y": 17},"id": 46,"legend": {"alignAsTable": true,"avg": true,"current": false,"hideZero": true,"max": true,"min": true,"show": true,"sort": "avg","sortDesc": true,"total": false,"values": true},"lines": true,"linewidth": 1,"links": [],"nullPointMode": "connected","percentage": false,"pointradius": 5,"points": false,"renderer": "flot","seriesOverrides": [],"spaceLength": 10,"stack": false,"steppedLine": false,"targets": [{"expr": "elasticsearch_thread_pool_active_count{cluster=\"$cluster\",name=~\"$name\"}","format": "time_series","intervalFactor": 2,"legendFormat": "{{name}}: {{ type }}","refId": "A","step": 20}],"thresholds": [],"timeFrom": null,"timeShift": null,"title": "Thread Pool operations queued","tooltip": {"msResolution": false,"shared": true,"sort": 2,"value_type": "individual"},"type": "graph","xaxis": {"buckets": null,"mode": "time","name": null,"show": true,"values": []},"yaxes": [{"format": "short","label": null,"logBase": 1,"max": null,"min": null,"show": true},{"format": "short","label": null,"logBase": 1,"max": null,"min": null,"show": true}]},{"aliasColors": {},"bars": false,"dashLength": 10,"dashes": false,"datasource": "Prometheus","editable": true,"error": false,"fill": 1,"gridPos": {"h": 19,"w": 6,"x": 12,"y": 17},"height": "","id": 43,"legend": {"alignAsTable": true,"avg": true,"current": false,"hideZero": true,"max": true,"min": true,"show": true,"sort": "avg","sortDesc": true,"total": false,"values": true},"lines": true,"linewidth": 1,"links": [],"nullPointMode": "connected","percentage": false,"pointradius": 5,"points": false,"renderer": "flot","seriesOverrides": [],"spaceLength": 10,"stack": false,"steppedLine": false,"targets": [{"expr": "elasticsearch_thread_pool_active_count{cluster=\"$cluster\",name=~\"$name\"}","format": "time_series","intervalFactor": 2,"legendFormat": "{{name}}: {{ type }}","refId": "A","step": 20}],"thresholds": [],"timeFrom": null,"timeShift": null,"title": "Thread Pool threads active","tooltip": {"msResolution": false,"shared": true,"sort": 2,"value_type": "individual"},"type": "graph","xaxis": {"buckets": null,"mode": "time","name": null,"show": true,"values": []},"yaxes": [{"format": "short","label": null,"logBase": 1,"max": null,"min": null,"show": true},{"format": "short","label": null,"logBase": 1,"max": null,"min": null,"show": true}]},{"aliasColors": {},"bars": false,"dashLength": 10,"dashes": false,"datasource": "Prometheus","editable": true,"error": false,"fill": 1,"gridPos": {"h": 19,"w": 6,"x": 18,"y": 17},"id": 44,"legend": {"alignAsTable": true,"avg": true,"current": false,"hideZero": true,"max": true,"min": true,"show": true,"sort": "avg","sortDesc": true,"total": false,"values": true},"lines": true,"linewidth": 1,"links": [],"nullPointMode": "connected","percentage": false,"pointradius": 5,"points": false,"renderer": "flot","seriesOverrides": [],"spaceLength": 10,"stack": false,"steppedLine": false,"targets": [{"expr": "irate(elasticsearch_thread_pool_completed_count{cluster=\"$cluster\",name=~\"$name\"}[$interval])","format": "time_series","intervalFactor": 2,"legendFormat": "{{name}}: {{ type }}","refId": "A","step": 20}],"thresholds": [],"timeFrom": null,"timeShift": null,"title": "Thread Pool operations completed","tooltip": {"msResolution": false,"shared": true,"sort": 2,"value_type": "individual"},"type": "graph","xaxis": {"buckets": null,"mode": "time","name": null,"show": true,"values": []},"yaxes": [{"format": "short","label": null,"logBase": 1,"max": null,"min": null,"show": true},{"format": "short","label": null,"logBase": 1,"max": null,"min": null,"show": true}]}],"repeat": null,"title": "Thread Pool","type": "row"},{"collapsed": true,"gridPos": {"h": 1,"w": 24,"x": 0,"y": 27},"id": 101,"panels": [{"aliasColors": {},"bars": false,"dashLength": 10,"dashes": false,"datasource": "Prometheus","editable": true,"error": false,"fill": 1,"grid": {},"gridPos": {"h": 10,"w": 12,"x": 0,"y": 18},"height": "400","id": 4,"legend": {"alignAsTable": true,"avg": true,"current": true,"hideEmpty": false,"hideZero": false,"max": true,"min": true,"rightSide": false,"show": true,"sort": "avg","sortDesc": true,"total": false,"values": true},"lines": true,"linewidth": 1,"links": [],"nullPointMode": "connected","percentage": false,"pointradius": 5,"points": false,"renderer": "flot","seriesOverrides": [],"spaceLength": 10,"stack": true,"steppedLine": false,"targets": [{"expr": "elasticsearch_indices_fielddata_memory_size_bytes{cluster=\"$cluster\",name=~\"$name\"}","format": "time_series","interval": "","intervalFactor": 2,"legendFormat": "{{name}}","metric": "","refId": "A","step": 20}],"thresholds": [],"timeFrom": null,"timeShift": null,"title": "Field data memory size","tooltip": {"msResolution": false,"shared": true,"sort": 0,"value_type": "cumulative"},"transparent": false,"type": "graph","xaxis": {"buckets": null,"mode": "time","name": null,"show": true,"values": []},"yaxes": [{"format": "bytes","label": "Memory","logBase": 1,"max": null,"min": null,"show": true},{"format": "short","label": null,"logBase": 1,"max": null,"min": null,"show": false}]},{"aliasColors": {},"bars": false,"dashLength": 10,"dashes": false,"datasource": "Prometheus","editable": true,"error": false,"fill": 1,"grid": {},"gridPos": {"h": 10,"w": 12,"x": 12,"y": 18},"height": "400","id": 34,"legend": {"alignAsTable": true,"avg": true,"current": true,"hideEmpty": false,"hideZero": false,"max": true,"min": true,"rightSide": false,"show": true,"sort": "avg","sortDesc": true,"total": false,"values": true},"lines": true,"linewidth": 1,"links": [],"nullPointMode": "connected","percentage": false,"pointradius": 5,"points": false,"renderer": "flot","seriesOverrides": [],"spaceLength": 10,"stack": true,"steppedLine": false,"targets": [{"expr": "rate(elasticsearch_indices_fielddata_evictions{cluster=\"$cluster\",name=~\"$name\"}[$interval])","format": "time_series","interval": "","intervalFactor": 2,"legendFormat": "{{name}}","metric": "","refId": "A","step": 20}],"thresholds": [],"timeFrom": null,"timeShift": null,"title": "Field data evictions","tooltip": {"msResolution": false,"shared": true,"sort": 0,"value_type": "cumulative"},"transparent": false,"type": "graph","xaxis": {"buckets": null,"mode": "time","name": null,"show": true,"values": []},"yaxes": [{"format": "short","label": "Evictions/s","logBase": 1,"max": null,"min": null,"show": true},{"format": "short","label": null,"logBase": 1,"max": null,"min": null,"show": false}]},{"aliasColors": {},"bars": false,"dashLength": 10,"dashes": false,"datasource": "Prometheus","editable": true,"error": false,"fill": 1,"grid": {},"gridPos": {"h": 10,"w": 8,"x": 0,"y": 28},"height": "400","id": 35,"legend": {"alignAsTable": true,"avg": true,"current": true,"hideEmpty": false,"hideZero": false,"max": true,"min": true,"rightSide": false,"show": true,"sort": "avg","sortDesc": true,"total": false,"values": true},"lines": true,"linewidth": 1,"links": [],"nullPointMode": "connected","percentage": false,"pointradius": 5,"points": false,"renderer": "flot","seriesOverrides": [],"spaceLength": 10,"stack": true,"steppedLine": false,"targets": [{"expr": "elasticsearch_indices_query_cache_memory_size_bytes{cluster=\"$cluster\",name=~\"$name\"}","format": "time_series","interval": "","intervalFactor": 2,"legendFormat": "{{name}}","metric": "","refId": "A","step": 20}],"thresholds": [],"timeFrom": null,"timeShift": null,"title": "Query cache size","tooltip": {"msResolution": false,"shared": true,"sort": 0,"value_type": "cumulative"},"transparent": false,"type": "graph","xaxis": {"buckets": null,"mode": "time","name": null,"show": true,"values": []},"yaxes": [{"format": "bytes","label": "Size","logBase": 1,"max": null,"min": null,"show": true},{"format": "short","label": null,"logBase": 1,"max": null,"min": null,"show": false}]},{"aliasColors": {},"bars": false,"dashLength": 10,"dashes": false,"datasource": "Prometheus","editable": true,"error": false,"fill": 1,"grid": {},"gridPos": {"h": 10,"w": 8,"x": 8,"y": 28},"height": "400","id": 36,"legend": {"alignAsTable": true,"avg": true,"current": true,"hideEmpty": false,"hideZero": false,"max": true,"min": true,"rightSide": false,"show": true,"sort": "avg","sortDesc": true,"total": false,"values": true},"lines": true,"linewidth": 1,"links": [],"nullPointMode": "connected","percentage": false,"pointradius": 5,"points": false,"renderer": "flot","seriesOverrides": [],"spaceLength": 10,"stack": true,"steppedLine": false,"targets": [{"expr": "rate(elasticsearch_indices_query_cache_evictions{cluster=\"$cluster\",name=~\"$name\"}[$interval])","format": "time_series","interval": "","intervalFactor": 2,"legendFormat": "{{name}}","metric": "","refId": "A","step": 20}],"thresholds": [],"timeFrom": null,"timeShift": null,"title": "Query cache evictions","tooltip": {"msResolution": false,"shared": true,"sort": 0,"value_type": "cumulative"},"transparent": false,"type": "graph","xaxis": {"buckets": null,"mode": "time","name": null,"show": true,"values": []},"yaxes": [{"format": "short","label": "Evictions/s","logBase": 1,"max": null,"min": null,"show": true},{"format": "short","label": null,"logBase": 1,"max": null,"min": null,"show": false}]},{"aliasColors": {},"bars": false,"dashLength": 10,"dashes": false,"datasource": "Prometheus","editable": true,"error": false,"fill": 1,"grid": {},"gridPos": {"h": 10,"w": 8,"x": 16,"y": 28},"height": "400","id": 84,"legend": {"alignAsTable": true,"avg": true,"current": true,"hideEmpty": false,"hideZero": false,"max": true,"min": true,"rightSide": false,"show": true,"sort": "avg","sortDesc": true,"total": false,"values": true},"lines": true,"linewidth": 1,"links": [],"nullPointMode": "connected","percentage": false,"pointradius": 5,"points": false,"renderer": "flot","seriesOverrides": [],"spaceLength": 10,"stack": true,"steppedLine": false,"targets": [{"expr": "rate(elasticsearch_indices_filter_cache_evictions{cluster=\"$cluster\",name=~\"$name\"}[$interval])","format": "time_series","interval": "","intervalFactor": 2,"legendFormat": "{{name}}","metric": "","refId": "A","step": 20}],"thresholds": [],"timeFrom": null,"timeShift": null,"title": "Evictions from filter cache","tooltip": {"msResolution": false,"shared": true,"sort": 0,"value_type": "cumulative"},"transparent": false,"type": "graph","xaxis": {"buckets": null,"mode": "time","name": null,"show": true,"values": []},"yaxes": [{"format": "short","label": "Evictions/s","logBase": 1,"max": null,"min": null,"show": true},{"format": "short","label": null,"logBase": 1,"max": null,"min": null,"show": false}]}],"repeat": null,"title": "Caches","type": "row"},{"collapsed": true,"gridPos": {"h": 1,"w": 24,"x": 0,"y": 28},"id": 102,"panels": [{"aliasColors": {},"bars": false,"dashLength": 10,"dashes": false,"datasource": "Prometheus","fill": 1,"gridPos": {"h": 7,"w": 24,"x": 0,"y": 19},"id": 85,"legend": {"alignAsTable": true,"avg": true,"current": false,"max": true,"min": true,"rightSide": true,"show": true,"sort": "avg","sortDesc": true,"total": false,"values": true},"lines": true,"linewidth": 1,"links": [],"nullPointMode": "null","percentage": false,"pointradius": 5,"points": false,"renderer": "flot","seriesOverrides": [],"spaceLength": 10,"stack": true,"steppedLine": false,"targets": [{"expr": "elasticsearch_indices_segments_count{cluster=\"$cluster\",name=~\"$name\"}","format": "time_series","intervalFactor": 2,"legendFormat": "{{name}}","refId": "A"}],"thresholds": [],"timeFrom": null,"timeShift": null,"title": "Count of index segments","tooltip": {"shared": true,"sort": 2,"value_type": "individual"},"type": "graph","xaxis": {"buckets": null,"mode": "time","name": null,"show": true,"values": []},"yaxes": [{"format": "short","label": null,"logBase": 1,"max": null,"min": null,"show": true},{"format": "short","label": null,"logBase": 1,"max": null,"min": null,"show": false}]},{"aliasColors": {},"bars": false,"dashLength": 10,"dashes": false,"datasource": "Prometheus","fill": 1,"gridPos": {"h": 7,"w": 24,"x": 0,"y": 26},"id": 86,"legend": {"alignAsTable": true,"avg": true,"current": false,"max": true,"min": true,"rightSide": true,"show": true,"sort": "avg","sortDesc": true,"total": false,"values": true},"lines": true,"linewidth": 1,"links": [],"nullPointMode": "null","percentage": false,"pointradius": 5,"points": false,"renderer": "flot","seriesOverrides": [],"spaceLength": 10,"stack": true,"steppedLine": false,"targets": [{"expr": "elasticsearch_indices_segments_memory_bytes{cluster=\"$cluster\",name=~\"$name\"}","format": "time_series","intervalFactor": 2,"legendFormat": "{{name}}","refId": "A"}],"thresholds": [],"timeFrom": null,"timeShift": null,"title": "Current memory size of segments in bytes","tooltip": {"shared": true,"sort": 2,"value_type": "individual"},"type": "graph","xaxis": {"buckets": null,"mode": "time","name": null,"show": true,"values": []},"yaxes": [{"format": "bytes","label": null,"logBase": 1,"max": null,"min": null,"show": true},{"format": "short","label": null,"logBase": 1,"max": null,"min": null,"show": false}]}],"repeat": null,"title": "Segments","type": "row"},{"collapsed": true,"gridPos": {"h": 1,"w": 24,"x": 0,"y": 29},"id": 103,"panels": [{"aliasColors": {},"bars": false,"dashLength": 10,"dashes": false,"datasource": "Prometheus","fill": 1,"gridPos": {"h": 7,"w": 24,"x": 0,"y": 20},"id": 75,"legend": {"alignAsTable": true,"avg": true,"current": false,"max": true,"min": true,"rightSide": true,"show": true,"sort": "avg","sortDesc": true,"total": false,"values": true},"lines": true,"linewidth": 1,"links": [],"nullPointMode": "null","percentage": false,"pointradius": 5,"points": false,"renderer": "flot","seriesOverrides": [],"spaceLength": 10,"stack": false,"steppedLine": false,"targets": [{"expr": "elasticsearch_indices_docs{instance=~\"$instance\"}","format": "time_series","intervalFactor": 2,"legendFormat": "{{index}}","refId": "A"}],"thresholds": [],"timeFrom": null,"timeShift": null,"title": "Count of documents with only primary shards","tooltip": {"shared": true,"sort": 2,"value_type": "individual"},"type": "graph","xaxis": {"buckets": null,"mode": "time","name": null,"show": true,"values": []},"yaxes": [{"format": "short","label": null,"logBase": 1,"max": null,"min": null,"show": true},{"format": "short","label": null,"logBase": 1,"max": null,"min": null,"show": false}]},{"aliasColors": {},"bars": false,"dashLength": 10,"dashes": false,"datasource": "Prometheus","fill": 1,"gridPos": {"h": 7,"w": 24,"x": 0,"y": 27},"id": 83,"legend": {"alignAsTable": true,"avg": true,"current": false,"max": true,"min": true,"rightSide": true,"show": true,"sort": "avg","sortDesc": true,"total": false,"values": true},"lines": true,"linewidth": 1,"links": [],"nullPointMode": "null","percentage": false,"pointradius": 5,"points": false,"renderer": "flot","seriesOverrides": [],"spaceLength": 10,"stack": false,"steppedLine": false,"targets": [{"expr": "elasticsearch_indices_store_size_bytes{instance=~\"$instance\"}","format": "time_series","intervalFactor": 2,"legendFormat": "{{index}}","refId": "A"}],"thresholds": [],"timeFrom": null,"timeShift": null,"title": "Total size of stored index data in bytes with only primary shards on all nodes","tooltip": {"shared": true,"sort": 2,"value_type": "individual"},"type": "graph","xaxis": {"buckets": null,"mode": "time","name": null,"show": true,"values": []},"yaxes": [{"format": "bytes","label": null,"logBase": 1,"max": null,"min": null,"show": true},{"format": "short","label": null,"logBase": 1,"max": null,"min": null,"show": false}]},{"aliasColors": {},"bars": false,"dashLength": 10,"dashes": false,"datasource": "Prometheus","fill": 1,"gridPos": {"h": 7,"w": 24,"x": 0,"y": 34},"id": 76,"legend": {"alignAsTable": true,"avg": true,"current": false,"max": true,"min": true,"rightSide": true,"show": true,"sort": "avg","sortDesc": true,"total": false,"values": true},"lines": true,"linewidth": 1,"links": [],"nullPointMode": "null","percentage": false,"pointradius": 5,"points": false,"renderer": "flot","seriesOverrides": [],"spaceLength": 10,"stack": false,"steppedLine": false,"targets": [{"expr": "elasticsearch_indices_store_size_bytes_total{instance=~\"$instance\"}","format": "time_series","intervalFactor": 2,"legendFormat": "{{index}}","refId": "A"}],"thresholds": [],"timeFrom": null,"timeShift": null,"title": "Total size of stored index data in bytes with all shards on all nodes","tooltip": {"shared": true,"sort": 2,"value_type": "individual"},"type": "graph","xaxis": {"buckets": null,"mode": "time","name": null,"show": true,"values": []},"yaxes": [{"format": "bytes","label": null,"logBase": 1,"max": null,"min": null,"show": true},{"format": "short","label": null,"logBase": 1,"max": null,"min": null,"show": false}]}],"repeat": null,"title": "Indices: Count of documents and Total size","type": "row"},{"collapsed": false,"gridPos": {"h": 1,"w": 24,"x": 0,"y": 30},"id": 104,"panels": [],"repeat": null,"title": "Indices: Index writer","type": "row"},{"aliasColors": {},"bars": false,"dashLength": 10,"dashes": false,"datasource": "${DS_PROMETHEUS}","fill": 1,"gridPos": {"h": 7,"w": 24,"x": 0,"y": 31},"id": 61,"legend": {"alignAsTable": true,"avg": true,"current": false,"hideZero": true,"max": true,"min": true,"rightSide": true,"show": true,"sort": "avg","sortDesc": true,"total": false,"values": true},"lines": true,"linewidth": 1,"links": [],"nullPointMode": "null","percentage": false,"pointradius": 5,"points": false,"renderer": "flot","seriesOverrides": [],"spaceLength": 10,"stack": false,"steppedLine": false,"targets": [{"expr": "elasticsearch_indices_segments_memory_bytes{instance=~\"$instance\"}","format": "time_series","intervalFactor": 2,"legendFormat": "{{index}}","refId": "A"}],"thresholds": [],"timeFrom": null,"timeShift": null,"title": "Index writer with only primary shards on all nodes in bytes","tooltip": {"shared": true,"sort": 2,"value_type": "individual"},"type": "graph","xaxis": {"buckets": null,"mode": "time","name": null,"show": true,"values": []},"yaxes": [{"format": "bytes","label": null,"logBase": 1,"max": null,"min": null,"show": true},{"format": "short","label": null,"logBase": 1,"max": null,"min": null,"show": false}]},{"aliasColors": {},"bars": false,"dashLength": 10,"dashes": false,"datasource": "${DS_PROMETHEUS}","fill": 1,"gridPos": {"h": 7,"w": 24,"x": 0,"y": 38},"id": 62,"legend": {"alignAsTable": true,"avg": true,"current": false,"hideZero": true,"max": true,"min": true,"rightSide": true,"show": true,"sort": "avg","sortDesc": true,"total": false,"values": true},"lines": true,"linewidth": 1,"links": [],"nullPointMode": "null","percentage": false,"pointradius": 5,"points": false,"renderer": "flot","seriesOverrides": [],"spaceLength": 10,"stack": false,"steppedLine": false,"targets": [{"expr": "elasticsearch_indices_segments_count{instance=~\"$instance\"}","format": "time_series","intervalFactor": 2,"legendFormat": "{{index}}","refId": "A"}],"thresholds": [],"timeFrom": null,"timeShift": null,"title": "Index writer with all shards on all nodes in bytes","tooltip": {"shared": true,"sort": 2,"value_type": "individual"},"type": "graph","xaxis": {"buckets": null,"mode": "time","name": null,"show": true,"values": []},"yaxes": [{"format": "bytes","label": null,"logBase": 1,"max": null,"min": null,"show": true},{"format": "short","label": null,"logBase": 1,"max": null,"min": null,"show": false}]},{"collapsed": true,"gridPos": {"h": 1,"w": 24,"x": 0,"y": 45},"id": 105,"panels": [{"aliasColors": {},"bars": false,"dashLength": 10,"dashes": false,"datasource": "Prometheus","fill": 1,"gridPos": {"h": 7,"w": 24,"x": 0,"y": 22},"id": 55,"legend": {"alignAsTable": true,"avg": true,"current": false,"max": true,"min": true,"rightSide": true,"show": true,"sort": null,"sortDesc": null,"total": false,"values": true},"lines": true,"linewidth": 1,"links": [],"nullPointMode": "null","percentage": false,"pointradius": 5,"points": false,"renderer": "flot","seriesOverrides": [],"spaceLength": 10,"stack": false,"steppedLine": false,"targets": [{"expr": "elasticsearch_indices_segments_count{instance=~\"$instance\"}","format": "time_series","intervalFactor": 2,"legendFormat": "{{index}}","refId": "A"}],"thresholds": [],"timeFrom": null,"timeShift": null,"title": "Segments with only primary shards on all nodes","tooltip": {"shared": true,"sort": 2,"value_type": "individual"},"type": "graph","xaxis": {"buckets": null,"mode": "time","name": null,"show": true,"values": []},"yaxes": [{"format": "short","label": null,"logBase": 1,"max": null,"min": null,"show": true},{"format": "short","label": null,"logBase": 1,"max": null,"min": null,"show": true}]},{"aliasColors": {},"bars": false,"dashLength": 10,"dashes": false,"datasource": "Prometheus","fill": 1,"gridPos": {"h": 7,"w": 24,"x": 0,"y": 29},"id": 56,"legend": {"alignAsTable": true,"avg": true,"current": false,"max": true,"min": true,"rightSide": true,"show": true,"sort": "avg","sortDesc": true,"total": false,"values": true},"lines": true,"linewidth": 1,"links": [],"nullPointMode": "null","percentage": false,"pointradius": 5,"points": false,"renderer": "flot","seriesOverrides": [],"spaceLength": 10,"stack": false,"steppedLine": false,"targets": [{"expr": "elasticsearch_indices_segments_count{instance=~\"$instance\"}","format": "time_series","intervalFactor": 2,"legendFormat": "{{index}}","refId": "A"}],"thresholds": [],"timeFrom": null,"timeShift": null,"title": "Segments with all shards on all nodes","tooltip": {"shared": true,"sort": 2,"value_type": "individual"},"type": "graph","xaxis": {"buckets": null,"mode": "time","name": null,"show": true,"values": []},"yaxes": [{"format": "short","label": null,"logBase": 1,"max": null,"min": null,"show": true},{"format": "short","label": null,"logBase": 1,"max": null,"min": null,"show": true}]},{"aliasColors": {},"bars": false,"dashLength": 10,"dashes": false,"datasource": "Prometheus","fill": 1,"gridPos": {"h": 7,"w": 24,"x": 0,"y": 36},"id": 65,"legend": {"alignAsTable": true,"avg": true,"current": false,"max": true,"min": true,"rightSide": true,"show": true,"sort": "avg","sortDesc": true,"total": false,"values": true},"lines": true,"linewidth": 1,"links": [],"nullPointMode": "null","percentage": false,"pointradius": 5,"points": false,"renderer": "flot","seriesOverrides": [],"spaceLength": 10,"stack": false,"steppedLine": false,"targets": [{"expr": "elasticsearch_indices_segment_memory_bytes_primary{instance=~\"$instance\"}","format": "time_series","intervalFactor": 2,"legendFormat": "{{index}}","refId": "A"}],"thresholds": [],"timeFrom": null,"timeShift": null,"title": "Size of segments with only primary shards on all nodes in bytes","tooltip": {"shared": true,"sort": 2,"value_type": "individual"},"type": "graph","xaxis": {"buckets": null,"mode": "time","name": null,"show": true,"values": []},"yaxes": [{"format": "bytes","label": null,"logBase": 1,"max": null,"min": null,"show": true},{"format": "short","label": null,"logBase": 1,"max": null,"min": null,"show": false}]},{"aliasColors": {},"bars": false,"dashLength": 10,"dashes": false,"datasource": "Prometheus","fill": 1,"gridPos": {"h": 7,"w": 24,"x": 0,"y": 43},"id": 66,"legend": {"alignAsTable": true,"avg": true,"current": false,"max": true,"min": true,"rightSide": true,"show": true,"sort": "avg","sortDesc": true,"total": false,"values": true},"lines": true,"linewidth": 1,"links": [],"nullPointMode": "null","percentage": false,"pointradius": 5,"points": false,"renderer": "flot","seriesOverrides": [],"spaceLength": 10,"stack": false,"steppedLine": false,"targets": [{"expr": "elasticsearch_indices_segment_memory_bytes_total{instance=~\"$instance\"}","format": "time_series","intervalFactor": 2,"legendFormat": "{{index}}","refId": "A"}],"thresholds": [],"timeFrom": null,"timeShift": null,"title": "Size of segments with all shards on all nodes in bytes","tooltip": {"shared": true,"sort": 2,"value_type": "individual"},"type": "graph","xaxis": {"buckets": null,"mode": "time","name": null,"show": true,"values": []},"yaxes": [{"format": "bytes","label": null,"logBase": 1,"max": null,"min": null,"show": true},{"format": "short","label": null,"logBase": 1,"max": null,"min": null,"show": true}]}],"repeat": null,"title": "Indices: Segments","type": "row"},{"collapsed": true,"gridPos": {"h": 1,"w": 24,"x": 0,"y": 46},"id": 106,"panels": [{"aliasColors": {},"bars": false,"dashLength": 10,"dashes": false,"datasource": "Prometheus","fill": 1,"gridPos": {"h": 7,"w": 24,"x": 0,"y": 23},"id": 57,"legend": {"alignAsTable": true,"avg": true,"current": false,"max": true,"min": true,"rightSide": true,"show": true,"sort": "avg","sortDesc": true,"total": false,"values": true},"lines": true,"linewidth": 1,"links": [],"nullPointMode": "null","percentage": false,"pointradius": 5,"points": false,"renderer": "flot","seriesOverrides": [],"spaceLength": 10,"stack": false,"steppedLine": false,"targets": [{"expr": "elasticsearch_indices_segments_memory_bytes{instance=~\"$instance\"}","format": "time_series","intervalFactor": 2,"legendFormat": "{{index}}","refId": "A"}],"thresholds": [],"timeFrom": null,"timeShift": null,"title": "Doc values with only primary shards on all nodes in bytes","tooltip": {"shared": true,"sort": 2,"value_type": "individual"},"type": "graph","xaxis": {"buckets": null,"mode": "time","name": null,"show": true,"values": []},"yaxes": [{"format": "bytes","label": null,"logBase": 1,"max": null,"min": null,"show": true},{"format": "short","label": null,"logBase": 1,"max": null,"min": null,"show": false}]},{"aliasColors": {},"bars": false,"dashLength": 10,"dashes": false,"datasource": "Prometheus","fill": 1,"gridPos": {"h": 7,"w": 24,"x": 0,"y": 30},"id": 58,"legend": {"alignAsTable": true,"avg": true,"current": false,"max": true,"min": true,"rightSide": true,"show": true,"sort": "avg","sortDesc": true,"total": false,"values": true},"lines": true,"linewidth": 1,"links": [],"nullPointMode": "null","percentage": false,"pointradius": 5,"points": false,"renderer": "flot","seriesOverrides": [],"spaceLength": 10,"stack": false,"steppedLine": false,"targets": [{"expr": "elasticsearch_indices_segments_count{instance=~\"$instance\"}","format": "time_series","intervalFactor": 2,"legendFormat": "{{index}}","refId": "A"}],"thresholds": [],"timeFrom": null,"timeShift": null,"title": "Doc values with all shards on all nodes in bytes","tooltip": {"shared": true,"sort": 2,"value_type": "individual"},"type": "graph","xaxis": {"buckets": null,"mode": "time","name": null,"show": true,"values": []},"yaxes": [{"format": "bytes","label": null,"logBase": 1,"max": null,"min": null,"show": true},{"format": "short","label": null,"logBase": 1,"max": null,"min": null,"show": false}]}],"repeat": null,"title": "Indices: Doc values","type": "row"},{"collapsed": true,"gridPos": {"h": 1,"w": 24,"x": 0,"y": 47},"id": 107,"panels": [{"aliasColors": {},"bars": false,"dashLength": 10,"dashes": false,"datasource": "Prometheus","fill": 1,"gridPos": {"h": 7,"w": 24,"x": 0,"y": 24},"id": 59,"legend": {"alignAsTable": true,"avg": true,"current": false,"max": true,"min": true,"rightSide": true,"show": true,"sort": "avg","sortDesc": true,"total": false,"values": true},"lines": true,"linewidth": 1,"links": [],"nullPointMode": "null","percentage": false,"pointradius": 5,"points": false,"renderer": "flot","seriesOverrides": [],"spaceLength": 10,"stack": false,"steppedLine": false,"targets": [{"expr": "elasticsearch_indices_segments_memory_bytes{instance=~\"$instance\"}","format": "time_series","intervalFactor": 2,"legendFormat": "{{index}}","refId": "A"}],"thresholds": [],"timeFrom": null,"timeShift": null,"title": "Size of fields with only primary shards on all nodes in bytes","tooltip": {"shared": true,"sort": 2,"value_type": "individual"},"type": "graph","xaxis": {"buckets": null,"mode": "time","name": null,"show": true,"values": []},"yaxes": [{"format": "bytes","label": null,"logBase": 1,"max": null,"min": null,"show": true},{"format": "short","label": null,"logBase": 1,"max": null,"min": null,"show": false}]},{"aliasColors": {},"bars": false,"dashLength": 10,"dashes": false,"datasource": "Prometheus","fill": 1,"gridPos": {"h": 7,"w": 24,"x": 0,"y": 31},"id": 60,"legend": {"alignAsTable": true,"avg": true,"current": false,"max": true,"min": true,"rightSide": true,"show": true,"sort": "avg","sortDesc": true,"total": false,"values": true},"lines": true,"linewidth": 1,"links": [],"nullPointMode": "null","percentage": false,"pointradius": 5,"points": false,"renderer": "flot","seriesOverrides": [],"spaceLength": 10,"stack": false,"steppedLine": false,"targets": [{"expr": "elasticsearch_indices_segments_count{instance=~\"$instance\"}","format": "time_series","intervalFactor": 2,"legendFormat": "{{index}}","refId": "A"}],"thresholds": [],"timeFrom": null,"timeShift": null,"title": "Size of fields with all shards on all nodes in bytes","tooltip": {"shared": true,"sort": 2,"value_type": "individual"},"type": "graph","xaxis": {"buckets": null,"mode": "time","name": null,"show": true,"values": []},"yaxes": [{"format": "bytes","label": null,"logBase": 1,"max": null,"min": null,"show": true},{"format": "short","label": null,"logBase": 1,"max": null,"min": null,"show": false}]}],"repeat": null,"title": "Indices: Fields","type": "row"},{"collapsed": true,"gridPos": {"h": 1,"w": 24,"x": 0,"y": 48},"id": 108,"panels": [{"aliasColors": {},"bars": false,"dashLength": 10,"dashes": false,"datasource": "Prometheus","fill": 1,"gridPos": {"h": 7,"w": 24,"x": 0,"y": 25},"id": 63,"legend": {"alignAsTable": true,"avg": true,"current": false,"max": true,"min": true,"rightSide": true,"show": true,"sort": "avg","sortDesc": true,"total": false,"values": true},"lines": true,"linewidth": 1,"links": [],"nullPointMode": "null","percentage": false,"pointradius": 5,"points": false,"renderer": "flot","seriesOverrides": [],"spaceLength": 10,"stack": false,"steppedLine": false,"targets": [{"expr": "elasticsearch_indices_segments_memory_bytes{instance=~\"$instance\"}","format": "time_series","intervalFactor": 2,"legendFormat": "{{index}}","refId": "A"}],"thresholds": [],"timeFrom": null,"timeShift": null,"title": "Size of fixed bit with only primary shards on all nodes in bytes","tooltip": {"shared": true,"sort": 2,"value_type": "individual"},"type": "graph","xaxis": {"buckets": null,"mode": "time","name": null,"show": true,"values": []},"yaxes": [{"format": "bytes","label": null,"logBase": 1,"max": null,"min": null,"show": true},{"format": "short","label": null,"logBase": 1,"max": null,"min": null,"show": false}]},{"aliasColors": {},"bars": false,"dashLength": 10,"dashes": false,"datasource": "Prometheus","fill": 1,"gridPos": {"h": 7,"w": 24,"x": 0,"y": 32},"id": 64,"legend": {"alignAsTable": true,"avg": true,"current": false,"max": true,"min": true,"rightSide": true,"show": true,"sort": "avg","sortDesc": true,"total": false,"values": true},"lines": true,"linewidth": 1,"links": [],"nullPointMode": "null","percentage": false,"pointradius": 5,"points": false,"renderer": "flot","seriesOverrides": [],"spaceLength": 10,"stack": false,"steppedLine": false,"targets": [{"expr": "elasticsearch_indices_segments_count{instance=~\"$instance\"}","format": "time_series","intervalFactor": 2,"legendFormat": "{{index}}","refId": "A"}],"thresholds": [],"timeFrom": null,"timeShift": null,"title": "Size of fixed bit with all shards on all nodes in bytes","tooltip": {"shared": true,"sort": 2,"value_type": "individual"},"type": "graph","xaxis": {"buckets": null,"mode": "time","name": null,"show": true,"values": []},"yaxes": [{"format": "bytes","label": null,"logBase": 1,"max": null,"min": null,"show": true},{"format": "short","label": null,"logBase": 1,"max": null,"min": null,"show": false}]}],"repeat": null,"title": "Indices: Fixed bit","type": "row"},{"collapsed": true,"gridPos": {"h": 1,"w": 24,"x": 0,"y": 49},"id": 109,"panels": [{"aliasColors": {},"bars": false,"dashLength": 10,"dashes": false,"datasource": "Prometheus","fill": 1,"gridPos": {"h": 7,"w": 24,"x": 0,"y": 26},"id": 67,"legend": {"alignAsTable": true,"avg": true,"current": false,"max": true,"min": true,"rightSide": true,"show": true,"sort": "avg","sortDesc": true,"total": false,"values": true},"lines": true,"linewidth": 1,"links": [],"nullPointMode": "null","percentage": false,"pointradius": 5,"points": false,"renderer": "flot","seriesOverrides": [],"spaceLength": 10,"stack": false,"steppedLine": false,"targets": [{"expr": "elasticsearch_indices_segment_norms_memory_bytes_primary{instance=~\"$instance\"}","format": "time_series","intervalFactor": 2,"legendFormat": "{{index}}","refId": "A"}],"thresholds": [],"timeFrom": null,"timeShift": null,"title": "Size of norms with only primary shards on all nodes in bytes","tooltip": {"shared": true,"sort": 2,"value_type": "individual"},"type": "graph","xaxis": {"buckets": null,"mode": "time","name": null,"show": true,"values": []},"yaxes": [{"format": "bytes","label": null,"logBase": 1,"max": null,"min": null,"show": true},{"format": "short","label": null,"logBase": 1,"max": null,"min": null,"show": false}]},{"aliasColors": {},"bars": false,"dashLength": 10,"dashes": false,"datasource": "Prometheus","fill": 1,"gridPos": {"h": 7,"w": 24,"x": 0,"y": 33},"id": 68,"legend": {"alignAsTable": true,"avg": true,"current": false,"max": true,"min": true,"rightSide": true,"show": true,"sort": "avg","sortDesc": true,"total": false,"values": true},"lines": true,"linewidth": 1,"links": [],"nullPointMode": "null","percentage": false,"pointradius": 5,"points": false,"renderer": "flot","seriesOverrides": [],"spaceLength": 10,"stack": false,"steppedLine": false,"targets": [{"expr": "elasticsearch_indices_segment_norms_memory_bytes_total{instance=~\"$instance\"}","format": "time_series","intervalFactor": 2,"legendFormat": "{{index}}","refId": "A"}],"thresholds": [],"timeFrom": null,"timeShift": null,"title": "Size of norms with all shards on all nodes in bytes","tooltip": {"shared": true,"sort": 2,"value_type": "individual"},"type": "graph","xaxis": {"buckets": null,"mode": "time","name": null,"show": true,"values": []},"yaxes": [{"format": "bytes","label": null,"logBase": 1,"max": null,"min": null,"show": true},{"format": "short","label": null,"logBase": 1,"max": null,"min": null,"show": false}]}],"repeat": null,"title": "Indices: Norms","type": "row"},{"collapsed": true,"gridPos": {"h": 1,"w": 24,"x": 0,"y": 50},"id": 110,"panels": [{"aliasColors": {},"bars": false,"dashLength": 10,"dashes": false,"datasource": "Prometheus","fill": 1,"gridPos": {"h": 7,"w": 24,"x": 0,"y": 27},"id": 69,"legend": {"alignAsTable": true,"avg": true,"current": false,"max": true,"min": true,"rightSide": true,"show": true,"sort": "avg","sortDesc": true,"total": false,"values": true},"lines": true,"linewidth": 1,"links": [],"nullPointMode": "null","percentage": false,"pointradius": 5,"points": false,"renderer": "flot","seriesOverrides": [],"spaceLength": 10,"stack": false,"steppedLine": false,"targets": [{"expr": "elasticsearch_indices_segment_points_memory_bytes_primary{instance=~\"$instance\"}","format": "time_series","intervalFactor": 2,"legendFormat": "{{index}}","refId": "A"}],"thresholds": [],"timeFrom": null,"timeShift": null,"title": "Size of points with only primary shards on all nodes in bytes","tooltip": {"shared": true,"sort": 2,"value_type": "individual"},"type": "graph","xaxis": {"buckets": null,"mode": "time","name": null,"show": true,"values": []},"yaxes": [{"format": "bytes","label": null,"logBase": 1,"max": null,"min": null,"show": true},{"format": "short","label": null,"logBase": 1,"max": null,"min": null,"show": false}]},{"aliasColors": {},"bars": false,"dashLength": 10,"dashes": false,"datasource": "Prometheus","fill": 1,"gridPos": {"h": 7,"w": 24,"x": 0,"y": 34},"id": 70,"legend": {"alignAsTable": true,"avg": true,"current": false,"max": true,"min": true,"rightSide": true,"show": true,"sort": "avg","sortDesc": true,"total": false,"values": true},"lines": true,"linewidth": 1,"links": [],"nullPointMode": "null","percentage": false,"pointradius": 5,"points": false,"renderer": "flot","seriesOverrides": [],"spaceLength": 10,"stack": false,"steppedLine": false,"targets": [{"expr": "elasticsearch_indices_segment_points_memory_bytes_total{instance=~\"$instance\"}","format": "time_series","intervalFactor": 2,"legendFormat": "{{index}}","refId": "A"}],"thresholds": [],"timeFrom": null,"timeShift": null,"title": "Size of points with all shards on all nodes in bytes","tooltip": {"shared": true,"sort": 2,"value_type": "individual"},"type": "graph","xaxis": {"buckets": null,"mode": "time","name": null,"show": true,"values": []},"yaxes": [{"format": "bytes","label": null,"logBase": 1,"max": null,"min": null,"show": true},{"format": "short","label": null,"logBase": 1,"max": null,"min": null,"show": false}]}],"repeat": null,"title": "Indices: Points","type": "row"},{"collapsed": true,"gridPos": {"h": 1,"w": 24,"x": 0,"y": 51},"id": 111,"panels": [{"aliasColors": {},"bars": false,"dashLength": 10,"dashes": false,"datasource": "Prometheus","fill": 1,"gridPos": {"h": 7,"w": 24,"x": 0,"y": 28},"id": 71,"legend": {"alignAsTable": true,"avg": true,"current": false,"max": true,"min": true,"rightSide": true,"show": true,"sort": "avg","sortDesc": true,"total": false,"values": true},"lines": true,"linewidth": 1,"links": [],"nullPointMode": "null","percentage": false,"pointradius": 5,"points": false,"renderer": "flot","seriesOverrides": [],"spaceLength": 10,"stack": false,"steppedLine": false,"targets": [{"expr": "elasticsearch_indices_segment_terms_memory_primary{instance=~\"$instance\"}","format": "time_series","intervalFactor": 2,"legendFormat": "{{index}}","refId": "A"}],"thresholds": [],"timeFrom": null,"timeShift": null,"title": "Size of terms with only primary shards on all nodes in bytes","tooltip": {"shared": true,"sort": 2,"value_type": "individual"},"type": "graph","xaxis": {"buckets": null,"mode": "time","name": null,"show": true,"values": []},"yaxes": [{"format": "bytes","label": null,"logBase": 1,"max": null,"min": null,"show": true},{"format": "short","label": null,"logBase": 1,"max": null,"min": null,"show": false}]},{"aliasColors": {},"bars": false,"dashLength": 10,"dashes": false,"datasource": "Prometheus","fill": 1,"gridPos": {"h": 7,"w": 24,"x": 0,"y": 35},"id": 72,"legend": {"alignAsTable": true,"avg": true,"current": false,"max": true,"min": true,"rightSide": true,"show": true,"sort": "avg","sortDesc": true,"total": false,"values": true},"lines": true,"linewidth": 1,"links": [],"nullPointMode": "null","percentage": false,"pointradius": 5,"points": false,"renderer": "flot","seriesOverrides": [],"spaceLength": 10,"stack": false,"steppedLine": false,"targets": [{"expr": "elasticsearch_indices_segment_terms_memory_total{instance=~\"$instance\"}","format": "time_series","intervalFactor": 2,"legendFormat": "{{index}}","refId": "A"}],"thresholds": [],"timeFrom": null,"timeShift": null,"title": "Number of terms with all shards on all nodes in bytes","tooltip": {"shared": true,"sort": 2,"value_type": "individual"},"type": "graph","xaxis": {"buckets": null,"mode": "time","name": null,"show": true,"values": []},"yaxes": [{"format": "bytes","label": null,"logBase": 1,"max": null,"min": null,"show": true},{"format": "short","label": null,"logBase": 1,"max": null,"min": null,"show": false}]}],"repeat": null,"title": "Indices: Terms","type": "row"},{"collapsed": true,"gridPos": {"h": 1,"w": 24,"x": 0,"y": 52},"id": 112,"panels": [{"aliasColors": {},"bars": false,"dashLength": 10,"dashes": false,"datasource": "Prometheus","fill": 1,"gridPos": {"h": 7,"w": 24,"x": 0,"y": 29},"id": 73,"legend": {"alignAsTable": true,"avg": true,"current": false,"hideZero": true,"max": true,"min": true,"rightSide": true,"show": true,"sort": "avg","sortDesc": true,"total": false,"values": true},"lines": true,"linewidth": 1,"links": [],"nullPointMode": "null","percentage": false,"pointradius": 5,"points": false,"renderer": "flot","seriesOverrides": [],"spaceLength": 10,"stack": false,"steppedLine": false,"targets": [{"expr": "elasticsearch_indices_segment_version_map_memory_bytes_primary{instance=~\"$instance\"}","format": "time_series","intervalFactor": 2,"legendFormat": "{{index}}","refId": "A"}],"thresholds": [],"timeFrom": null,"timeShift": null,"title": "Size of version map with only primary shards on all nodes in bytes","tooltip": {"shared": true,"sort": 2,"value_type": "individual"},"type": "graph","xaxis": {"buckets": null,"mode": "time","name": null,"show": true,"values": []},"yaxes": [{"format": "bytes","label": null,"logBase": 1,"max": null,"min": null,"show": true},{"format": "short","label": null,"logBase": 1,"max": null,"min": null,"show": false}]},{"aliasColors": {},"bars": false,"dashLength": 10,"dashes": false,"datasource": "Prometheus","fill": 1,"gridPos": {"h": 7,"w": 24,"x": 0,"y": 36},"id": 74,"legend": {"alignAsTable": true,"avg": true,"current": false,"hideZero": true,"max": true,"min": true,"rightSide": true,"show": true,"sort": "avg","sortDesc": true,"total": false,"values": true},"lines": true,"linewidth": 1,"links": [],"nullPointMode": "null","percentage": false,"pointradius": 5,"points": false,"renderer": "flot","seriesOverrides": [],"spaceLength": 10,"stack": false,"steppedLine": false,"targets": [{"expr": "elasticsearch_indices_segment_version_map_memory_bytes_total{instance=~\"$instance\"}","format": "time_series","intervalFactor": 2,"legendFormat": "{{index}}","refId": "A"}],"thresholds": [],"timeFrom": null,"timeShift": null,"title": "Size of version map with all shards on all nodes in bytes","tooltip": {"shared": true,"sort": 2,"value_type": "individual"},"type": "graph","xaxis": {"buckets": null,"mode": "time","name": null,"show": true,"values": []},"yaxes": [{"format": "bytes","label": null,"logBase": 1,"max": null,"min": null,"show": true},{"format": "short","label": null,"logBase": 1,"max": null,"min": null,"show": false}]}],"repeat": null,"title": "Indices: Version map","type": "row"}],"refresh": false,"schemaVersion": 16,"style": "dark","tags": ["elasticsearch","App"],"templating": {"list": [{"auto": true,"auto_count": 30,"auto_min": "10s","current": {"text": "auto","value": "$__auto_interval_interval"},"hide": 0,"label": "Interval","name": "interval","options": [{"selected": true,"text": "auto","value": "$__auto_interval_interval"},{"selected": false,"text": "5m","value": "5m"},{"selected": false,"text": "10m","value": "10m"},{"selected": false,"text": "30m","value": "30m"},{"selected": false,"text": "1h","value": "1h"},{"selected": false,"text": "6h","value": "6h"},{"selected": false,"text": "12h","value": "12h"},{"selected": false,"text": "1d","value": "1d"},{"selected": false,"text": "7d","value": "7d"},{"selected": false,"text": "14d","value": "14d"},{"selected": false,"text": "30d","value": "30d"}],"query": "5m,10m,30m,1h,6h,12h,1d,7d,14d,30d","refresh": 2,"type": "interval"},{"allValue": null,"current": {},"datasource": "${DS_PROMETHEUS}","hide": 0,"includeAll": false,"label": "Сluster","multi": false,"name": "cluster","options": [],"query": "label_values(elasticsearch_indices_docs,cluster)","refresh": 1,"regex": "","sort": 1,"tagValuesQuery": null,"tags": [],"tagsQuery": null,"type": "query","useTags": false},{"allValue": null,"current": {},"datasource": "${DS_PROMETHEUS}","hide": 0,"includeAll": true,"label": "Node name","multi": true,"name": "name","options": [],"query": "label_values(elasticsearch_indices_docs{cluster=\"$cluster\", name!=\"\"},name)","refresh": 1,"regex": "","sort": 1,"tagValuesQuery": null,"tags": [],"tagsQuery": null,"type": "query","useTags": false},{"allValue": null,"current": {},"datasource": "${DS_PROMETHEUS}","hide": 0,"includeAll": false,"label": "Source of metrics","multi": false,"name": "instance","options": [],"query": "label_values(elasticsearch_indices_docs{cluster=\"$cluster\", name!=\"\"},instance)","refresh": 1,"regex": "","sort": 1,"tagValuesQuery": null,"tags": [],"tagsQuery": null,"type": "query","useTags": false}]},"time": {"from": "now-1h","to": "now"},"timepicker": {"refresh_intervals": ["5s","10s","30s","1m","5m","15m","30m","1h","2h","1d"],"time_options": ["5m","15m","1h","6h","12h","24h","2d","7d","30d"]},"timezone": "browser","title": "ElasticSearch_Exporter-2","uid": "pp7R0LPiz","version": 8
}

　

最后監控面板如下：

?

altermanager告警策略

  - name: kafkaStatsAlertrules:- alert: Kafka_Cluster_Healthexpr: kafka_brokers < 3for: 1mlabels:severity: criticalannotations:summary: "Kafka running on less than 3 nodes"description: "There are only {{$value}} < 3 Kafka nodes running"- name: kafkaTagAlertrules:- alert: Kafka_Cluster_Tagexpr: sum(kafka_consumergroup_lag{}) by(consumergroup,topic)  > 500000#expr: sum(kafka_consumergroup_lag{consumergroup="appLogs",topic="appLogs"}) > 500000for: 1mlabels:severity: criticalannotations:summary: "Kafka group:{{ $labels.consumergroup }} ,topic:{{ $labels.topic }},消費延遲:{{ $value }}"description: "Kafka group:{{ $labels.consumergroup }} ,topic:{{ $labels.topic }} Tag > 500000"

　　

?

轉載于:https://www.cnblogs.com/xuliang666/p/11211920.html

總結

以上是生活随笔為你收集整理的elasticsearch 监控的全部內容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網站內容還不錯，歡迎將生活随笔推薦給好友。