Grafana loki

1. Intro

最近在了解日志聚合系統(tǒng)，正好前幾天看到一篇文章《用了日志系統(tǒng)新貴Loki，ELK突然不香了！》，所以就決定動(dòng)手體驗(yàn)一下。本文就帶大家快速了解下Loki，并簡(jiǎn)單介紹.NET Core如何集成Loki。

2. What's Loki Stack

Grafana Loki like Prometheus, but for logs。其是一個(gè)水平可擴(kuò)展，高可用性，多租戶的日志聚合系統(tǒng)，基于Apatch 2.0開源。其有三部分組成：

Loki 是主服務(wù)器，負(fù)責(zé)存儲(chǔ)日志和處理查詢。對(duì)標(biāo)ELK中的ElasticSearch。

Promtail 是代理，負(fù)責(zé)收集日志并將其發(fā)送給loki。對(duì)標(biāo)ELK中的Logstash。

Grafana提供用戶界面。對(duì)標(biāo)ELK中的Kibana。

3. Why Use Loki

日志聚合系統(tǒng)的目的是為了方便我們進(jìn)行日志跟蹤和故障排查，尤其在云原生的環(huán)境之下。目前主流的日志聚合系統(tǒng)，當(dāng)數(shù)ELK、EFK和Loki。Loki相較于ELK Stack有以下優(yōu)勢(shì)：

• Elasticsearch中的數(shù)據(jù)作為非結(jié)構(gòu)化JSON對(duì)象存儲(chǔ)在磁盤上，Loki以二進(jìn)制的形式存儲(chǔ)。

• Elasticsearch采用全文索引，倒排索引的切分和共享的成本較高。Loki僅索引元數(shù)據(jù)，比如標(biāo)簽。

• 和Prometheus無縫集成。

4. How Use Loki

首先我們先來基于Heml安裝Loki到本地K8S集群。

1. 添加Loki Chart 倉庫：

PS C:\Users\Shengjie> helm repo add loki https://grafana.github.io/loki/charts "loki" has been added to your repositories PS C:\Users\Shengjie> helm repo update Hang tight while we grab the latest from your chart repositories... ...Successfully got an update from the "stable" chart repository ...Successfully got an update from the "loki" chart repository Update Complete. ? Happy Helming!?

2. 安裝Loki Stack

PS C:\Users\Shengjie> helm search hub loki-stack URL CHART VERSION APP VERSION DESCRIPTION https://hub.helm.sh/charts/loki/loki-stack 0.38.3 v1.5.0 Loki: like Prometheus, but for logs. PS C:\Users\Shengjie> helm show values loki/loki-stack loki:enabled: truepromtail:enabled: truefluent-bit:enabled: falsegrafana:enabled: falsesidecar:datasources:enabled: trueimage:tag: 6.7.0prometheus:enabled: falsefilebeat:enabled: falsefilebeatConfig:filebeat.yml: |# logging.level: debugfilebeat.inputs:- type: containerpaths:- /var/log/containers/*.logprocessors:- add_kubernetes_metadata:host: ${NODE_NAME}matchers:- logs_path:logs_path: "/var/log/containers/"output.logstash:hosts: ["logstash-loki:5044"]logstash:enabled: falseimage:repository: grafana/logstash-output-lokitag: 1.0.1filters:main: |-filter {if [kubernetes] {mutate {add_field => {"container_name" => "%{[kubernetes][container][name]}""namespace" => "%{[kubernetes][namespace]}""pod" => "%{[kubernetes][pod][name]}"}replace => { "host" => "%{[kubernetes][node][name]}"}}}mutate {remove_field => ["tags"]}}outputs:main: |-output {loki {url => "http://loki:3100/loki/api/v1/push"#username => "test"#password => "test"}# stdout { codec => rubydebug }}

從上面的Values中，可以看出，可以自定義啟用fluent-bit、grafana、filebeat、prometheus、logstash組件。這里我們僅啟用grafana日志界面。

PS?C:\Users\Shengjie>?helm?install?loki-stack?loki/loki-stack?--set?grafana.enabled=true coalesce.go:165: warning: skipped value for filters: Not a table. coalesce.go:165: warning: skipped value for filters: Not a table. NAME: loki-stack LAST DEPLOYED: Sun Jul 26 11:58:11 2020 NAMESPACE: default STATUS: deployed REVISION: 1 NOTES: The Loki stack has been deployed to your cluster. Loki can now be added as a datasource in Grafana.See http://docs.grafana.org/features/datasources/loki/ for more detail.

3. 登錄Grafana

首先確認(rèn)Loki是否成功部署：

PS C:\Users\Shengjie> helm list NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION loki-stack default 1 2020-07-26 11:58:11.022896 +0800 CST deployed loki-stack-0.38.3 v1.5.0 PS C:\Users\Shengjie> kubectl get pod -w NAME READY STATUS RESTARTS AGE loki-stack-0 1/1 Running 0 2m33s loki-stack-grafana-c447cfbd-z6tbg 1/1 Running 0 2m33s loki-stack-promtail-j47hl 1/1 Running 0 2m33s PS C:\Users\Shengjie> kubectl get svc -w NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 56d loki-stack ClusterIP 10.110.83.209 <none> 3100/TCP 2m52s loki-stack-grafana ClusterIP 10.111.24.26 <none> 80/TCP 2m52s loki-stack-headless ClusterIP None <none> 3100/TCP 2m52s

從上面可知，已經(jīng)成功啟動(dòng)，其中l(wèi)oki暴露的容器端口為3100，grafana暴露的端口為80。因此我們下一步需要進(jìn)行端口轉(zhuǎn)發(fā)，才能訪問grafana。

PS C:\Users\Shengjie> kubectl port-forward svc/loki-stack-grafana 3000:80 Forwarding from 127.0.0.1:3000 -> 3000 Forwarding from [::1]:3000 -> 3000

然后本地瀏覽器打開http://localhost:3000/就可以訪問了。其默認(rèn)用戶是admin，默認(rèn)密碼在哪里呢，別慌，我們?nèi)タ聪耹oki-stack chart 的readme：

PS C:\Users\Shengjie> helm show readme loki/loki-stack # Loki-Stack Helm Chart ## PrerequisitesMake sure you have Helm [installed](https://helm.sh/docs/using_helm/#installing-helm) and [deployed](https://helm.sh/docs/using_helm/#installing-tiller) to your cluster. Then add Loki's chart repository to Helm: $ helm repo add loki https://grafana.github.io/loki/chartsYou can update the chart repository by running: $ helm repo update## Deploy Loki and Promtail to your cluster ### Deploy with default config $ helm upgrade --install loki loki/loki-stack ### Deploy in a custom namespace $ helm upgrade --install loki --namespace=loki-stack loki/loki-stack### Deploy with custom config$ helm upgrade --install loki loki/loki-stack --set "key1=val1,key2=val2,..." ## Deploy Loki and Fluent Bit to your cluster $ helm upgrade --install loki loki/loki-stack \--set fluent-bit.enabled=true,promtail.enabled=false ## Deploy Grafana to your cluster The chart loki-stack contains a pre-configured Grafana, simply use `--set grafana.enabled=true`To get the admin password for the Grafana pod, run the following command: $ kubectl get secret --namespace <YOUR-NAMESPACE> loki-grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo To access the Grafana UI, run the following command: $ kubectl port-forward --namespace <YOUR-NAMESPACE> service/loki-grafana 3000:80 Navigate to http://localhost:3000 and login with `admin` and the password output above. Then follow the [instructions for adding the loki datasource](/docs/getting-started/grafana.md), using the URL `http://loki:3100/`

上面已經(jīng)說了很清楚了，可以從secret中獲取。

如果在windows powersheel中執(zhí)行，需要分兩步： 1. 先獲取base64加密的密碼 PS C:\Users\Shengjie> $pwd= kubectl get secret --namespace default loki-stack-grafana -o jsonpath="{.data.admin-password}" 2. decode base64 PS C:\Users\Shengjie> [Text.Encoding]::Utf8.GetString([Convert]::FromBase64String($pwd)) CjnbkkQmwQynZ96gCsynSf0elYQLOp4dyuDnp9jJ------------------------ Linux命令行執(zhí)行： shengjie@Thinkpad:/mnt/c/Users/Shengjie$ kubectl get secret --namespace default loki-stack-grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo CjnbkkQmwQynZ96gCsynSf0elYQLOp4dyuDnp9jJ

然后使用admin/CjnbkkQmwQynZ96gCsynSf0elYQLOp4dyuDnp9jJ即可成功登錄http://localhost:3000/。

Grafnan Explore

5. Use Loki With .NET Core

下一步，我們就來創(chuàng)建一個(gè)ASP.NET Core Web 應(yīng)用，將日志記錄到Loki，并通過Grafana進(jìn)行聚合分析。

PS C:\Users\Shengjie> dotnet new web -n Loki.K8s.Demo The template "ASP.NET Core Empty" was created successfully.Processing post-creation actions... Running 'dotnet restore' on Loki.K8s.Demo\Loki.K8s.Demo.csproj...Determining projects to restore...Restored C:\Users\Shengjie\Loki.K8s.Demo\Loki.K8s.Demo.csproj (in 150 ms).Restore succeeded. PS C:\Users\Shengjie> cd .\Loki.K8s.Demo\ # 添加Serilog.AspNetCore和Serilog.Sinks.Loki Nuget包。 PS C:\Users\Shengjie\Loki.K8s.Demo> dotnet add package Serilog.AspNetCore PS C:\Users\Shengjie\Loki.K8s.Demo> dotnet add package Serilog.Sinks.Loki

從上可知，日志組件選用的是Serilog，因?yàn)槠渲С殖志没罩镜絃oki。修改Program.cs如下：

public static IHostBuilder CreateHostBuilder(string[] args) =>Host.CreateDefaultBuilder(args).ConfigureWebHostDefaults(webBuilder =>{webBuilder.UseStartup<Startup>();}).UseSerilog((ctx, cfg) =>{//cfg.MinimumLevel.Override("Microsoft", LogEventLevel.Warning);//Microsoft框架本身的日志，僅輸出Warning以上級(jí)別cfg.Enrich.FromLogContext().Enrich.WithProperty("App", ctx.HostingEnvironment.ApplicationName).Enrich.WithProperty("ENV", ctx.HostingEnvironment.EnvironmentName).WriteTo.LokiHttp(new NoAuthCredentials("http://localhost:3100"))//配置Loki Url和認(rèn)證方式.WriteTo.Console();});

修改Startup.cs的Configure方法如下：

public void Configure(IApplicationBuilder app, IWebHostEnvironment env,ILogger<Startup> logger) {if (env.IsDevelopment()){app.UseDeveloperExceptionPage();}app.UseRouting();app.UseEndpoints(endpoints =>{endpoints.MapGet("/", async context =>{logger.LogInformation("start handle request!");await context.Response.WriteAsync("Hello World!");logger.LogInformation("end handle request!");});}); }

因?yàn)椴幌氚褢?yīng)用打包成鏡像運(yùn)行到K8S中，所以我們需要把K8S的Loki服務(wù)做一次端口轉(zhuǎn)發(fā)暴露到本機(jī)，轉(zhuǎn)發(fā)后，就可以使用http://localhost:3100作為L(zhǎng)oki的Url進(jìn)行日志寫入啦。

PS C:\Users\Shengjie\Loki.K8s.Demo> kubectl port-forward svc/loki-stack 3100:3100 Forwarding from 127.0.0.1:3100 -> 3100 Forwarding from [::1]:3100 -> 3100

運(yùn)行項(xiàng)目后，重新打開Grafana，添加過濾條件，就可以查看應(yīng)用日志了。

參考資料：

日志聚合工具loki

CSDN-linkt1234-Loki

Loki官方文檔

總結(jié)

以上是生活随笔為你收集整理的.NET Core + K8S + Loki 玩转日志聚合的全部?jī)?nèi)容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網(wǎng)站內(nèi)容還不錯(cuò)，歡迎將生活随笔推薦給好友。