在前文介紹的k8s master高可用實(shí)踐方案中，需要對(duì)kube-apiserver的證書(shū)進(jìn)行更新，加入VIP和從節(jié)點(diǎn)的IP，然后重新下發(fā)證書(shū)?；仡橩8S集群整個(gè)搭建過(guò)程中，最容易讓人懵圈的也就是配置證書(shū)環(huán)節(jié)，因此本文對(duì)K8S集群所用到的證書(shū)進(jìn)行梳理一下。

一、根證書(shū)

ca.pem 根證書(shū)公鑰文件
ca-key.pem 根證書(shū)私鑰文件
ca.csr 證書(shū)簽名請(qǐng)求，用于交叉簽名或重新簽名
ca-config.json 使用cfssl工具生成其他類型證書(shū)需要引用的配置文件
ca.pem用于簽發(fā)后續(xù)其他的證書(shū)文件，因此ca.pem文件需要分發(fā)到集群中的每臺(tái)服務(wù)器上去。

證書(shū)生成命令，默認(rèn)生成的證書(shū)有效期5年

# echo '{"CN":"CA","key":{"algo":"rsa","size":2048}}' | cfssl gencert -initca - | cfssljson -bare ca - # echo '{"signing":{"default":{"expiry":"43800h","usages":["signing","key encipherment","server auth","client auth"]}}}' > ca-config.json

二、flannel證書(shū)

證書(shū)生成命令，默認(rèn)生成的證書(shū)有效期5年

# cfssl gencert -ca=/etc/ssl/etcd/ca.pem \-ca-key=/etc/ssl/etcd/ca-key.pem \ -config=/etc/ssl/etcd/ca-config.json \-profile=kubernetes flanneld-csr.json | cfssljson -bare flanneld

這里生成證書(shū)需要flanneld-csr.json文件

# cat flanneld-csr.json {"CN": "flanneld","hosts": [],"key": {"algo": "rsa","size": 2048},"names": [{"C": "CN","ST": "FuZhou","L": "FuZhou","O": "k8s","OU": "System"}] }

flannel啟動(dòng)文件配置

# cat /usr/lib/systemd/system/flanneld.service [Unit] Description=Flanneld overlay address etcd agent After=network.target After=network-online.target Wants=network-online.target After=etcd.service Before=docker.service[Service] Type=notify ExecStart=/usr/local/bin/flanneld \-etcd-cafile=/etc/ssl/etcd/ca.pem \-etcd-certfile=/etc/ssl/flanneld/flanneld.pem \-etcd-keyfile=/etc/ssl/flanneld/flanneld-key.pem \-etcd-endpoints=https://192.168.115.5,https://192.168.115.6:2379,https://192.168.115.7:2379 \-etcd-prefix=/kubernetes/network ExecStartPost=/usr/local/bin/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/docker Restart=on-failure[Install] WantedBy=multi-user.target RequiredBy=docker.service

三、etcd證書(shū)

1、服務(wù)端證書(shū)

server.pem etcd服務(wù)端證書(shū)公鑰文件
server-key.pem etcd服務(wù)端證書(shū)私鑰文件
server.csr 證書(shū)簽名請(qǐng)求

證書(shū)生成命令，默認(rèn)生成的證書(shū)有效期5年

# export ADDRESS=192.168.115.5,192.168.115.6,192.168.115.7,vm1,vm2,vm3 # export NAME=server # echo '{"CN":"'$NAME'","hosts":[""],"key":{"algo":"rsa","size":2048}}' | \ cfssl gencert -config=ca-config.json -ca=ca.pem -ca-key=ca-key.pem \ -hostname="$ADDRESS" - | cfssljson -bare $NAME

server.pem、server-key.pem文件用來(lái)etcd集群間通信加解密，因此所有的etcd服務(wù)器都需要有這兩個(gè)文件

# tail -15 /usr/lib/systemd/system/etcd.service --initial-cluster-token=etcd-cluster-token \ --initial-cluster-state=new \ --cert-file=/etc/ssl/etcd/server.pem \ --key-file=/etc/ssl/etcd/server-key.pem \ --peer-cert-file=/etc/ssl/etcd/server.pem \ --peer-key-file=/etc/ssl/etcd/server-key.pem \ --trusted-ca-file=/etc/ssl/etcd/ca.pem \ --peer-trusted-ca-file=/etc/ssl/etcd/ca.pem \ --peer-client-cert-auth=true \ --client-cert-auth=true" Restart=on-failure LimitNOFILE=65536[Install] WantedBy=multi-user.target

2、客戶端證書(shū)

client.pem etcd客戶端證書(shū)公鑰文件
client-key.pem etcd客戶端證書(shū)私鑰文件
client.csr 證書(shū)簽名請(qǐng)求

證書(shū)生成命令，默認(rèn)生成的證書(shū)有效期5年

# export ADDRESS= # export NAME=client # echo '{"CN":"'$NAME'","hosts":[""],"key":{"algo":"rsa","size":2048}}' | \cfssl gencert -config=ca-config.json -ca=ca.pem -ca-key=ca-key.pem \ -hostname="$ADDRESS" - | cfssljson -bare $NAME

client.pem、client-key.pem文件給etcdctl客戶端用來(lái)和etcd服務(wù)器進(jìn)行通信，可根據(jù)實(shí)際需要進(jìn)行配置。

# export ETCDCTL_API=3 # etcdctl --write-out=table \ --cert=/etc/ssl/etcd/client.pem \ --key=/etc/ssl/etcd/client-key.pem \ --cacert=/etc/ssl/etcd/ca.pem \ --endpoints=https://192.168.115.5:2379,https://192.168.115.6:2379,https://192.168.115.7:2379 \ member list

四、Master節(jié)點(diǎn)證書(shū)

Kube-apiserver證書(shū)
Kubernetes.pem kube-apiserver證書(shū)公鑰文件
Kubernetes-key.pem kube-apiserver證書(shū)私鑰文件
kuberentes.csr kube-apiserver證書(shū)簽名請(qǐng)求

證書(shū)生成命令，默認(rèn)生成的證書(shū)有效期5年

# cfssl gencert -ca=/etc/ssl/etcd/ca.pem \-ca-key=/etc/ssl/etcd/ca-key.pem \-config=/etc/ssl/etcd/ca-config.json \-profile=kubernetes k8s-csr.json | cfssljson -bare kubernetes

這里生成證書(shū)需要k8s-csr.json文件，其中定義了master節(jié)點(diǎn)的IP列表等信息

# cat k8s-csr.json {"CN": "kubernetes","hosts": ["127.0.0.1","192.168.115.4","192.168.115.5","192.168.115.6","192.168.115.7","10.254.0.1","kubernetes","kubernetes.default","kubernetes.default.svc","kubernetes.default.svc.cluster","kubernetes.default.svc.cluster.local"],"key": {"algo": "rsa","size": 2048},"names": [{"C": "CN","ST": "FuZhou","L": "FuZhou","O": "k8s","OU": "System"}] }

五、Node節(jié)點(diǎn)證書(shū)

1、kube-proxy證書(shū)

Kube-proxy.pem kube-proxy證書(shū)公鑰文件
Kube-proxy-key.pem kube-proxy證書(shū)私鑰文件
Kube-proxy.csr kube-proxy證書(shū)簽名請(qǐng)求

證書(shū)生成命令，默認(rèn)生成的證書(shū)有效期5年

# cfssl gencert -ca=/etc/ssl/etcd/ca.pem \-ca-key=/etc/ssl/etcd/ca-key.pem \-config=/etc/ssl/etcd/ca-config.json \-profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy

這里生成證書(shū)需要kube-proxy-csr.json文件

# cat kube-proxy-csr.json {"CN": "system:kube-proxy","hosts": [],"key": {"algo": "rsa","size": 2048},"names": [{"C": "CN","ST": "FuZhou","L": "FuZhou","O": "k8s","OU": "System"}] }

2、Kubelet證書(shū)

Kubelet-client.crt: kubectl客戶端證書(shū)公鑰文件
Kubelet-client.key: kubectl客戶端私鑰文件
Kubelet.crt：kubelet服務(wù)端證書(shū)公鑰文件
Kubelet.key：kubelet服務(wù)端證書(shū)私鑰文件

> kubelet-client.crt 文件在 kubelet 完成 TLS bootstrapping 后生成，有效期為 1 年。此證書(shū)是由 controller manager 簽署的，此后 kubelet 將會(huì)加載該證書(shū)，用于與 apiserver 建立 TLS 通訊，同時(shí)使用該證書(shū)的 CN 字段作為用戶名，O 字段作為用戶組向 apiserver 發(fā)起其他請(qǐng)求。
kubelet.crt 文件在 kubelet 完成 TLS bootstrapping 后且沒(méi)有配置 --feature-gates=RotateKubeletServerCertificate=true 時(shí)才會(huì)生成；該文件為一個(gè)獨(dú)立于 apiserver CA 的自簽 CA 證書(shū)，有效期為 1 年；被用作 kubelet 10250 api 端口

關(guān)于kubelet首次啟動(dòng) TLS bootstrapping的介紹（先有雞還是先有蛋問(wèn)題的解決方案）可參考文檔，https://mritd.me/2018/01/07/kubernetes-tls-bootstrapping-note/

六、配置證書(shū)自動(dòng)續(xù)期

默認(rèn)簽署kubectl客戶端和kubelet服務(wù)端證書(shū)只有 1 年有效期，如果想要調(diào)整證書(shū)有效期可以通過(guò)設(shè)置 kube-controller-manager 的?--experimental-cluster-signing-duration?參數(shù)實(shí)現(xiàn)，該參數(shù)默認(rèn)值為?8760h0m0s。下面我們來(lái)介紹一下如何實(shí)現(xiàn)證書(shū)到期的自動(dòng)續(xù)簽。這個(gè)問(wèn)題如果處理不當(dāng)，證書(shū)過(guò)期之后會(huì)出現(xiàn)所有的node節(jié)點(diǎn)連接不上的情況。

1、kcm服務(wù)，這里為了方便測(cè)試，過(guò)期時(shí)間修改為30分鐘

# egrep 'feature|experimental' /usr/lib/systemd/system/kube-controller-manager.service --feature-gates=RotateKubeletClientCertificate=true,RotateKubeletServerCertificate=true \--experimental-cluster-signing-duration=30m0s \ # systemctl daemon-reload # systemctl restart kube-controller-manager

2、kubelet服務(wù)
配置完成刪掉Kubelet-client.crt、Kubelet-client.key、Kubelet.crt、Kubelet.key四個(gè)文件后重啟kubelet服務(wù)。

# egrep 'feature|rotate' /usr/lib/systemd/system/kubelet.service --feature-gates=RotateKubeletClientCertificate=true,RotateKubeletServerCertificate=true \--rotate-certificates=true \ # systemctl daemon-reload # systemctl restart kubelet

3、手工簽發(fā)證書(shū)

# kubectl create clusterrolebinding kubelet-clinet \ --clusterrole=system:node \ --user=system:anonymous

如果缺少對(duì)system:anonymous用戶的授權(quán)，kubelet啟動(dòng)的時(shí)候會(huì)報(bào)錯(cuò)如下：
error: failed to run Kubelet: cannot create certificate signing request: certificatesigningrequests.certificates.k8s.io is forbidden: User "system:anonymous" cannot create certificatesigningrequests.certificates.k8s.io at the cluster scope

# kubectl get csr # kubectl certificate approve csr-fn946 # kubectl certificate approve csr-kwvg9

node節(jié)點(diǎn)將會(huì)重新生成kubectl客戶端和kubelet服務(wù)端證書(shū)

4、配置自動(dòng)簽發(fā)證書(shū)，在大規(guī)模集群下，這個(gè)配置是必須的

# cat rbac.yaml apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata:annotations:rbac.authorization.kubernetes.io/autoupdate: "true"labels:kubernetes.io/bootstrapping: rbac-defaultsname: system:certificates.k8s.io:certificatesigningrequests:selfnodeserver rules: - apiGroups:- certificates.k8s.ioresources:- certificatesigningrequests/selfnodeserververbs:- create --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata:name: kubeadm:node-autoapprove-certificate-server roleRef:apiGroup: rbac.authorization.k8s.iokind: ClusterRolename: system:certificates.k8s.io:certificatesigningrequests:selfnodeserver subjects: - apiGroup: rbac.authorization.k8s.iokind: Groupname: system:nodes# kubectl create -f rbac.yaml # kubectl create clusterrolebinding node-client-auto-approve-csr \ --clusterrole=system:certificates.k8s.io:certificatesigningrequests:nodeclient \ --group=system:bootstrappers # kubectl create clusterrolebinding node-client-auto-renew-crt \ --clusterrole=system:certificates.k8s.io:certificatesigningrequests:selfnodeclient \ --group=system:nodes # kubectl create clusterrolebinding node-server-auto-renew-crt \ --clusterrole=system:certificates.k8s.io:certificatesigningrequests:selfnodeserver \ --group=system:nodes

這里需要注意的是刪除掉Kubelet-client.crt、Kubelet-client.key兩個(gè)文件之后，啟動(dòng)kubelet服務(wù)，首先會(huì)生成一個(gè)Kubelet-client.key文件，我們需要對(duì)這個(gè)證書(shū)的crs請(qǐng)求進(jìn)行approve，否則node節(jié)點(diǎn)無(wú)法正常啟動(dòng)。
其次，如果kubelet.kubeconfig文件中配置的client-certificate、client-key目錄位置和kubelet的啟動(dòng)參數(shù)--cert-dir不一致，則kubelet.kubeconfig文件中的配置文件會(huì)被自動(dòng)更新。

總結(jié)

以上是生活随笔為你收集整理的K8S集群tls证书管理的全部?jī)?nèi)容，希望文章能夠幫你解決所遇到的問(wèn)題。

如果覺(jué)得生活随笔網(wǎng)站內(nèi)容還不錯(cuò)，歡迎將生活随笔推薦給好友。