作者：老Z，中電信數(shù)智科技有限公司山東分公司運(yùn)維架構(gòu)師，云原生愛好者，目前專注于云原生運(yùn)維，云原生領(lǐng)域技術(shù)棧涉及Kubernetes、KubeSphere、DevOps、OpenStack、Ansible等。

KubeKey 是一個(gè)用于部署 K8s 集群的開源輕量級(jí)工具。

它提供了一種靈活、快速、便捷的方式來(lái)僅安裝 Kubernetes/K3s，或同時(shí)安裝 K8s/K3s 和 KubeSphere，以及其他云原生插件。除此之外，它也是擴(kuò)展和升級(jí)集群的有效工具。

KubeKey v2.1.0 版本新增了清單 (manifest) 和制品 (artifact) 的概念，為用戶離線部署 K8s 集群提供了一種解決方案。

manifest 是一個(gè)描述當(dāng)前 K8s 集群信息和定義 artifact 制品中需要包含哪些內(nèi)容的文本文件。

在過去，用戶需要準(zhǔn)備部署工具，鏡像 tar 包和其他相關(guān)的二進(jìn)制文件，每位用戶需要部署的 K8s 版本和需要部署的鏡像都是不同的。現(xiàn)在使用 KubeKey，用戶只需使用清單 manifest 文件來(lái)定義將要離線部署的集群環(huán)境需要的內(nèi)容，再通過該 manifest 來(lái)導(dǎo)出制品 artifact 文件即可完成準(zhǔn)備工作。離線部署時(shí)只需要 KubeKey 和 artifact 就可快速、簡(jiǎn)單的在環(huán)境中部署鏡像倉(cāng)庫(kù)和 K8s 集群。

KubeKey 生成 manifest 文件有兩種方式。

• 利用現(xiàn)有運(yùn)行中的集群作為源生成 manifest 文件，也是官方推薦的一種方式，具體參考 KubeSphere 官網(wǎng)的離線部署文檔。
• 根據(jù) 模板文件 手動(dòng)編寫 manifest 文件。

第一種方式的好處是可以構(gòu)建 1:1 的運(yùn)行環(huán)境，但是需要提前部署一個(gè)集群，不夠靈活度，并不是所有人都具備這種條件的。

因此，本文參考官方的離線文檔，采用手寫 manifest 文件的方式，實(shí)現(xiàn)離線環(huán)境的安裝部署。

本文知識(shí)點(diǎn)

• 定級(jí)：入門級(jí)
• 了解清單 (manifest) 和制品 (artifact) 的概念
• 掌握 manifest 清單的編寫方法
• 根據(jù) manifest 清單制作 artifact
• 離線部署 KubeSphere 和 Kubernetes

演示服務(wù)器配置

主機(jī)名IPCPU內(nèi)存系統(tǒng)盤數(shù)據(jù)盤用途

zdeops-master	192.168.9.9	2	4	40	200	Ansible 運(yùn)維控制節(jié)點(diǎn)
ks-k8s-master-0	192.168.9.91	4	16	40	200+200	KubeSphere/k8s-master/k8s-worker/Ceph
ks-k8s-master-1	192.168.9.92	4	16	40	200+200	KubeSphere/k8s-master/k8s-worker/Ceph
ks-k8s-master-2	192.168.9.93	4	16	40	200+200	KubeSphere/k8s-master/k8s-worker/Ceph
es-node-0	192.168.9.95	2	8	40	200	ElasticSearch
es-node-1	192.168.9.96	2	8	40	200	ElasticSearch
es-node-2	192.168.9.97	2	8	40	200	ElasticSearch
harbor	192.168.9.89	2	8	40	200	Harbor
合計(jì)	8	22	84	320	2200

演示環(huán)境涉及軟件版本信息

• 操作系統(tǒng)：CentOS-7.9-x86_64

• KubeSphere：3.3.0

• Kubernetes：1.24.1

• Kubekey：v2.2.1

• Ansible：2.8.20

• Harbor：2.5.1

離線部署資源制作

下載 KubeKey

# 在zdevops-master 運(yùn)維開發(fā)服務(wù)器執(zhí)行# 選擇中文區(qū)下載(訪問github受限時(shí)使用) $ export KKZONE=cn# 下載KubeKey $ mkdir /data/kubekey $ cd /data/kubekey/ $ curl -sfL https://get-kk.kubesphere.io | VERSION=v2.2.1 sh -

獲取 manifest 模板

參考 https://github.com/kubesphere/kubekey/blob/master/docs/manifest-example.md

有兩個(gè)參考用例，一個(gè)簡(jiǎn)單版，一個(gè)完整版。參考簡(jiǎn)單版就可以。

獲取 ks-installer images-list

$ wget https://github.com/kubesphere/ks-installer/releases/download/v3.3.0/images-list.txt

文中的 image 列表選用的 Docker Hub 倉(cāng)庫(kù)其他組件存放的公共倉(cāng)庫(kù)，國(guó)內(nèi)建議統(tǒng)一更改前綴為 registry.cn-beijing.aliyuncs.com/kubesphereio

修改后的完整的鏡像列表在下面的 manifest 文件中展示。

請(qǐng)注意，example-images 包含的 image 中只保留了 busybox，其他的在本文中沒有使用。

獲取操作系統(tǒng)依賴包

$ wget https://github.com/kubesphere/kubekey/releases/download/v2.2.1/centos7-rpms-amd64.iso

將該 ISO 文件放到制作離線鏡像的服務(wù)器的 /data/kubekey 目錄下

生成 manifest 文件

根據(jù)上面的文件及相關(guān)信息，生成最終 manifest.yaml。

命名為 ks-v3.3.0-manifest.yaml

apiVersion: kubekey.kubesphere.io/v1alpha2 kind: Manifest metadata:name: sample spec:arches:- amd64operatingSystems:- arch: amd64type: linuxid: centosversion: "7"osImage: CentOS Linux 7 (Core)repository:iso:localPath: "/data/kubekey/centos7-rpms-amd64.iso"url:kubernetesDistributions:- type: kubernetesversion: v1.24.1components:helm: version: v3.6.3cni: version: v0.9.1etcd: version: v3.4.13containerRuntimes:- type: containerdversion: 1.6.4crictl: version: v1.24.0docker-registry:version: "2"harbor:version: v2.4.1docker-compose:version: v2.2.2images:- registry.cn-beijing.aliyuncs.com/kubesphereio/kube-apiserver:v1.23.7- registry.cn-beijing.aliyuncs.com/kubesphereio/kube-controller-manager:v1.23.7- registry.cn-beijing.aliyuncs.com/kubesphereio/kube-proxy:v1.23.7- registry.cn-beijing.aliyuncs.com/kubesphereio/kube-scheduler:v1.23.7- registry.cn-beijing.aliyuncs.com/kubesphereio/kube-apiserver:v1.24.1- registry.cn-beijing.aliyuncs.com/kubesphereio/kube-controller-manager:v1.24.1- registry.cn-beijing.aliyuncs.com/kubesphereio/kube-proxy:v1.24.1- registry.cn-beijing.aliyuncs.com/kubesphereio/kube-scheduler:v1.24.1- registry.cn-beijing.aliyuncs.com/kubesphereio/kube-apiserver:v1.22.10- registry.cn-beijing.aliyuncs.com/kubesphereio/kube-controller-manager:v1.22.10- registry.cn-beijing.aliyuncs.com/kubesphereio/kube-proxy:v1.22.10- registry.cn-beijing.aliyuncs.com/kubesphereio/kube-scheduler:v1.22.10- registry.cn-beijing.aliyuncs.com/kubesphereio/kube-apiserver:v1.21.13- registry.cn-beijing.aliyuncs.com/kubesphereio/kube-controller-manager:v1.21.13- registry.cn-beijing.aliyuncs.com/kubesphereio/kube-proxy:v1.21.13- registry.cn-beijing.aliyuncs.com/kubesphereio/kube-scheduler:v1.21.13- registry.cn-beijing.aliyuncs.com/kubesphereio/pause:3.7- registry.cn-beijing.aliyuncs.com/kubesphereio/pause:3.6- registry.cn-beijing.aliyuncs.com/kubesphereio/pause:3.5- registry.cn-beijing.aliyuncs.com/kubesphereio/pause:3.4.1- registry.cn-beijing.aliyuncs.com/kubesphereio/coredns:1.8.0- registry.cn-beijing.aliyuncs.com/kubesphereio/coredns:1.8.6- registry.cn-beijing.aliyuncs.com/kubesphereio/cni:v3.20.0- registry.cn-beijing.aliyuncs.com/kubesphereio/kube-controllers:v3.20.0- registry.cn-beijing.aliyuncs.com/kubesphereio/node:v3.20.0- registry.cn-beijing.aliyuncs.com/kubesphereio/pod2daemon-flexvol:v3.20.0- registry.cn-beijing.aliyuncs.com/kubesphereio/typha:v3.20.0- registry.cn-beijing.aliyuncs.com/kubesphereio/flannel:v0.12.0- registry.cn-beijing.aliyuncs.com/kubesphereio/provisioner-localpv:2.10.1- registry.cn-beijing.aliyuncs.com/kubesphereio/linux-utils:2.10.0- registry.cn-beijing.aliyuncs.com/kubesphereio/haproxy:2.3- registry.cn-beijing.aliyuncs.com/kubesphereio/nfs-subdir-external-provisioner:v4.0.2- registry.cn-beijing.aliyuncs.com/kubesphereio/k8s-dns-node-cache:1.15.12- registry.cn-beijing.aliyuncs.com/kubesphereio/ks-installer:v3.3.0- registry.cn-beijing.aliyuncs.com/kubesphereio/ks-apiserver:v3.3.0- registry.cn-beijing.aliyuncs.com/kubesphereio/ks-console:v3.3.0- registry.cn-beijing.aliyuncs.com/kubesphereio/ks-controller-manager:v3.3.0- registry.cn-beijing.aliyuncs.com/kubesphereio/kubectl:v1.22.0- registry.cn-beijing.aliyuncs.com/kubesphereio/kubectl:v1.21.0- registry.cn-beijing.aliyuncs.com/kubesphereio/kubectl:v1.20.0- registry.cn-beijing.aliyuncs.com/kubesphereio/kubefed:v0.8.1- registry.cn-beijing.aliyuncs.com/kubesphereio/tower:v0.2.0- registry.cn-beijing.aliyuncs.com/kubesphereio/minio:RELEASE.2019-08-07T01-59-21Z- registry.cn-beijing.aliyuncs.com/kubesphereio/mc:RELEASE.2019-08-07T23-14-43Z- registry.cn-beijing.aliyuncs.com/kubesphereio/snapshot-controller:v4.0.0- registry.cn-beijing.aliyuncs.com/kubesphereio/nginx-ingress-controller:v1.1.0- registry.cn-beijing.aliyuncs.com/kubesphereio/defaultbackend-amd64:1.4- registry.cn-beijing.aliyuncs.com/kubesphereio/metrics-server:v0.4.2- registry.cn-beijing.aliyuncs.com/kubesphereio/redis:5.0.14-alpine- registry.cn-beijing.aliyuncs.com/kubesphereio/haproxy:2.0.25-alpine- registry.cn-beijing.aliyuncs.com/kubesphereio/alpine:3.14- registry.cn-beijing.aliyuncs.com/kubesphereio/openldap:1.3.0- registry.cn-beijing.aliyuncs.com/kubesphereio/netshoot:v1.0- registry.cn-beijing.aliyuncs.com/kubesphereio/cloudcore:v1.9.2- registry.cn-beijing.aliyuncs.com/kubesphereio/iptables-manager:v1.9.2- registry.cn-beijing.aliyuncs.com/kubesphereio/edgeservice:v0.2.0- registry.cn-beijing.aliyuncs.com/kubesphereio/gatekeeper:v3.5.2- registry.cn-beijing.aliyuncs.com/kubesphereio/openpitrix-jobs:v3.2.1- registry.cn-beijing.aliyuncs.com/kubesphereio/devops-apiserver:v3.3.0- registry.cn-beijing.aliyuncs.com/kubesphereio/devops-controller:v3.3.0- registry.cn-beijing.aliyuncs.com/kubesphereio/devops-tools:v3.3.0- registry.cn-beijing.aliyuncs.com/kubesphereio/ks-jenkins:v3.3.0-2.319.1- registry.cn-beijing.aliyuncs.com/kubesphereio/inbound-agent:4.10-2- registry.cn-beijing.aliyuncs.com/kubesphereio/builder-base:v3.2.2- registry.cn-beijing.aliyuncs.com/kubesphereio/builder-nodejs:v3.2.0- registry.cn-beijing.aliyuncs.com/kubesphereio/builder-maven:v3.2.0- registry.cn-beijing.aliyuncs.com/kubesphereio/builder-maven:v3.2.1-jdk11- registry.cn-beijing.aliyuncs.com/kubesphereio/builder-python:v3.2.0- registry.cn-beijing.aliyuncs.com/kubesphereio/builder-go:v3.2.0- registry.cn-beijing.aliyuncs.com/kubesphereio/builder-go:v3.2.2-1.16- registry.cn-beijing.aliyuncs.com/kubesphereio/builder-go:v3.2.2-1.17- registry.cn-beijing.aliyuncs.com/kubesphereio/builder-go:v3.2.2-1.18- registry.cn-beijing.aliyuncs.com/kubesphereio/builder-base:v3.2.2-podman- registry.cn-beijing.aliyuncs.com/kubesphereio/builder-nodejs:v3.2.0-podman- registry.cn-beijing.aliyuncs.com/kubesphereio/builder-maven:v3.2.0-podman- registry.cn-beijing.aliyuncs.com/kubesphereio/builder-maven:v3.2.1-jdk11-podman- registry.cn-beijing.aliyuncs.com/kubesphereio/builder-python:v3.2.0-podman- registry.cn-beijing.aliyuncs.com/kubesphereio/builder-go:v3.2.0-podman- registry.cn-beijing.aliyuncs.com/kubesphereio/builder-go:v3.2.2-1.16-podman- registry.cn-beijing.aliyuncs.com/kubesphereio/builder-go:v3.2.2-1.17-podman- registry.cn-beijing.aliyuncs.com/kubesphereio/builder-go:v3.2.2-1.18-podman- registry.cn-beijing.aliyuncs.com/kubesphereio/s2ioperator:v3.2.1- registry.cn-beijing.aliyuncs.com/kubesphereio/s2irun:v3.2.0- registry.cn-beijing.aliyuncs.com/kubesphereio/s2i-binary:v3.2.0- registry.cn-beijing.aliyuncs.com/kubesphereio/tomcat85-java11-centos7:v3.2.0- registry.cn-beijing.aliyuncs.com/kubesphereio/tomcat85-java11-runtime:v3.2.0- registry.cn-beijing.aliyuncs.com/kubesphereio/tomcat85-java8-centos7:v3.2.0- registry.cn-beijing.aliyuncs.com/kubesphereio/tomcat85-java8-runtime:v3.2.0- registry.cn-beijing.aliyuncs.com/kubesphereio/java-11-centos7:v3.2.0- registry.cn-beijing.aliyuncs.com/kubesphereio/java-8-centos7:v3.2.0- registry.cn-beijing.aliyuncs.com/kubesphereio/java-8-runtime:v3.2.0- registry.cn-beijing.aliyuncs.com/kubesphereio/java-11-runtime:v3.2.0- registry.cn-beijing.aliyuncs.com/kubesphereio/nodejs-8-centos7:v3.2.0- registry.cn-beijing.aliyuncs.com/kubesphereio/nodejs-6-centos7:v3.2.0- registry.cn-beijing.aliyuncs.com/kubesphereio/nodejs-4-centos7:v3.2.0- registry.cn-beijing.aliyuncs.com/kubesphereio/python-36-centos7:v3.2.0- registry.cn-beijing.aliyuncs.com/kubesphereio/python-35-centos7:v3.2.0- registry.cn-beijing.aliyuncs.com/kubesphereio/python-34-centos7:v3.2.0- registry.cn-beijing.aliyuncs.com/kubesphereio/python-27-centos7:v3.2.0- registry.cn-beijing.aliyuncs.com/kubesphereio/argocd:v2.3.3- registry.cn-beijing.aliyuncs.com/kubesphereio/argocd-applicationset:v0.4.1- registry.cn-beijing.aliyuncs.com/kubesphereio/dex:v2.30.2- registry.cn-beijing.aliyuncs.com/kubesphereio/redis:6.2.6-alpine- registry.cn-beijing.aliyuncs.com/kubesphereio/configmap-reload:v0.5.0- registry.cn-beijing.aliyuncs.com/kubesphereio/prometheus:v2.34.0- registry.cn-beijing.aliyuncs.com/kubesphereio/prometheus-config-reloader:v0.55.1- registry.cn-beijing.aliyuncs.com/kubesphereio/prometheus-operator:v0.55.1- registry.cn-beijing.aliyuncs.com/kubesphereio/kube-rbac-proxy:v0.11.0- registry.cn-beijing.aliyuncs.com/kubesphereio/kube-state-metrics:v2.3.0- registry.cn-beijing.aliyuncs.com/kubesphereio/node-exporter:v1.3.1- registry.cn-beijing.aliyuncs.com/kubesphereio/alertmanager:v0.23.0- registry.cn-beijing.aliyuncs.com/kubesphereio/thanos:v0.25.2- registry.cn-beijing.aliyuncs.com/kubesphereio/grafana:8.3.3- registry.cn-beijing.aliyuncs.com/kubesphereio/kube-rbac-proxy:v0.8.0- registry.cn-beijing.aliyuncs.com/kubesphereio/notification-manager-operator:v1.4.0- registry.cn-beijing.aliyuncs.com/kubesphereio/notification-manager:v1.4.0- registry.cn-beijing.aliyuncs.com/kubesphereio/notification-tenant-sidecar:v3.2.0- registry.cn-beijing.aliyuncs.com/kubesphereio/elasticsearch-curator:v5.7.6- registry.cn-beijing.aliyuncs.com/kubesphereio/elasticsearch-oss:6.8.22- registry.cn-beijing.aliyuncs.com/kubesphereio/fluentbit-operator:v0.13.0- registry.cn-beijing.aliyuncs.com/kubesphereio/docker:19.03- registry.cn-beijing.aliyuncs.com/kubesphereio/fluent-bit:v1.8.11- registry.cn-beijing.aliyuncs.com/kubesphereio/log-sidecar-injector:1.1- registry.cn-beijing.aliyuncs.com/kubesphereio/filebeat:6.7.0- registry.cn-beijing.aliyuncs.com/kubesphereio/kube-events-operator:v0.4.0- registry.cn-beijing.aliyuncs.com/kubesphereio/kube-events-exporter:v0.4.0- registry.cn-beijing.aliyuncs.com/kubesphereio/kube-events-ruler:v0.4.0- registry.cn-beijing.aliyuncs.com/kubesphereio/kube-auditing-operator:v0.2.0- registry.cn-beijing.aliyuncs.com/kubesphereio/kube-auditing-webhook:v0.2.0- registry.cn-beijing.aliyuncs.com/kubesphereio/pilot:1.11.1- registry.cn-beijing.aliyuncs.com/kubesphereio/proxyv2:1.11.1- registry.cn-beijing.aliyuncs.com/kubesphereio/jaeger-operator:1.27- registry.cn-beijing.aliyuncs.com/kubesphereio/jaeger-agent:1.27- registry.cn-beijing.aliyuncs.com/kubesphereio/jaeger-collector:1.27- registry.cn-beijing.aliyuncs.com/kubesphereio/jaeger-query:1.27- registry.cn-beijing.aliyuncs.com/kubesphereio/jaeger-es-index-cleaner:1.27- registry.cn-beijing.aliyuncs.com/kubesphereio/kiali-operator:v1.38.1- registry.cn-beijing.aliyuncs.com/kubesphereio/kiali:v1.38- registry.cn-beijing.aliyuncs.com/kubesphereio/busybox:1.31.1- registry.cn-beijing.aliyuncs.com/kubesphereio/scope:1.13.0registry:auths: {}

manifest 修改說明

• 開啟 harbor 和 docker-compose 配置項(xiàng)，為后面通過 KubeKey 自建 harbor 倉(cāng)庫(kù)推送鏡像使用。
• 默認(rèn)創(chuàng)建的 manifest 里面的鏡像列表從 docker.io 獲取，替換前綴為 registry.cn-beijing.aliyuncs.com/kubesphereio。
• 若需要導(dǎo)出的 artifact 文件中包含操作系統(tǒng)依賴文件（如：conntarck、chrony 等），可在 operationSystem 元素中的 .repostiory.iso.url 中配置相應(yīng)的 ISO 依賴文件下載地址為 localPath ，填寫提前下載好的 ISO 包在本地的存放路徑，并將 url 配置項(xiàng)置空。
• 您可以訪問 https://github.com/kubesphere/kubekey/releases/tag/v2.2.1 下載 ISO 文件。

導(dǎo)出制品 artifact

$ export KKZONE=cn$ ./kk artifact export -m ks-v3.3.0-manifest.yaml -o kubesphere-v3.3.0-artifact.tar.gz

制品 (artifact) 說明

• 制品（artifact）是一個(gè)根據(jù)指定的 manifest 文件內(nèi)容導(dǎo)出的包含鏡像 tar 包和相關(guān)二進(jìn)制文件的 tgz 包。
• 在 KubeKey 初始化鏡像倉(cāng)庫(kù)、創(chuàng)建集群、添加節(jié)點(diǎn)和升級(jí)集群的命令中均可指定一個(gè) artifact，KubeKey 將自動(dòng)解包該 artifact 并在執(zhí)行命令時(shí)直接使用解包出來(lái)的文件。

導(dǎo)出 Kubekey

$ tar zcvf kubekey-v2.2.1.tar.gz kk kubekey-v2.2.1-linux-amd64.tar.gz

K8s 服務(wù)器初始化配置

本節(jié)執(zhí)行離線環(huán)境 K8s 服務(wù)器初始化配置。

Ansible hosts 配置

[k8s] ks-k8s-master-0 ansible_ssh_host=192.168.9.91 host_name=ks-k8s-master-0 ks-k8s-master-1 ansible_ssh_host=192.168.9.92 host_name=ks-k8s-master-1 ks-k8s-master-2 ansible_ssh_host=192.168.9.93 host_name=ks-k8s-master-2[es] es-node-0 ansible_ssh_host=192.168.9.95 host_name=es-node-0 es-node-1 ansible_ssh_host=192.168.9.96 host_name=es-node-1 es-node-2 ansible_ssh_host=192.168.9.97 host_name=es-node-2harbor ansible_ssh_host=192.168.9.89 host_name=harbor[servers:children] k8s es[servers:vars] ansible_connection=paramiko ansible_ssh_user=root ansible_ssh_pass=F@ywwpTj4bJtYwzpwCqD

檢測(cè)服務(wù)器連通性

# 利用 ansible 檢測(cè)服務(wù)器的連通性$ cd /data/ansible/ansible-zdevops/inventories/dev/ $ source /opt/ansible2.8/bin/activate $ ansible -m ping all

初始化服務(wù)器配置

# 利用 ansible-playbook 初始化服務(wù)器配置$ ansible-playbook ../../playbooks/init-base.yaml -l k8s

掛載數(shù)據(jù)盤

• 掛載第一塊數(shù)據(jù)盤

# 利用 ansible-playbook 初始化主機(jī)數(shù)據(jù)盤 # 注意 -e data_disk_path="/data" 指定掛載目錄, 用于存儲(chǔ) Docker 容器數(shù)據(jù)$ ansible-playbook ../../playbooks/init-disk.yaml -e data_disk_path="/data" -l k8s

• 掛載驗(yàn)證

# 利用 ansible 驗(yàn)證數(shù)據(jù)盤是否格式化并掛載 $ ansible harbor -m shell -a 'df -h'# 利用 ansible 驗(yàn)證數(shù)據(jù)盤是否配置自動(dòng)掛載$ ansible harbor -m shell -a 'tail -1 /etc/fstab'

安裝 K8s 系統(tǒng)依賴包

# 利用 ansible-playbook 安裝 kubernetes 系統(tǒng)依賴包 # ansible-playbook 中設(shè)置了啟用 GlusterFS 存儲(chǔ)的開關(guān)，默認(rèn)開啟,不需要的可以將參數(shù)設(shè)置為 False$ ansible-playbook ../../playbooks/deploy-kubesphere.yaml -e k8s_storage_glusterfs=false -l k8s

離線安裝集群

傳輸離線部署資源到部署節(jié)點(diǎn)

將以下離線部署資源，傳到部署節(jié)點(diǎn) (通常是第一個(gè) master 節(jié)點(diǎn)) 的 /data/kubekey 目錄。

• Kubekey：kubekey-v2.2.1.tar.gz
• 制品 artifact：kubesphere-v3.3.0-artifact.tar.gz

執(zhí)行以下操作，解壓 kubekey。

$ cd /data/kubekey $ tar xvf kubekey-v2.2.1.tar.gz

創(chuàng)建離線集群配置文件

• 創(chuàng)建配置文件

$ ./kk create config --with-kubesphere v3.3.0 --with-kubernetes v1.24.1 -f config-sample.yaml

• 修改配置文件

$ vim config-sample.yaml

修改內(nèi)容說明

• 按照實(shí)際離線環(huán)境配置修改節(jié)點(diǎn)信息。
• 按實(shí)際情況添加 registry 的相關(guān)信息。

apiVersion: kubekey.kubesphere.io/v1alpha2 kind: Cluster metadata:name: sample spec:hosts:- {name: ks-k8s-master-0, address: 192.168.9.91, internalAddress: 192.168.9.91, user: root, password: "F@ywwpTj4bJtYwzpwCqD"}- {name: ks-k8s-master-1, address: 192.168.9.92, internalAddress: 192.168.9.92, user: root, password: "F@ywwpTj4bJtYwzpwCqD"}- {name: ks-k8s-master-2, address: 192.168.9.93, internalAddress: 192.168.9.93, user: root, password: "F@ywwpTj4bJtYwzpwCqD"}roleGroups:etcd:- ks-k8s-master-0- ks-k8s-master-1- ks-k8s-master-2control-plane: - ks-k8s-master-0- ks-k8s-master-1- ks-k8s-master-2worker:- ks-k8s-master-0- ks-k8s-master-1- ks-k8s-master-2controlPlaneEndpoint:## Internal loadbalancer for apiservers internalLoadbalancer: haproxydomain: lb.zdevops.com.cnaddress: ""port: 6443kubernetes:version: v1.24.1clusterName: zdevops.com.cnautoRenewCerts: truecontainerManager: containerdetcd:type: kubekeynetwork:plugin: calicokubePodsCIDR: 10.233.64.0/18kubeServiceCIDR: 10.233.0.0/18## multus support. https://github.com/k8snetworkplumbingwg/multus-cnimultusCNI:enabled: falseregistry:type: "harbor"auths:"registry.zdevops.com.cn":username: adminpassword: Harbor12345privateRegistry: "registry.zdevops.com.cn"namespaceOverride: "kubesphereio"registryMirrors: []insecureRegistries: []addons: []# 下面的內(nèi)容不修改，不做展示

在 Harbor 中創(chuàng)建項(xiàng)目

本文采用提前部署好的 Harbor 來(lái)存放鏡像，部署過程參考我之前寫的 基于 KubeSphere 玩轉(zhuǎn) k8s-Harbor 安裝手記。

你可以使用 kk 工具自動(dòng)部署 Harbor，具體參考官方離線部署文檔。

• 下載創(chuàng)建項(xiàng)目腳本模板

$ curl -O https://raw.githubusercontent.com/kubesphere/ks-installer/master/scripts/create_project_harbor.sh

• 根據(jù)實(shí)際情況修改項(xiàng)目腳本

#!/usr/bin/env bash# Harbor 倉(cāng)庫(kù)地址 url="https://registry.zdevops.com.cn"# 訪問 Harbor 倉(cāng)庫(kù)用戶 user="admin"# 訪問 Harbor 倉(cāng)庫(kù)用戶密碼 passwd="Harbor12345"# 需要?jiǎng)?chuàng)建的項(xiàng)目名列表，正常只需要?jiǎng)?chuàng)建一個(gè)**kubesphereio**即可，這里為了保留變量可擴(kuò)展性多寫了兩個(gè)。 harbor_projects=(librarykubesphereiokubesphere )for project in "${harbor_projects[@]}"; doecho "creating $project"curl -u "${user}:${passwd}" -X POST -H "Content-Type: application/json" "${url}/api/v2.0/projects" -d "{ \"project_name\": \"${project}\", \"public\": true}" done

• 執(zhí)行腳本創(chuàng)建項(xiàng)目

$ sh create_project_harbor.sh

推送離線鏡像到 Harbor 倉(cāng)庫(kù)

將提前準(zhǔn)備好的離線鏡像推送到 Harbor 倉(cāng)庫(kù)，這一步為可選項(xiàng)，因?yàn)閯?chuàng)建集群的時(shí)候也會(huì)再次推送鏡像。為了部署一次成功率，建議先推送。

$ ./kk artifact image push -f config-sample.yaml -a kubesphere-v3.3.0-artifact.tar.gz

創(chuàng)建集群并安裝 OS 依賴

$ ./kk create cluster -f config-sample.yaml -a kubesphere-v3.3.0-artifact.tar.gz --with-packages

參數(shù)說明

• config-sample.yaml：離線環(huán)境集群的配置文件。
• kubesphere-v3.3.0-artifact.tar.gz：制品包的 tar 包鏡像。
• --with-packages：若需要安裝操作系統(tǒng)依賴，需指定該選項(xiàng)。

查看集群狀態(tài)

$ kubectl logs -n kubesphere-system $(kubectl get pod -n kubesphere-system -l 'app in (ks-install, ks-installer)' -o jsonpath='{.items[0].metadata.name}') -f

正確安裝完成后，您會(huì)看到以下內(nèi)容：

************************************************** Collecting installation results ... ##################################################### ### Welcome to KubeSphere! ### #####################################################Console: http://192.168.9.91:30880 Account: admin Password: P@88w0rdNOTES：1. After you log into the console, please check themonitoring status of service components in"Cluster Management". If any service is notready, please wait patiently until all components are up and running.2. Please change the default password after login.##################################################### https://kubesphere.io 2022-06-30 14:30:19 #####################################################

登錄 Web 控制臺(tái)

通過 http://{IP}:30880 使用默認(rèn)帳戶和密碼 admin/P@88w0rd 訪問 KubeSphere 的 Web 控制臺(tái)，進(jìn)行后續(xù)的操作配置。

總結(jié)

感謝您完整的閱讀完本文，為此，您應(yīng)該 Get 到了以下技能

• 了解了清單 (manifest) 和制品 (artifact) 的概念
• 了解 manifest 和 image 資源的獲取地址
• 手寫 manifest 清單
• 根據(jù) manifest 清單制作 artifact
• 離線部署 KubeSphere 和 Kubernetes
• Harbor 鏡像倉(cāng)庫(kù)自動(dòng)創(chuàng)建項(xiàng)目
• Ansible 使用的小技巧

目前為止，我們已經(jīng)完成了最小化環(huán)境的 KubeSphere 和 K8s 集群的部署。但是，這僅僅是一個(gè)開始，后續(xù)還有很多配置和使用技巧，敬請(qǐng)期待 ...

本文由博客一文多發(fā)平臺(tái) OpenWrite 發(fā)布！

總結(jié)

以上是生活随笔為你收集整理的KubeSphere 3.3.0 离线安装教程的全部?jī)?nèi)容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網(wǎng)站內(nèi)容還不錯(cuò)，歡迎將生活随笔推薦給好友。