虛擬機創建

在自己的Mac系統里面利用Parallels Desktop創建3臺虛擬機，具體信息如下：

CentOS7-Node1: 10.211.55.7 parallels/centos-testCentOS7-Node2: 10.211.55.8parallels/centos-testCentOS7-Node3: 10.211.55.9parallels/centos-test

Master安裝

選擇CentOS7-Node1機器作為Master節點。

配置yum

更新yum源：

[parallels@CentOS7-Node1 yum.repos.d]$ cd /etc/yum.repos.d [parallels@CentOS7-Node1 yum.repos.d]$ sudo touch kubernetes.repo [kubernetes] name=Kubernetes baseurl=http://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64 enabled=1 gpgcheck=0 repo_gpgcheck=0

安裝Kubernetes環境

評估下來，利用kubeadm來搭建是大家比較推薦的，而且公司的集群也是。所以毫不憂慮就用kubeadm。

[parallels@CentOS7-Node1 yum.repos.d]$ yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes Loaded plugins: fastestmirror, langpacks You need to be root to perform this command. [parallels@CentOS7-Node1 yum.repos.d]$ sudo yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes Loaded plugins: fastestmirror, langpacks Loading mirror speeds from cached hostfile * base: mirrors.aliyun.com* extras: mirrors.aliyun.com* updates: mirrors.aliyun.com kubernetes | 1.4 kB 00:00:00 kubernetes/primary | 58 kB 00:00:00 kubernetes 421/421 Resolving Dependencies --> Running transaction check ...... # 省略一堆無意義的日志 Dependency Installed: conntrack-tools.x86_64 0:1.4.4-5.el7_7.2 cri-tools.x86_64 0:1.13.0-0 kubernetes-cni.x86_64 0:0.7.5-0 libnetfilter_cthelper.x86_64 0:1.0.0-10.el7_7.1 libnetfilter_cttimeout.x86_64 0:1.0.0-6.el7_7.1 libnetfilter_queue.x86_64 0:1.0.2-2.el7_2 socat.x86_64 0:1.7.3.2-2.el7 Complete!

關于yum的配置與升級

yum install -y yum-utils device-mapper-persistent-data lvm2 yum update

啟動docker

啟動Docker，加入開啟機動項：

[parallels@CentOS7-Node1 ~]$ sudo systemctl enable docker && systemctl start docker [sudo] password for parallels: Created symlink from /etc/systemd/system/multi-user.target.wants/docker.service to /usr/lib/systemd/system/docker.service.

啟動kubelet

啟動kubelet，加入開機啟動項：

sudo systemctl enable kubelet && systemctl start kubelet [parallels@CentOS7-Node1 ~]$ sudo systemctl enable kubelet && systemctl start kubelet Created symlink from /etc/systemd/system/multi-user.target.wants/kubelet.service to /usr/lib/systemd/system/kubelet.service. ==== AUTHENTICATING FOR org.freedesktop.systemd1.manage-units === Authentication is required to manage system services or units. Authenticating as: Parallels (parallels) Password: ==== AUTHENTICATION COMPLETE ===

kubeadm config

[parallels@CentOS7-Node1 Workspace]$ kubeadm config print init-defaults apiVersion: kubeadm.k8s.io/v1beta2 bootstrapTokens: - groups:- system:bootstrappers:kubeadm:default-node-tokentoken: abcdef.0123456789abcdefttl: 24h0m0susages:- signing- authentication kind: InitConfiguration localAPIEndpoint: advertiseAddress: 1.2.3.4bindPort: 6443 nodeRegistration: criSocket: /var/run/dockershim.sockname: centos7-node1taints:- effect: NoSchedulekey: node-role.kubernetes.io/master --- apiServer: timeoutForControlPlane: 4m0s apiVersion: kubeadm.k8s.io/v1beta2 certificatesDir: /etc/kubernetes/pki clusterName: kubernetes controllerManager: {} dns: type: CoreDNS etcd: local:dataDir: /var/lib/etcd imageRepository: k8s.gcr.io kind: ClusterConfiguration kubernetesVersion: v1.16.0 networking: dnsDomain: cluster.localserviceSubnet: 10.96.0.0/12 scheduler: {} kubeadm config print init-defaults > /home/parallels/Workspace/init.default.yaml

配置Docker

首先要安裝好Docker環境，請參考之前的?http://www.cyblogs.com/centos7shang-an-zhuang-docker/

Docker的一些相關命令

yum install docker-ce-18.09.9-3.el7 # 指定版本為18.09.9-3.el7systemctl status docker systemctl restart docker systemctl daemon-reload

下載kubernetes的相關鏡像

配置鏡像地址，但沒什么用。后面還是需要用到國內的鏡像：

echo '{"registry-mirrors":["https://docker.mirrors.ustc.edu.cn"]}' > /etc/docker/daemon.json # 如果提示沒有權限，就手動vim添加進去。然后重啟docker服務

查看一下kubernetes依賴的鏡像名稱以及版本

[parallels@CentOS7-Node1 Workspace]$ kubeadm config images list W1022 13:51:12.550171 19704 version.go:101] could not fetch a Kubernetes version from the internet: unable to get URL "https://dl.k8s.io/release/stable-1.txt": Get https://dl.k8s.io/release/stable-1.txt: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) W1022 13:51:12.550458 19704 version.go:102] falling back to the local client version: v1.16.2 k8s.gcr.io/kube-apiserver:v1.16.2 k8s.gcr.io/kube-controller-manager:v1.16.2 k8s.gcr.io/kube-scheduler:v1.16.2 k8s.gcr.io/kube-proxy:v1.16.2 k8s.gcr.io/pause:3.1 k8s.gcr.io/etcd:3.3.15-0 k8s.gcr.io/coredns:1.6.2

如果網絡OK，應該直接執行這個命令即可，但實際會報錯誤。

[parallels@CentOS7-Node1 Workspace]$ sudo kubeadm config images pull --config=/home/parallels/Workspace/init.default.yaml# 這里由于網絡拉取鏡像的問題，基本無法操作，只能先去aliyun獲取回來后再修改tag的方式,錯誤如下。 [parallels@CentOS7-Node1 Workspace]$ sudo kubeadm config images pull --config=/home/parallels/Workspace/init.default.yaml [sudo] password for parallels: failed to pull image "k8s.gcr.io/kube-apiserver:v1.16.0": output: Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) , error: exit status 1 To see the stack trace of this error execute with --v=5 or higher

獲取鏡像

通過另外一種方式來獲取鏡像：

touch kubeadm.sh#!/bin/bashKUBE_VERSION=v1.16.0 KUBE_PAUSE_VERSION=3.1 ETCD_VERSION=3.3.15-0 CORE_DNS_VERSION=1.6.2GCR_URL=k8s.gcr.io ALIYUN_URL=registry.cn-hangzhou.aliyuncs.com/google_containersimages=( kube-apiserver:${KUBE_VERSION}kube-controller-manager:${KUBE_VERSION}kube-scheduler:${KUBE_VERSION}kube-proxy:${KUBE_VERSION}pause:${KUBE_PAUSE_VERSION}etcd:${ETCD_VERSION}coredns:${CORE_DNS_VERSION} )for imageName in ${images[@]} ; do docker pull $ALIYUN_URL/$imageNamedocker tag $ALIYUN_URL/$imageName $GCR_URL/$imageNamedocker rmi $ALIYUN_URL/$imageName done

拉取鏡像

chmod u+x kubeadm.sh # 添加權限 sudo ./kubeadm.sh

剩下的就是耐心等待......

查看最終本地的鏡像

[root@CentOS7-Node1 Workspace]# docker images REPOSITORY TAG IMAGE ID CREATED SIZE k8s.gcr.io/kube-apiserver v1.16.0 b305571ca60a 4 weeks ago 217MB k8s.gcr.io/kube-proxy v1.16.0 c21b0c7400f9 4 weeks ago 86.1MB k8s.gcr.io/kube-controller-manager v1.16.0 06a629a7e51c 4 weeks ago 163MB k8s.gcr.io/kube-scheduler v1.16.0 301ddc62b80b 4 weeks ago 87.3MB k8s.gcr.io/etcd 3.3.15-0 b2756210eeab 6 weeks ago 247MB k8s.gcr.io/coredns 1.6.2 bf261d157914 2 months ago 44.1MB k8s.gcr.io/pause 3.1 da86e6ba6ca1 22 months ago 742kB [parallels@CentOS7-Node1 Workspace]$ sudo kubeadm init --config=init.default.yaml [init] Using Kubernetes version: v1.16.0 [preflight] Running pre-flight checks[WARNING Firewalld]: firewalld is active, please ensure ports [6443 10250] are open or your cluster may not function correctly[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 19.03.4. Latest validated version: 18.09 error execution phase preflight: [preflight] Some fatal errors occurred: [ERROR Swap]: running with swap on is not supported. Please disable swap [preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...` To see the stack trace of this error execute with --v=5 or higher

關閉防火墻

解決掉防火墻的問題，請參閱：http://www.cyblogs.com/centos7cha-kan-he-guan-bi-fang-huo-qiang/

cgroupfs錯誤

detected "cgroupfs" as the Docker cgroup driver {"registry-mirrors": ["https://registry.docker-cn.com"],"live-restore": true,"exec-opts": ["native.cgroupdriver=systemd" # 修改用戶] }# 重新啟動Docker systemctl restart docker systemctl status docker

禁止swap

還是發現需要禁止掉swap

Oct 22 16:35:36 CentOS7-Node1 kubelet[1395]: F1022 16:35:36.065168 1395 server.go:271] failed to run Kubelet: running with swap on is not supported, please disable swap! or set --fail-swap-on flag to false. /proc/swaps contained: [Filename Type Size Used Priority /dev/dm-1 partition 2097148 29952 -1] swapoff -a #要永久禁掉swap分區，打開如下文件注釋掉swap那一行 sudo vi /etc/fstab

再次啟動kubeadm init

kubeadm init --config=init.default.yaml[init] Using Kubernetes version: v1.16.2 ... [preflight] Pulling images required for setting up a Kubernetes cluster ... [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" ... [certs] Using certificateDir folder "/etc/kubernetes/pki" ... [kubeconfig] Using kubeconfig folder "/etc/kubernetes" ... [kubelet-check] Initial timeout of 40s passed. [kubelet-check] It seems like the kubelet isn't running or healthy. [kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp [::1]:10248: connect: connection refused. [kubelet-check] It seems like the kubelet isn't running or healthy. [kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp [::1]:10248: connect: connection refused. [kubelet-check] It seems like the kubelet isn't running or healthy. [kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp [::1]:10248: connect: connection refused. [kubelet-check] It seems like the kubelet isn't running or healthy. [kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp [::1]:10248: connect: connection refused. [kubelet-check] It seems like the kubelet isn't running or healthy. [kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp [::1]:10248: connect: connection refused.

出現錯誤了,變更Docker的版本后，繼續執行，還是會報錯誤。

[ERROR FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml]: /etc/kubernetes/manifests/kube-apiserver.yaml already exists[ERROR FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml]: /etc/kubernetes/manifests/kube-controller-manager.yaml already exists

重設kubeadm

這里需要重設kubeadm了。具體操作如下:

kubeadm reset echo '1' > /proc/sys/net/bridge/bridge-nf-call-iptables

journalctl查看日志

journalctl -xefu kubelet

這里還是會報錯，因為之前的

apiVersion: kubeadm.k8s.io/v1beta2 kind: ClusterConfiguration imageRepository: k8s.gcr.io kubernetesVersion: v1.16.0 networking: dnsDomain: cluster.localserviceSubnet: "10.96.0.0/16"

繼續執行init的過程，kubeadm init --config=/home/parallels/Workspace/init.default.yaml

[addons] Applied essential addon: CoreDNS [addons] Applied essential addon: kube-proxyYour Kubernetes control-plane has initialized successfully!To start using your cluster, you need to run the following as a regular user:mkdir -p $HOME/.kubesudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/configsudo chown $(id -u):$(id -g) $HOME/.kube/configYou should now deploy a pod network to the cluster. Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at: https://kubernetes.io/docs/concepts/cluster-administration/addons/Then you can join any number of worker nodes by running the following on each as root:kubeadm join 10.211.55.7:6443 --token imwj34.ksfiwzj5ga80du0r \ --discovery-token-ca-cert-hash sha256:7ffef85880ed43dd539afa045715f9ad5bef15e904cede96213d6cfd4adb0795

真心不容易，這里一直反反復復執行。只要是images的版本問題以及init的過程容易出錯。

驗證configmap

[root@CentOS7-Node1 ~]# kubectl get -n kube-system configmap NAME DATA AGE coredns 1 5m49s extension-apiserver-authentication 6 5m53s kube-proxy 2 5m49s kubeadm-config 2 5m50s kubelet-config-1.16 1 5m50s

安裝Node，加入集群

安裝跟Master一直的基本環境，包括docker，kubelet，kubeadm等，重復上面的動作。

scp root@10.211.55.7:/home/parallels/Workspace/init.default.yaml . scp root@10.211.55.7:/home/parallels/Workspace/kubeadm.sh . yum install docker-ce-18.06.3.ce-3.el7

為kubeadm命令生成配置文件，創建join-config.yaml，內容如下：

apiVersion: kubeadm.k8s.io/v1beta2 kind: JoinConfiguration discovery: bootstrapToken:apiServerEndpoint: 10.211.55.7:6443token: imwj34.ksfiwzj5ga80du0runsafeSkipCAVerification: truetlsBootstrapToken: imwj34.ksfiwzj5ga80du0r

其中，apiServerEndpoint的值來自于Master的服務器地址，這里就是10.211.55.7。token和tlsBootstrapToken的值就來自于kubeadm init安裝Master的最后一行提示信息。這里一定要注意yaml文件的格式，否則執行會報錯誤。

[root@CentOS7-Node2 Workspace]# kubeadm join --config=join-config.yaml [preflight] Running pre-flight checks error execution phase preflight: [preflight] Some fatal errors occurred: [ERROR Swap]: running with swap on is not supported. Please disable swap [preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...` To see the stack trace of this error execute with --v=5 or higher [root@CentOS7-Node2 Workspace]# swapoff -a [root@CentOS7-Node2 Workspace]# kubeadm join --config=join-config.yaml [preflight] Running pre-flight checks [preflight] Reading configuration from the cluster... [preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml' [kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.16" ConfigMap in the kube-system namespace [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" [kubelet-start] Activating the kubelet service [kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...This node has joined the cluster: * Certificate signing request was sent to apiserver and a response was received. * The Kubelet was informed of the new secure connection details.Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

安裝網絡插件

去Master機器，執行：

[root@CentOS7-Node1 Workspace]# kubectl get nodes NAME STATUS ROLES AGE VERSION centos7-node1 NotReady master 154m v1.16.2 centos7-node2 NotReady <none> 2m49s v1.16.2

這里顯示的是NotReady狀態，是因為還沒有安裝CNI網絡插件。我們選擇weave插件來安裝。

[root@CentOS7-Node1 Workspace]# kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')" serviceaccount/weave-net created clusterrole.rbac.authorization.k8s.io/weave-net created clusterrolebinding.rbac.authorization.k8s.io/weave-net created role.rbac.authorization.k8s.io/weave-net created rolebinding.rbac.authorization.k8s.io/weave-net created daemonset.apps/weave-net created

驗證集群是否安裝完成

[root@CentOS7-Node1 Workspace]# kubectl get pods -n kube-system NAME READY STATUS RESTARTS AGE coredns-5644d7b6d9-9fr9p 0/1 ContainerCreating 0 172m coredns-5644d7b6d9-pmpkq 0/1 ContainerCreating 0 172m etcd-centos7-node1 1/1 Running 0 171m kube-apiserver-centos7-node1 1/1 Running 0 171m kube-controller-manager-centos7-node1 1/1 Running 0 171m kube-proxy-ccnht 1/1 Running 0 21m kube-proxy-rdq9l 1/1 Running 0 172m kube-scheduler-centos7-node1 1/1 Running 0 171m weave-net-6hw26 2/2 Running 0 8m7s weave-net-qv8vz 2/2 Running 0 8m7s

發現coredns一直處于ContainerCreating的狀態。具體的看一下錯誤信息。

[root@CentOS7-Node1 Workspace]# kubectl describe pod coredns-5644d7b6d9-9fr9p -n kube-system Name: coredns-5644d7b6d9-9fr9p Namespace: kube-system Priority: 2000000000 Priority Class Name: system-cluster-critical Node: centos7-node2/10.211.55.8 Start Time: Tue, 22 Oct 2019 20:49:47 +0800 Labels: k8s-app=kube-dns pod-template-hash=5644d7b6d9 .... # 此處省略一些 Events: Type Reason Age From Message---- ------ ---- ---- -------Warning FailedScheduling <unknown> default-scheduler 0/1 nodes are available: 1 node(s) had taints that the pod didn't tolerate.Warning FailedScheduling <unknown> default-scheduler 0/2 nodes are available: 2 node(s) had taints that the pod didn't tolerate.Normal Scheduled <unknown> default-scheduler Successfully assigned kube-system/coredns-5644d7b6d9-9fr9p to centos7-node2Warning FailedCreatePodSandBox 2m kubelet, centos7-node2 Failed create pod sandbox: rpc error: code = DeadlineExceeded desc = context deadline exceededNormal SandboxChanged 119s kubelet, centos7-node2 Pod sandbox changed, it will be killed and re-created.

這里可以看出一些錯誤：

Oct 22 10:50:15 CentOS7-Node1 kubelet[7649]: F1022 10:50:15.170550 7649 server.go:196] failed to load Kubelet config file /var/lib/kubelet/config.yaml, error failed to read kubelet config file "/var/lib/kubelet/config.yaml", Oct 22 10:50:15 CentOS7-Node1 systemd[1]: kubelet.service: main process exited, code=exited, status=255/n/a

可以刪除掉一個pod的方式讓它重新啟動：

[root@CentOS7-Node1 ~]# kubectl delete pod coredns-5644d7b6d9-9fr9p -n kube-system pod "coredns-5644d7b6d9-9fr9p" deleted

看了太多的文章與博客，發現沒有幾個寫的太完全的，都是寫的成功的經驗，實際上中間不知道有各種奇怪問題。說句實話，k8s很方便，但是門檻很高，依賴的東西真的太多太多了。特別是版本問題導致的問題，很難解決掉。

最后看一下成功的圖片吧

常用命令匯總

systemctl daemon-reloadsystemctl restart kubeletkubectl get pods -n kube-systemkubectl describe pod coredns-5644d7b6d9-lqtks -n kube-systemkubectl delete pod coredns-5644d7b6d9-qh4bc -n kube-system # 允許master節點部署pod kubectl taint nodes --all node-role.kubernetes.io/master- # 禁止master部署pod kubectl taint nodes k8s node-role.kubernetes.io/master=true:NoSchedulekubeadm resetsystemctl enable docker && systemctl start dockersystemctl enable kubelet && systemctl start kubeletjournalctl -xefu kubelet

參考地址

• https://www.cnblogs.com/yufeng218/p/8370670.html
• https://stackoverflow.com/questions/55531834/kubeadm-fails-to-initialize-when-kubeadm-init-is-called
• https://zhuanlan.zhihu.com/p/31398416
• https://blog.csdn.net/M82_A1/article/details/97626309
• https://blog.csdn.net/liumiaocn/article/details/99608323
• https://www.hi-linux.com/posts/54191.html
• https://blog.csdn.net/BigData_Mining/article/details/88683459

總結

以上是生活随笔為你收集整理的CentOS7系统上Kubernetes集群搭建的全部內容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網站內容還不錯，歡迎將生活随笔推薦給好友。