部署gluster
部署環境
vmware15

ubuntu16.04 192.168.245.133 k8s-master gluster heketi ubuntu16.04 192.168.245.134 k8s-node gluster ubuntu16.04 192.168.245.135 k8s-node gluster

添加硬盤
添加額外存儲
在vmware中選中虛擬機->編輯虛擬機設置->列表中選中硬盤->添加

啟動虛擬機后查看硬盤，多了一個5G的/dev/sdb盤

root@ubuntu:/usr/local/heketi/heketi# fdisk -l Disk /dev/sdb: 5 GiB, 5368709120 bytes, 10485760 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes

修改磁盤類型，選擇gpt

root@ubuntu:/usr/local/heketi/heketi# fdisk /dev/sdbWelcome to fdisk (util-linux 2.27.1).Changes will remain in memory only, until you decide to write them.Be careful before using the write command.Command (m for help):#依次輸入g->w保存退出

查看/dev/sdb盤信息，發現磁盤類型變味了gpt

root@ubuntu:/usr/local/heketi/heketi# fdisk -l Disk /dev/sdb: 5 GiB, 5368709120 bytes, 10485760 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: gpt Disk identifier: D2003869-2A4C-47F4-B600-74E60998A0A1

為磁盤/dev/sdb分區，我們這里只分一個區

root@ubuntu:/usr/local/heketi/heketi# fdisk /dev/sdb Welcome to fdisk (util-linux 2.27.1). Changes will remain in memory only, until you decide to write them. Be careful before using the write command. Command (m for help): #依次輸入n->一直按回車->w保存退出

查看分區信息，多了一個/dev/sdb1的物理卷

root@ubuntu:/usr/local/heketi/heketi# fdisk -l Disk /dev/sdb: 5 GiB, 5368709120 bytes, 10485760 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: gpt Disk identifier: D2003869-2A4C-47F4-B600-74E60998A0A1Device Start End Sectors Size Type /dev/sdb1 2048 10485726 10483679 5G Linux filesystem

gluster部署
所有節點安裝gluster

apt-get install glusterfs-server glusterfs-common glusterfs-client fuse systemctl start glusterfs-server systemctl enable glusterfs-server

在133節點上添加節點

gluster peer probe 192.168.245.134 gluster peer probe 192.168.245.135

不用添加本地節點，添加本地節點會報錯

root@ubuntu:/usr/local/heketi/heketi# gluster peer probe 192.168.245.133 peer probe: success. Probe on localhost not needed

heketi部署
下載heketi包

wget https://github.com/heketi/heketi/releases/download/v5.0.1/heketi-client-v5.0.1.linux.amd64.tar.gzwget https://github.com/heketi/heketi/releases/download/v5.0.1/heketi-v5.0.1.linux.amd64.tar.gz

解壓

tar -zxvf heketi-client-v5.0.1.linux.amd64.tar.gz tar -zxvf heketi-v5.0.1.linux.amd64.tar.gz

修改配置文件heketi.json

"_port_comment": "Heketi Server Port Number","port": "18080",#修改端口"_use_auth": "Enable JWT authorization. Please enable for deployment","use_auth": true,#修改auth"_jwt": "Private keys for access","jwt": {"_admin": "Admin has access to all APIs","admin": {"key": "adminkey" #修改admin登錄密碼},"_user": "User only has access to /volumes endpoint","user": {"key": "My Secret"}},---"executor": "ssh",# 修改登錄方式"_sshexec_comment": "SSH username and private key file information","sshexec": {"keyfile": "/root/.ssh/id_rsa", #秘鑰地址"user": "root",# 登錄用戶"port": "22",# 端口"fstab": "/etc/fstab"# /etc/fstab文件負責配置Linux開機時自動掛載的分區},

配置節點間的免密登錄
ssh-keygen #生成公鑰，然后將公鑰互相copy到各自的authorized_keys文件中即可
向heketi中添加cluster、device
編輯配置文件topology-sample.json

{"clusters": [{"nodes": [{"node": {"hostnames": {"manage": ["192.168.245.133"],"storage": ["192.168.245.133"]},"zone": 1},"devices": ["/dev/sdb1"]},{"node": {"hostnames": {"manage": ["192.168.245.134"],"storage": ["192.168.245.134"]},"zone": 1},"devices": ["/dev/sdb1"]},{"node": {"hostnames": {"manage": ["192.168.245.135"],"storage": ["192.168.245.135"]},"zone": 1},"devices": ["/dev/sdb1"]} ]}] }

執行命令

root@ubuntu:/usr/local/heketi/heketi# ./heketi-cli --server "http://192.168.245.133:18080" --user "admin" --secret "adminkey" topology load --json topology-sample.json Creating node 192.168.245.133 ... ID: 1204399251b630ccd34cd25c7c93a597Adding device /dev/sdb1 Creating node 192.168.245.134 ... ID: a77370a6e9d5ee52de9cb71a27475d4aAdding device /dev/sdb1 ... OK Creating node 192.168.245.135 ... ID: 4668bb56bee757caaf6032638b9b8269Adding device /dev/sdb1

查看cluster

root@ubuntu:/usr/local/heketi/heketi# ./heketi-cli --server "http://192.168.245.133:18080" --user "admin" --secret "adminkey" cluster listClusters:Id:baf191bf97b91c6a8b8fad77e99c4ffc

查看node

root@ubuntu:/usr/local/heketi/heketi# ./heketi-cli --server "http://192.168.245.133:18080" --user "admin" --secret "adminkey" node listId:1204399251b630ccd34cd25c7c93a597 Cluster:baf191bf97b91c6a8b8fad77e99c4ffcId:4668bb56bee757caaf6032638b9b8269 Cluster:baf191bf97b91c6a8b8fad77e99c4ffcId:a77370a6e9d5ee52de9cb71a27475d4a Cluster:baf191bf97b91c6a8b8fad77e99c4ffc

k8s使用gluster

創建storageclass，storageclass-glusterfs.yamlapiVersion: storage.k8s.io/v1beta1 kind: StorageClassmetadata:name: glusterfsprovisioner: kubernetes.io/glusterfs #表示存儲分配器，需要根據后端存儲的不同而變更parameters:resturl: "http://192.168.245.133:18080" #heketi API服務提供的urlrestauthenabled: "true" #可選參數，默認值為”false”，heketi服務開啟認證時必須設置為”true”restuser: "admin" #可選參數，開啟認證時設置相應用戶名；restuserkey: "adminkey" #可選參數，開啟認證時設置相應密碼；volumetype: "replicate:2" #可選參數，設置卷類型及其參數，如果未分配卷類型，則有分配器決定卷類型；如”volumetype: replicate:3”表示3副本的replicate卷，”volumetype: disperse:4:2”表示disperse卷，其中‘4’是數據，’2’是冗余校驗，”volumetype: none”表示distribute卷

執行命令創建

kubectl apply -f storageclass-glusterfs.yaml

查看storageclass

root@ubuntu:/usr/local/heketi/heketi# kubectl get storageclassNAME PROVISIONER AGEglusterfs kubernetes.io/glusterfs 70m 創建pvc，glusterfs-pvc.yamlkind: PersistentVolumeClaimapiVersion: v1metadata:name: glusterfs-testnamespace: defaultannotations:volume.beta.kubernetes.io/storage-class: "glusterfs"spec:accessModes:- ReadWriteManyresources:requests:storage: 1Gi

執行命令創建

root@ubuntu:/usr/local/heketi/heketi# kubectl create -f glusterfs-pvc.yaml persistentvolumeclaim/glusterfs-test created

查看pvc
狀態為Bound說明創建成功

root@ubuntu:/usr/local/heketi/heketi# kubectl get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE glusterfs-test Bound pvc-f705938e-1bf9-11ea-b5d9-000c29509b70 1Gi RWX glusterfs 12s

查看pv
這里pv為動態創建的

root@ubuntu:/usr/local/heketi/heketi# kubectl get pv NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE pvc-f705938e-1bf9-11ea-b5d9-000c29509b70 1Gi RWX Delete Bound default/glusterfs-test glusterfs 81s

問題
heketi有些卷明明存在但是卻刪不了
直接刪除heketi存儲目錄/var/lib/heketi/ 下的mounts/文件夾，然后> heketi.db 清空db文件，重新來
Can’t initialize physical volume “/dev/sdb1” of volume group “vg1” without –ff
這是因為沒有卸載之前的vg和pv
使用命令vgremove,pvremove依次刪除卷組，邏輯卷即可

總結

以上是生活随笔為你收集整理的k8s+gluster+heketi分布式存储的全部內容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網站內容還不錯，歡迎將生活随笔推薦給好友。