實驗環境 ?
-?10Gb網絡環境
-?gluster1??192.168.100.155 三塊磁盤
-?gluster2??192.168.100.156 三塊磁盤
-?gluster2??192.168.100.157 三塊磁盤
-?gluster2??192.168.100.158 三塊磁盤

安裝配置

gluster1存儲節點安裝如下，其它節點同樣操作

#?格式化磁盤?#mkfs.ext4?-T?largefile?/dev/sdb?#?超過2T的磁盤需要加入largefile,簡稱快格mkfs.ext4?-T?largefile?/dev/sdc#?掛載磁盤?#mkdir?-p?/glusterfs/disk{1，2}/?#創建磁盤掛載目錄vim?/etc/fstab?#?添加如下內容/dev/sdb???/glusterfs/disk1????ext4?defaults?0?2/dev/sdc???/glusterfs/disk2????ext4?defaults?0?2mount?-a?#?根據/etc/fstab掛載所有磁盤mkdir?-p?/glusterfs/disk{1，2}/data??#掛載完畢后，創建data文件夾 #?hosts文件?#vim?/etc/hosts?#?添加如下內容192.168.1.155?gluster1192.168.1.156?gluster2192.168.1.157?gluster3192.168.1.158?gluster4????#?ntp服務?#yum?-y?install?ntpservice?ntpd?startchkconfig?ntpd?on#??glusterfs?server?#wget?-P?/etc/yum.repos.d?http://download.gluster.org/pub/gluster/glusterfs/LATEST/CentOS/glusterfs-epel.repoyum?-y?install?glusterfs-serverservice?glusterd?startchkconfig?glusterd?on****gluster??#直接敲gluster,然后按Enter會進入gluster?shell環境，就跟bash?shell一樣，有命令提示功能gluster?peer?probe?gluster2??#?probe?other?serversgluster?peer?probe?gluster3gluster?peer?probe?gluster4????gluster?peer?status?#查看gluster?peer狀態#創建一個名稱為openstack的volumegluster?volume?create?openstack?stripe?4?replica?2?transport?tcp??gluster1:/glusterfs/disk1/data?gluster1:/glusterfs/disk2/data?gluster2:/glusterfs/disk1/data?gluster2:/glusterfs/disk2/data?gluster3:/glusterfs/disk1/data?gluster3:/glusterfs/disk2/data?gluster4:/glusterfs/disk1/data?gluster4:/glusterfs/disk2/data#stripe:相當于raid0#replica:相當于raid1#bricks?=?stripe?*?replicagluster?volume?start?openstack?gluster?volume?info#?glusterfs?client?#wget?-P?/etc/yum.repos.d?http://download.gluster.org/pub/gluster/glusterfs/LATEST/CentOS/glusterfs-epel.repoyum?-y?install?glusterfs?glusterfs-fusemount?-t?glusterfs?gluster1:/openstack?/mnt?#掛載volume?#開機自動掛載glusterfs卷cat?/etc/fstabgluster1:/openstack?/mnt?glusterfs?defaults,_netdev?0?0#?_netdev：文件系統位于需要網絡連接的設備上(系統網絡啟動時，文件系統才被掛載)

Glusterfs卷類型
????Glusterfs支持五種Volume，即Distribute卷、Stripe卷、Replica卷、Distribute stripe卷和Distribute replica卷，這五種卷可以滿足不同應用對高性能、高可用的需求。?
????
????(1)distribute volume：分布式卷，文件通過hash算法分布到brick server上，這種卷是glusterfs的基礎和最大特點；????
????
????(2)stripe volume：條帶卷，類似RAID0，條帶數=brick server數量，文件分成數據塊以Round Robin方式分布到brick server上，并發粒度是數據塊，大文件性能高；
????
????(3)replica volume：鏡像卷，類似RAID1，鏡像數=brick server數量，所以brick server上文件數據相同，構成n-way鏡像，可用性高；??
????
????(4)distribute stripe volume：分布式條帶卷，brick server數量是條帶數的倍數，兼具distribute和stripe卷的特點；???
????
????(5)distribute replica volume：分布式鏡像卷，brick server數量是鏡像數的倍數，兼具distribute和replica卷的特點；
? ??

GlusterFS specifcation in QEMU

gluster[+transport]://[server[:port]]/volname/p_w_picpath[?socket=...]gluster?is?the?protocol.?Examples:gluster://1.2.3.4/testvol/a.imggluster+tcp://1.2.3.4/testvol/a.imggluster+tcp://1.2.3.4:24007/testvol/dir/a.imggluster+tcp://[1:2:3:4:5:6:7:8]/testvol/dir/a.imggluster+tcp://[1:2:3:4:5:6:7:8]:24007/testvol/dir/a.imggluster+tcp://server.domain.com:24007/testvol/dir/a.imggluster+unix:///testvol/dir/a.img?socket=/tmp/glusterd.socketgluster+rdma://1.2.3.4:24007/testvol/a.img

Glusterfs集成flashcache
???Flashcache是Facebook技術團隊開發的一個內核模塊，通過在文件系統（VFS）和設備驅動之間新增一次緩存層，可以用SSD作為介質的緩存，通過將傳統硬盤上的熱門數據緩存到SSD上，加速服務器磁盤讀寫性能。

????#準備編譯環境flashcache是ko文件，需要在當前版本的內核編譯出來的模塊才能使用。如果安裝的目標機器內核版本跟編譯機器的內核版本一致，那么如下操作即可：yum?install?-y?kernel-devel#下載flashcache?tar包wget?--no-check-certificate?-O?flashcache.tar.gz?https://github.com/facebook/flashcache/tarball/master#解壓tar?xf?flashcache.tar.gz????#編譯安裝cd?facebook-flashcache-d359499/make?KERNEL_TREE='/usr/src/kernels/2.6.32-431.20.3.el6.x86_64/'?install????#編譯后的產物拷貝到目標機器scp?src/{flashcache.ko,utils/*}?192.168.1.191:/root/????#到目標機器上將相應文件拷貝到相應目錄mkdir?/lib/modules/2.6.32-431.20.3.el6.x86_64/extra/flashcache/mv?flashcache.ko?/lib/modules/2.6.32-431.el6.x86_64/extra/flashcache/mv?utils/*?/sbin/depmod?-a???#加載模塊modprobe?flashcache????#創建flashcache盤映射flashcache_create?-p?back?cachedev?/dev/vdc?/dev/vdb使用flashcache_create命令來創建映射，需要如下參數：-p?back|thru|around?模式writeback，writethrough，writearoundcachedev?生成lvm映射的盤名ssd_devname?SSD盤盤符(不需要格式化)disk_devname?數據盤盤符#掛載mount?/dev/mapper/cachedev??/glusterfs/disk1/#重新加載flashcache盤映射系統重啟的時候，需要對flashcache重新加載，使用命令flashcache_load來達到目的，執行完load后再進行掛載。flashcache_load?/dev/vdc#刪除flashcache盤映射使用命令flashcache_destroy來對flashcache映射進行刪除操作，刪除后mapper映射仍然存在，需要對其進行remove操作：flashcache_destroy?/dev/vdbdmsetup?remove?/dev/mapper/flashcache#flashcache內核參數調優vim?/etc/sysctl.confdev.flashcache.vdc+vdb.skip_seq_thresh_kb?=?1024?#超過1M的順序寫直接落到磁盤上

Glusterfs 運維常用命令

????1、刪除卷?gluster?volume?stop?img?gluster?volume?delete?img?2、將機器移出集群?gluster?peer?detach?172.28.26.102?只允許172.28.0.0的網絡訪問glusterfs?gluster?volume?set?img?auth.allow?172.28.26.*?3、加入新的機器并添加到卷里(由于副本數設置為2,至少要添加2（4、6、8..）臺機器)?gluster?peer?probe?172.28.26.105?gluster?peer?probe?172.28.26.106?gluster?volume?add-brick?img?172.28.26.105:/data/gluster?172.28.26.106:/data/gluster#卷擴容后，需要執行rebalancea、執行rebalance前getfattr?-n?trusted.glusterfs.pathinfo?nova/???#?查看文件系統的擴展屬性，getfattr依賴attr?rpm包file:?nova/trusted.glusterfs.pathinfo="((<DISTRIBUTE:openstack-dht>?(<REPLICATE:openstack-replicate-0>?<POSIX(/mnt/gluster):gluster-1:/mnt/gluster/nova>?<POSIX(/mnt/gluster):gluster-2:/mnt/gluster/nova>)?(<REPLICATE:openstack-replicate-1>?<POSIX(/mnt/gluster):compute-2:/mnt/gluster/nova>?<POSIX(/mnt/gluster):compute-1:/mnt/gluster/nova>))?(openstack-dht-layout?(openstack-replicate-0?0?4294967295)?(openstack-replicate-1?0?0)))"??#?(openstack-replicate-0?0?4294967295)?hash值b、執行rebalance后gluster?volume?rebalance?openstack?fix-layout?start???#??fix-layout不搬遷數據volume?rebalance:?openstack:?success:?Starting?rebalance?on?volume?openstack?has?been?successful.ID:?a7fc8c7f-a3e2-4783-b59e-3051327e52ecgetfattr?-n?trusted.glusterfs.pathinfo?nova/??????????#??再次查看文件系統的擴展屬性file:?nova/trusted.glusterfs.pathinfo="((<DISTRIBUTE:openstack-dht>?(<REPLICATE:openstack-replicate-0>?<POSIX(/mnt/gluster):gluster-1:/mnt/gluster/nova>?<POSIX(/mnt/gluster):gluster-2:/mnt/gluster/nova>)?(<REPLICATE:openstack-replicate-1>?<POSIX(/mnt/gluster):compute-2:/mnt/gluster/nova>?<POSIX(/mnt/gluster):compute-1:/mnt/gluster/nova>))?(openstack-dht-layout?(openstack-replicate-0?0?2147483646)?(openstack-replicate-1?2147483647?4294967295)))"??#?hash值重新分布了4、收縮卷?#收縮卷前gluster需要先移動數據到其他位置?gluster?volume?remove-brick?img?172.28.26.101:/data/gluster/img?172.28.26.102:/data/gluster/img?start?#查看遷移狀態?gluster?volume?remove-brick?img?172.28.26.101:/data/gluster/img?172.28.26.102:/data/gluster/img?status?#遷移完成后提交?gluster?volume?remove-brick?img?172.28.26.101:/data/gluster/img?172.28.26.102:/data/gluster/img?commit?遷移卷?#?將172.28.26.101的數據遷移到,先將172.28.26.107加入集群?gluster?peer?probe?172.28.26.107?gluster?volume?replace-brick?img?172.28.26.101:/data/gluster/img?172.28.26.107:/data/gluster/img?start?#?查看遷移狀態gluster?volume?replace-brick?img?172.28.26.101:/data/gluster/img?172.28.26.107:/data/gluster/img?status?#?數據遷移完畢后提交gluster?volume?replace-brick?img?172.28.26.101:/data/gluster/img?172.28.26.107:/data/gluster/img?commit?#?如果機器172.28.26.101出現故障已經不能運行,執行強制提交然后要求gluster馬上執行一次同步?gluster?volume?replace-brick?img?172.28.26.101:/data/gluster/img?172.28.26.102:/data/gluster/img?commit?-force?gluster?volume?heal?imgs?full

Glusterfs內核參數調優?

????vm.dirty_background_ratio?=?5??#?5%的意思，根據實際內存大小來定，數據從內存flush到磁盤上，vm.dirty_ratio?=?20vm.dirty_writeback_centisecs?=?100??#?100%秒，數據flush到磁盤上vm.dirty_expire_centisecs?=?3000

Glusterfs TroubleShooting

1、錯誤：?GlusterFS:?{path}?or?a?prefix?of?it?is?already?part?of?a?volume???? 解決方法:?setfattr?-x?trusted.glusterfs.volume-id?$brick_pathsetfattr?-x?trusted.gfid?$brick_pathrm?-rf?$brick_path/.glusterfs2、錯誤：glusterd服務起不來 解決方法：less?/var/log/glusterfs/etc-glusterfs-glusterd.vol.log??#?查看這個日志文件,一般都是volume配置文件出錯cd?/var/lib/glusterd/volsrm?-rf?openstack/?ssd/service?glusterd?restart

Glusterfs小知識

1、glusterfs集群中的任意一個節點都可以拿到所有volume的配置文件，不一定是brick的volume的節點 2、掛載glusterfs?volume的時候，可以通過管理網掛載，只要這個ip可以到達glusterfs節點就行，它就可以拿到volume配置；glusterfs集群數據流量還是走存儲網。3、glusterfs:?客戶端進程名glusterfsd:?服務端進程名glusterd:?volume管理進程、I/O調度進程/var/log/glusterfs/[volume掛在點名字].log:?glusterfs客戶端log

參考鏈接?
劉愛貴glusterfs分布式文件系統研究
flashcache編譯安裝
QEMU-GlusterFS native integration

轉載于:https://blog.51cto.com/iceyao/1634839

總結

以上是生活随笔為你收集整理的Glusterfs入门的全部內容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網站內容還不錯，歡迎將生活随笔推薦給好友。