實驗環(huán)境 ?
-?10Gb網(wǎng)絡(luò)環(huán)境
-?gluster1??192.168.100.155 三塊磁盤
-?gluster2??192.168.100.156 三塊磁盤
-?gluster2??192.168.100.157 三塊磁盤
-?gluster2??192.168.100.158 三塊磁盤

安裝配置

gluster1存儲節(jié)點安裝如下，其它節(jié)點同樣操作

#?格式化磁盤?#mkfs.ext4?-T?largefile?/dev/sdb?#?超過2T的磁盤需要加入largefile,簡稱快格mkfs.ext4?-T?largefile?/dev/sdc#?掛載磁盤?#mkdir?-p?/glusterfs/disk{1，2}/?#創(chuàng)建磁盤掛載目錄vim?/etc/fstab?#?添加如下內(nèi)容/dev/sdb???/glusterfs/disk1????ext4?defaults?0?2/dev/sdc???/glusterfs/disk2????ext4?defaults?0?2mount?-a?#?根據(jù)/etc/fstab掛載所有磁盤mkdir?-p?/glusterfs/disk{1，2}/data??#掛載完畢后，創(chuàng)建data文件夾 #?hosts文件?#vim?/etc/hosts?#?添加如下內(nèi)容192.168.1.155?gluster1192.168.1.156?gluster2192.168.1.157?gluster3192.168.1.158?gluster4????#?ntp服務(wù)?#yum?-y?install?ntpservice?ntpd?startchkconfig?ntpd?on#??glusterfs?server?#wget?-P?/etc/yum.repos.d?http://download.gluster.org/pub/gluster/glusterfs/LATEST/CentOS/glusterfs-epel.repoyum?-y?install?glusterfs-serverservice?glusterd?startchkconfig?glusterd?on****gluster??#直接敲gluster,然后按Enter會進入gluster?shell環(huán)境，就跟bash?shell一樣，有命令提示功能gluster?peer?probe?gluster2??#?probe?other?serversgluster?peer?probe?gluster3gluster?peer?probe?gluster4????gluster?peer?status?#查看gluster?peer狀態(tài)#創(chuàng)建一個名稱為openstack的volumegluster?volume?create?openstack?stripe?4?replica?2?transport?tcp??gluster1:/glusterfs/disk1/data?gluster1:/glusterfs/disk2/data?gluster2:/glusterfs/disk1/data?gluster2:/glusterfs/disk2/data?gluster3:/glusterfs/disk1/data?gluster3:/glusterfs/disk2/data?gluster4:/glusterfs/disk1/data?gluster4:/glusterfs/disk2/data#stripe:相當(dāng)于raid0#replica:相當(dāng)于raid1#bricks?=?stripe?*?replicagluster?volume?start?openstack?gluster?volume?info#?glusterfs?client?#wget?-P?/etc/yum.repos.d?http://download.gluster.org/pub/gluster/glusterfs/LATEST/CentOS/glusterfs-epel.repoyum?-y?install?glusterfs?glusterfs-fusemount?-t?glusterfs?gluster1:/openstack?/mnt?#掛載volume?#開機自動掛載glusterfs卷cat?/etc/fstabgluster1:/openstack?/mnt?glusterfs?defaults,_netdev?0?0#?_netdev：文件系統(tǒng)位于需要網(wǎng)絡(luò)連接的設(shè)備上(系統(tǒng)網(wǎng)絡(luò)啟動時，文件系統(tǒng)才被掛載)

Glusterfs卷類型
????Glusterfs支持五種Volume，即Distribute卷、Stripe卷、Replica卷、Distribute stripe卷和Distribute replica卷，這五種卷可以滿足不同應(yīng)用對高性能、高可用的需求。?
????
????(1)distribute volume：分布式卷，文件通過hash算法分布到brick server上，這種卷是glusterfs的基礎(chǔ)和最大特點；????
????
????(2)stripe volume：條帶卷，類似RAID0，條帶數(shù)=brick server數(shù)量，文件分成數(shù)據(jù)塊以Round Robin方式分布到brick server上，并發(fā)粒度是數(shù)據(jù)塊，大文件性能高；
????
????(3)replica volume：鏡像卷，類似RAID1，鏡像數(shù)=brick server數(shù)量，所以brick server上文件數(shù)據(jù)相同，構(gòu)成n-way鏡像，可用性高；??
????
????(4)distribute stripe volume：分布式條帶卷，brick server數(shù)量是條帶數(shù)的倍數(shù)，兼具distribute和stripe卷的特點；???
????
????(5)distribute replica volume：分布式鏡像卷，brick server數(shù)量是鏡像數(shù)的倍數(shù)，兼具distribute和replica卷的特點；
? ??

GlusterFS specifcation in QEMU

gluster[+transport]://[server[:port]]/volname/p_w_picpath[?socket=...]gluster?is?the?protocol.?Examples:gluster://1.2.3.4/testvol/a.imggluster+tcp://1.2.3.4/testvol/a.imggluster+tcp://1.2.3.4:24007/testvol/dir/a.imggluster+tcp://[1:2:3:4:5:6:7:8]/testvol/dir/a.imggluster+tcp://[1:2:3:4:5:6:7:8]:24007/testvol/dir/a.imggluster+tcp://server.domain.com:24007/testvol/dir/a.imggluster+unix:///testvol/dir/a.img?socket=/tmp/glusterd.socketgluster+rdma://1.2.3.4:24007/testvol/a.img

Glusterfs集成flashcache
???Flashcache是Facebook技術(shù)團隊開發(fā)的一個內(nèi)核模塊，通過在文件系統(tǒng)（VFS）和設(shè)備驅(qū)動之間新增一次緩存層，可以用SSD作為介質(zhì)的緩存，通過將傳統(tǒng)硬盤上的熱門數(shù)據(jù)緩存到SSD上，加速服務(wù)器磁盤讀寫性能。

????#準(zhǔn)備編譯環(huán)境flashcache是ko文件，需要在當(dāng)前版本的內(nèi)核編譯出來的模塊才能使用。如果安裝的目標(biāo)機器內(nèi)核版本跟編譯機器的內(nèi)核版本一致，那么如下操作即可：yum?install?-y?kernel-devel#下載flashcache?tar包wget?--no-check-certificate?-O?flashcache.tar.gz?https://github.com/facebook/flashcache/tarball/master#解壓tar?xf?flashcache.tar.gz????#編譯安裝cd?facebook-flashcache-d359499/make?KERNEL_TREE='/usr/src/kernels/2.6.32-431.20.3.el6.x86_64/'?install????#編譯后的產(chǎn)物拷貝到目標(biāo)機器scp?src/{flashcache.ko,utils/*}?192.168.1.191:/root/????#到目標(biāo)機器上將相應(yīng)文件拷貝到相應(yīng)目錄mkdir?/lib/modules/2.6.32-431.20.3.el6.x86_64/extra/flashcache/mv?flashcache.ko?/lib/modules/2.6.32-431.el6.x86_64/extra/flashcache/mv?utils/*?/sbin/depmod?-a???#加載模塊modprobe?flashcache????#創(chuàng)建flashcache盤映射flashcache_create?-p?back?cachedev?/dev/vdc?/dev/vdb使用flashcache_create命令來創(chuàng)建映射，需要如下參數(shù)：-p?back|thru|around?模式writeback，writethrough，writearoundcachedev?生成lvm映射的盤名ssd_devname?SSD盤盤符(不需要格式化)disk_devname?數(shù)據(jù)盤盤符#掛載mount?/dev/mapper/cachedev??/glusterfs/disk1/#重新加載flashcache盤映射系統(tǒng)重啟的時候，需要對flashcache重新加載，使用命令flashcache_load來達到目的，執(zhí)行完load后再進行掛載。flashcache_load?/dev/vdc#刪除flashcache盤映射使用命令flashcache_destroy來對flashcache映射進行刪除操作，刪除后mapper映射仍然存在，需要對其進行remove操作：flashcache_destroy?/dev/vdbdmsetup?remove?/dev/mapper/flashcache#flashcache內(nèi)核參數(shù)調(diào)優(yōu)vim?/etc/sysctl.confdev.flashcache.vdc+vdb.skip_seq_thresh_kb?=?1024?#超過1M的順序?qū)懼苯勇涞酱疟P上

Glusterfs 運維常用命令

????1、刪除卷?gluster?volume?stop?img?gluster?volume?delete?img?2、將機器移出集群?gluster?peer?detach?172.28.26.102?只允許172.28.0.0的網(wǎng)絡(luò)訪問glusterfs?gluster?volume?set?img?auth.allow?172.28.26.*?3、加入新的機器并添加到卷里(由于副本數(shù)設(shè)置為2,至少要添加2（4、6、8..）臺機器)?gluster?peer?probe?172.28.26.105?gluster?peer?probe?172.28.26.106?gluster?volume?add-brick?img?172.28.26.105:/data/gluster?172.28.26.106:/data/gluster#卷擴容后，需要執(zhí)行rebalancea、執(zhí)行rebalance前getfattr?-n?trusted.glusterfs.pathinfo?nova/???#?查看文件系統(tǒng)的擴展屬性，getfattr依賴attr?rpm包file:?nova/trusted.glusterfs.pathinfo="((<DISTRIBUTE:openstack-dht>?(<REPLICATE:openstack-replicate-0>?<POSIX(/mnt/gluster):gluster-1:/mnt/gluster/nova>?<POSIX(/mnt/gluster):gluster-2:/mnt/gluster/nova>)?(<REPLICATE:openstack-replicate-1>?<POSIX(/mnt/gluster):compute-2:/mnt/gluster/nova>?<POSIX(/mnt/gluster):compute-1:/mnt/gluster/nova>))?(openstack-dht-layout?(openstack-replicate-0?0?4294967295)?(openstack-replicate-1?0?0)))"??#?(openstack-replicate-0?0?4294967295)?hash值b、執(zhí)行rebalance后gluster?volume?rebalance?openstack?fix-layout?start???#??fix-layout不搬遷數(shù)據(jù)volume?rebalance:?openstack:?success:?Starting?rebalance?on?volume?openstack?has?been?successful.ID:?a7fc8c7f-a3e2-4783-b59e-3051327e52ecgetfattr?-n?trusted.glusterfs.pathinfo?nova/??????????#??再次查看文件系統(tǒng)的擴展屬性file:?nova/trusted.glusterfs.pathinfo="((<DISTRIBUTE:openstack-dht>?(<REPLICATE:openstack-replicate-0>?<POSIX(/mnt/gluster):gluster-1:/mnt/gluster/nova>?<POSIX(/mnt/gluster):gluster-2:/mnt/gluster/nova>)?(<REPLICATE:openstack-replicate-1>?<POSIX(/mnt/gluster):compute-2:/mnt/gluster/nova>?<POSIX(/mnt/gluster):compute-1:/mnt/gluster/nova>))?(openstack-dht-layout?(openstack-replicate-0?0?2147483646)?(openstack-replicate-1?2147483647?4294967295)))"??#?hash值重新分布了4、收縮卷?#收縮卷前gluster需要先移動數(shù)據(jù)到其他位置?gluster?volume?remove-brick?img?172.28.26.101:/data/gluster/img?172.28.26.102:/data/gluster/img?start?#查看遷移狀態(tài)?gluster?volume?remove-brick?img?172.28.26.101:/data/gluster/img?172.28.26.102:/data/gluster/img?status?#遷移完成后提交?gluster?volume?remove-brick?img?172.28.26.101:/data/gluster/img?172.28.26.102:/data/gluster/img?commit?遷移卷?#?將172.28.26.101的數(shù)據(jù)遷移到,先將172.28.26.107加入集群?gluster?peer?probe?172.28.26.107?gluster?volume?replace-brick?img?172.28.26.101:/data/gluster/img?172.28.26.107:/data/gluster/img?start?#?查看遷移狀態(tài)gluster?volume?replace-brick?img?172.28.26.101:/data/gluster/img?172.28.26.107:/data/gluster/img?status?#?數(shù)據(jù)遷移完畢后提交gluster?volume?replace-brick?img?172.28.26.101:/data/gluster/img?172.28.26.107:/data/gluster/img?commit?#?如果機器172.28.26.101出現(xiàn)故障已經(jīng)不能運行,執(zhí)行強制提交然后要求gluster馬上執(zhí)行一次同步?gluster?volume?replace-brick?img?172.28.26.101:/data/gluster/img?172.28.26.102:/data/gluster/img?commit?-force?gluster?volume?heal?imgs?full

Glusterfs內(nèi)核參數(shù)調(diào)優(yōu)?

????vm.dirty_background_ratio?=?5??#?5%的意思，根據(jù)實際內(nèi)存大小來定，數(shù)據(jù)從內(nèi)存flush到磁盤上，vm.dirty_ratio?=?20vm.dirty_writeback_centisecs?=?100??#?100%秒，數(shù)據(jù)flush到磁盤上vm.dirty_expire_centisecs?=?3000

Glusterfs TroubleShooting

1、錯誤：?GlusterFS:?{path}?or?a?prefix?of?it?is?already?part?of?a?volume???? 解決方法:?setfattr?-x?trusted.glusterfs.volume-id?$brick_pathsetfattr?-x?trusted.gfid?$brick_pathrm?-rf?$brick_path/.glusterfs2、錯誤：glusterd服務(wù)起不來 解決方法：less?/var/log/glusterfs/etc-glusterfs-glusterd.vol.log??#?查看這個日志文件,一般都是volume配置文件出錯cd?/var/lib/glusterd/volsrm?-rf?openstack/?ssd/service?glusterd?restart

Glusterfs小知識

1、glusterfs集群中的任意一個節(jié)點都可以拿到所有volume的配置文件，不一定是brick的volume的節(jié)點 2、掛載glusterfs?volume的時候，可以通過管理網(wǎng)掛載，只要這個ip可以到達glusterfs節(jié)點就行，它就可以拿到volume配置；glusterfs集群數(shù)據(jù)流量還是走存儲網(wǎng)。3、glusterfs:?客戶端進程名glusterfsd:?服務(wù)端進程名glusterd:?volume管理進程、I/O調(diào)度進程/var/log/glusterfs/[volume掛在點名字].log:?glusterfs客戶端log

參考鏈接?
劉愛貴glusterfs分布式文件系統(tǒng)研究
flashcache編譯安裝
QEMU-GlusterFS native integration

轉(zhuǎn)載于:https://blog.51cto.com/iceyao/1634839

總結(jié)

以上是生活随笔為你收集整理的Glusterfs入门的全部內(nèi)容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網(wǎng)站內(nèi)容還不錯，歡迎將生活随笔推薦給好友。