HA集群之三：corosync+pacemaker实现httpd服务的高可用
一、基礎概念
1、集群的組成結構
HA Cluster:
Messaging and Infrastructure Layer|Heartbeat Layer 集群信息事務層
Membership Layer 集群成員關系層
CCM 投票系統
Resource Allocation Layer 資源分配層
CRM，
DC：LRM,PE,TE,CIB
Other:LRM,CIB
Resource Layer 資源代理
RA
2、OpenAIS: 開放式應用接口規范
提供了一種集群模式，包含集群框架、集群成員管理、通信方式、集群監測，但沒有集群資源管理功能；
組件包括：AMF, CLM, CPKT, EVT等；分支不同，包含的組件略有區別；
分支：picacho, whitetank, wilson,
corosync (集群管理引擎)
是openais的一個子組件；
分裂成為兩個項目：
corosync, wilson(ais的接口標準)
CentOS 5:
cman + rgmanager (RHCS 系統自帶)
CentOS 6:
cman + rgmanager
corosync + pacemaker
3、命令行管理工具：
crmsh: suse, CentOS 6.4-自帶
pcs: RedHat, CentOS 6.5+自帶
二、案例：實現HA httpd
1、安裝corosync + pacemaker
注意：確定HA集群的前提：時間同步，集群節點基于hostname命令顯示的主機名通信，節點之間的root用戶能夠基于密鑰認證，考慮仲裁設備是否要使用
yum install corosync pacemaker -y
[root@BAIYU_175 ~]# rpm -ql corosync
/etc/corosync
/etc/corosync/corosync.conf.example
/etc/corosync/corosync.conf.example.udpu
/etc/corosync/service.d
/etc/corosync/uidgid.d
/etc/dbus-1/system.d/corosync-signals.conf
/etc/rc.d/init.d/corosync
/etc/rc.d/init.d/corosync-notifyd
/etc/sysconfig/corosync-notifyd
/usr/bin/corosync-blackbox
/usr/libexec/lcrso
/usr/libexec/lcrso/coroparse.lcrso
/usr/libexec/lcrso/objdb.lcrso
/usr/libexec/lcrso/quorum_testquorum.lcrso
/usr/libexec/lcrso/quorum_votequorum.lcrso
/usr/libexec/lcrso/service_cfg.lcrso
/usr/libexec/lcrso/service_confdb.lcrso
/usr/libexec/lcrso/service_cpg.lcrso
/usr/libexec/lcrso/service_evs.lcrso
/usr/libexec/lcrso/service_pload.lcrso
/usr/libexec/lcrso/vsf_quorum.lcrso
/usr/libexec/lcrso/vsf_ykd.lcrso
/usr/sbin/corosync
/usr/sbin/corosync-cfgtool
/usr/sbin/corosync-cpgtool
/usr/sbin/corosync-fplay
/usr/sbin/corosync-keygen
/usr/sbin/corosync-notifyd
/usr/sbin/corosync-objctl
/usr/sbin/corosync-pload
/usr/sbin/corosync-quorumtool
/usr/share/doc/corosync-1.4.7
/usr/share/doc/corosync-1.4.7/LICENSE
/usr/share/doc/corosync-1.4.7/SECURITY
/usr/share/man/man5/corosync.conf.5.gz
/usr/share/man/man8/confdb_keys.8.gz
/usr/share/man/man8/corosync-blackbox.8.gz
/usr/share/man/man8/corosync-cfgtool.8.gz
/usr/share/man/man8/corosync-cpgtool.8.gz
/usr/share/man/man8/corosync-fplay.8.gz
/usr/share/man/man8/corosync-keygen.8.gz
/usr/share/man/man8/corosync-notifyd.8.gz
/usr/share/man/man8/corosync-objctl.8.gz
/usr/share/man/man8/corosync-pload.8.gz
/usr/share/man/man8/corosync-quorumtool.8.gz
/usr/share/man/man8/corosync.8.gz
/usr/share/man/man8/corosync_overview.8.gz
/usr/share/snmp/mibs/COROSYNC-MIB.txt
/var/lib/corosync
/var/log/cluster
2、配置corosync
[root@BAIYU_173 ~]# cd /etc/corosync/
[root@BAIYU_173 corosync]# ls
corosync.conf.example corosync.conf.example.udpu service.d uidgid.d
[root@BAIYU_173 corosync]# cp corosync.conf.example corosync.conf
1）主配置文件/etc/corosync.conf詳解：
compatibility: whitetank #兼容08.以前的版本
secauth:off 不打開集群安全認證 推薦打開 on 使用 corosync-keygen 生成密鑰
threads: 0 定義多線程工作模式 0表示不使用線程而使用進程
ringnumber:0 環數目，類似ttl 默認即可
bindnetaddr: 192.168.1.0 多播地址監聽哪個網絡地址，填上自己的網絡地址即可：192.168.100.0
mcastaddr: 239.255.1.1 指定多播地址 225.25.25.25
mcastport: 5405 多播使用的端口utp
to_logfile: yes
to_syslog: yes 使用一個文件記錄地址即可 off
在/etc/crorosync/corosync.conf中添加以下內容：
amf {
mode: disabled
}
查看修改后的/etc/corosync/corosync.conf：
[root@BAIYU_173 corosync]# grep -v '#' corosync.conf|grep -v '^$'
compatibility: whitetank
totem {
version: 2
secauth: on
threads: 0
interface {
ringnumber: 0
bindnetaddr: 192.168.100.0
mcastaddr: 225.25.25.25
mcastport: 5405
ttl: 1
}
}
logging {
fileline: off
to_logfile: yes
logfile: /var/log/cluster/corosync.log
to_syslog: off
debug: off
timestamp: on
logger_subsys {
subsys: AMF
debug: off
}
}
amf {
mode: disabled
}
2）在/etc/corosync/service.d中新建pacemaker文件：
[root@BAIYU_173 service.d]# cat pacemaker
service {
name: pacemaker
ver: 1
}
[root@BAIYU_173 corosync]# corosync-keygen #生成用于集群驗證時的密鑰
Corosync Cluster Engine Authentication key generator.
Gathering 1024 bits for key from /dev/random.
Press keys on your keyboard to generate entropy.
Writing corosync key to /etc/corosync/authkey.
[root@BAIYU_173 corosync]# ls
authkey corosync.conf.example corosync.conf.orig uidgid.d
corosync.conf corosync.conf.example.udpu service.d
3）將密鑰和corosync的配置文件復制給集群其它節點：
[root@BAIYU_173 corosync]# scp -p authkey corosync.conf 192.168.100.175:/etc/corosync
3、啟動corosync并驗證:
[root@BAIYU_173 corosync]# service corosync start #要先啟動pacemaker依賴于corosync所以要先啟動corosync
Starting Corosync Cluster Engine (corosync): [確定]
[root@BAIYU_173 corosync]# netstat -nlptu
Active Internet connections (only servers)
[root@BAIYU_173 ~]# service pacemaker start #如果啟動失敗，先重啟corosync再重啟pacemaker
Starting Pacemaker Cluster Manager[確定]
udp 0 0 0.0.0.0:53243 0.0.0.0:* 1372/rpc.statd
[root@BAIYU_173 ~]# netstat -nlptu
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN 1348/rpcbind
tcp 0 0 0.0.0.0:57269 0.0.0.0:* LISTEN 1372/rpc.statd
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1551/sshd
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 1642/master
udp 0 0 192.168.100.173:5404 0.0.0.0:* 2330/corosync
udp 0 0 192.168.100.173:5405 0.0.0.0:* 2330/corosync
udp 0 0 225.25.25.25:5405 0.0.0.0:* 2330/corosync
udp 0 0 0.0.0.0:671 0.0.0.0:* 1348/rpcbind
udp 0 0 127.0.0.1:703 0.0.0.0:* 1372/rpc.statd
udp 0 0 0.0.0.0:111 0.0.0.0:* 1348/rpcbind
udp 0 0 0.0.0.0:53243 0.0.0.0:* 1372/rpc.statd
1）驗證corosync引擎是否正常啟動：
[root@BAIYU_173 corosync]# grep -e 'Corosync Cluster Engine' -e 'configuration file' /
var/log/cluster/corosync.log
Oct 24 17:55:00 corosync [MAIN ] Corosync Cluster Engine ('1.4.7'): started and ready to provide service.
Oct 24 17:55:00 corosync [MAIN ] Successfully read main configuration file '/etc/corosync/corosync.conf'.
2）驗證初始化成員節點通知是否正常發出：
[root@BAIYU_173 corosync]# grep TOTEM /var/log/cluster/corosync.log
Oct 24 17:55:00 corosync [TOTEM ] Initializing transport (UDP/IP Multicast).
Oct 24 17:55:00 corosync [TOTEM ] Initializing transmit/receive security: libtomcrypt SOBER128/SHA1HMAC (mode 0).
Oct 24 17:55:00 corosync [TOTEM ] The network interface [192.168.100.173] is now up.
Oct 24 17:55:00 corosync [TOTEM ] Process pause detected for 613 ms, flushing membership messages.
Oct 24 17:55:00 corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
Oct 24 17:55:23 corosync [TOTEM ] A processor failed, forming new configuration.
Oct 24 17:55:23 corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
Oct 24 17:55:27 corosync [TOTEM ] A processor failed, forming new configuration.
Oct 24 17:55:27 corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
3）檢查啟動過程中是否有錯誤產生：
下面的錯誤信息表示pacemaker不久之后講不再作為corosync的插件運行，因此，建議使用cman作為集群基礎架構服務，此處可安全忽略
[root@BAIYU_173 corosync]# grep ERROR: /var/log/cluster/corosync.log | grep -v unpack_
resources
Oct 24 17:55:00 corosync [pcmk ] ERROR: process_ais_conf: You have configured a cluster using the Pacemaker plugin for Corosync. The plugin is not supported in this environment and will be removed very soon.
Oct 24 17:55:00 corosync [pcmk ] ERROR: process_ais_conf: Please see Chapter 8 of 'Clusters from Scratch' (http://www.clusterlabs.org/doc) for details on using Pacemaker with CMAN
Oct 24 17:55:02 corosync [pcmk ] ERROR: pcmk_wait_dispatch: Child process mgmtd exited (pid=25335, rc=100) # 可以忽略，
3）查看pacemaker是否正常啟動：
[root@BAIYU_173 corosync]# grep pcmk_startup /var/log/cluster/corosync.log
Oct 24 17:55:00 corosync [pcmk ] info: pcmk_startup: CRM: Initialized
Oct 24 17:55:00 corosync [pcmk ] Logging: Initialized pcmk_startup
Oct 24 17:55:00 corosync [pcmk ] info: pcmk_startup: Maximum core file size is: 18446744073709551615
Oct 24 17:55:00 corosync [pcmk ] info: pcmk_startup: Service: 9
Oct 24 17:55:00 corosync [pcmk ] info: pcmk_startup: Local hostname: BAIYU_173
4、安裝crm_sh:
定義資源的命令行
可以只在一個節點上安裝crm_sh，配置結果會送給DC，然后DC同步給其它節點，DC在集群系統啟動時集群系統自動選舉出
[root@BAIYU_173 ~]# cd /etc/yum.repos.d/
[root@BAIYU_173 ~]# wget http://download.opensuse.org/repositories/network:/ha-clustering:/Stable/CentOS_CentOS-6/network:ha-clustering:Stable.repo
[root@BAIYU_173 ~]# yum install crmsh
[root@BAIYU_173 ~]# crm
crm(live)# status # 查看集群系統狀態；如果這里報錯很可能是pacemaker停止了或前面配置有誤
Last updated: Sun Oct 25 22:41:42 2015
Last change: Sun Oct 25 22:11:26 2015
Stack: classic openais (with plugin)
Current DC: BAIYU_175 - partition with quorum
Version: 1.1.11-97629de
2 Nodes configured, 2 expected votes
0 Resources configured
Online: [ BAIYU_173 BAIYU_175 ]
crm(live)# configure
crm(live)configure# show # 查看集群系統信息
node BAIYU_173
node BAIYU_175
property cib-bootstrap-options: \
dc-version=1.1.11-97629de \
cluster-infrastructure="classic openais (with plugin)" \
expected-quorum-votes=2
crm(live)configure#
5、crm使用詳解
crm的常用一級子命令：
cib manage shadow CIBs #cib沙盒
resource resources management #管理資源 ，所有的資源的狀態都在這個子命令后定義
configure CRM cluster configuration #編輯集群配置信息，所有資源的定義
node nodes management #集群節點管理子命令
options user preferences #用戶優先級
history CRM cluster history#
site Geo-cluster support
ra resource agents information center #資源代理子命令（所有與資源代理相關的程都在此命令之下）
status show cluster status #顯示當前集群的狀態信息
help,? show help (help topics for list of topics)#查看當前區域可能的命令
end,cd,up go back one level #返回第一級crm(live)#
quit,bye,exit exit the program #退出crm（live）交互模式
configure常用的子命令：
所有資源的定義都是在此子命令下完成的
node define a cluster node #定義一個集群節點
primitive define a resource #定義資源
monitor add monitor operation to a primitive #對一個資源添加監控選項（如超時時間，啟動失敗后的操作）
group define a group #定義一個組類型（將多個資源整合在一起）
clone define a clone #定義一個克隆類型（可以設置總的克隆數，每一個節點上可以運行幾個克隆）
ms define a master-slave resource #定義一個主從類型（集群內的節點只能有一個運行主資源，其它從的做備用）
rsc_template define a resource template #定義一個資源模板
location a location preference #定義位置約束優先級（默認運行于那一個節點（如果位置約束的值相同，默認傾向性那一個高，就在那一個節點上運行））
colocation colocate resources #排列約束資源（多個資源在一起的可能性）
order order resources #資源的啟動的先后順序
rsc_ticket resources ticket dependency
property set a cluster property #設置集群全局屬性
rsc_defaults set resource defaults #設置資源默認屬性（粘性）
fencing_topology node fencing order #隔離節點順序
role define role access rights #定義角色的訪問權限
user define user access rights #定義用用戶訪問權限
op_defaults set resource operations defaults #設置資源默認選項
schema set or display current CIB RNG schema
show display CIB objects #顯示集群信息庫對
edit edit CIB objects #編輯集群信息庫對象（vim模式下編輯）
filter filter CIB objects #過濾CIB對象
delete delete CIB objects #刪除CIB對象
default-timeouts set timeouts for operations to minimums from the meta-data
rename rename a CIB object #重命名CIB對象
modgroup modify group #改變資源組
refresh refresh from CIB #重新讀取CIB信息
erase erase the CIB #清除CIB信息
ptest show cluster actions if changes were committed
rsctest test resources as currently configured
cib CIB shadow management
cibstatus CIB status management and editing
template edit and import a configuration from a template
commit commit the changes to the CIB #將更改后的信息提交寫入CIB
verify verify the CIB with crm_verify #CIB語法驗證
upgrade upgrade the CIB to version 1.0
save save the CIB to a file #將當前CIB導出到一個文件中（導出的文件存于切換crm 之前的目錄）
load import the CIB from a file #從文件內容載入CIB
graph generate a directed graph
xml raw xml
help show help (help topics for list of topics) #顯示幫助信息
end go back one level #回到第一級(crm(live)#)
quit exit the program #退出crm交互模式
常用的全局屬性配置：
configure
property
stonith-eanabled=true|false true 默認 “沒有報頭設備時使用”
no-quorum-policy=stopped 停止 默認 “簡單的2節點集群時使用”
ignore 忽略 仍然運行
freeze 凍結 舊連接可以繼續訪問，新的連接被拒絕
suicide 殉情
default-resource-stickiness=#|-#|inf|-inf “資源對當前節點的粘性”
symetric-cluster=true|false 是否是對稱集群 true：rsc可以運行于集群中的任意節點
resource子命令
所有的資源狀態都此處控制
status show status of resources?#顯示資源狀態信息
start start a resource?#啟動一個資源
stop stop a resource?#停止一個資源
restart restart a resource?#重啟一個資源
promote promote a master-slave resource?#提升一個主從資源
demote demote a master-slave resource?#降級一個主從資源
manage put a resource into managed mode
unmanage put a resource into unmanaged mode
migrate migrate a resource to another node?#將資源遷移到另一個節點上
unmigrate unmigrate a resource to another node
param manage a parameter of a resource?#管理資源的參數
secret manage sensitive parameters?#管理敏感參數
meta manage a meta attribute?#管理源屬性
utilization manage a utilization attribute
failcount manage failcounts?#管理失效計數器
cleanup cleanup resource status?#清理資源狀態
refresh refresh CIB from the LRM status?#從LRM（LRM本地資源管理）更新CIB（集群信息庫），在
reprobe probe?for?resources not started by the CRM?#探測在CRM中沒有啟動的資源
trace start RA tracing?#啟用資源代理（RA）追蹤
untrace stop RA tracing?#禁用資源代理（RA）追蹤
help show help (help topics?for?list of topics)?#顯示幫助
end go back one level?#返回一級（crm(live)#）
quit?exit?the program?#退出交互式程序
node子命令
節點管理和狀態命令
status show nodes status as XML #以xml格式顯示節點狀態信息
show show node #命令行格式顯示節點狀態信息
standby put node into standby #模擬指定節點離線（standby在后面必須的FQDN）
online set node online # 節點重新上線
maintenance put node into maintenance mode
ready put node into ready mode
fence fence node #隔離節點
clearstate Clear node state #清理節點狀態信息
delete delete node #刪除 一個節點
attribute manage attributes
utilization manage utilization attributes
status-attr manage status attributes
help show help (help topics for list of topics)
end go back one level
quit exit the program
ra子命令
資源代理類別都在此處
classes list classes and providers #為資源代理分類
list list RA for a class (and provider)#顯示一個類別中的提供的資源
meta show meta data for a RA #顯示一個資源代理序的可用參數（如meta ocf:heartbeat:IPaddr2）
providers show providers for a RA and a class
help show help (help topics for list of topics)
end go back one level
quit exit the program
6、定義資源
webip: 192.168.100.100
那在這里，我們如何去配置一個資源呢，雖然它跟heartbeat略有區別，但是概念基本上是一樣的，下面我們就來配置一個web資源吧！
1）配置兩點的corosync+pacemaker集群，設置兩個全局屬性：
stonith-enabled=false
no-quorum-policy=ignore
由于我們的corosync默認是啟用stonith功能的，但是我們這里沒有stonith設備，如果我們直接去配置資源的話，由于沒有stonith設備，所以資源的切換并不會完成，所以要禁用stonith功能，但禁用stonoith需要我們去配置集群的全局stonith屬性，全局屬性是對所有的節點都生效；
要注意：2節點的集群，如果一個節點掛了，就不擁有法定票數了，那資源是不會切換的，資源默認會停止
集群的策略有幾種：
stopped ：停止服務 （默認）?
ignore ：忽略，繼續運行?
freeze ：凍結，已經連接的請求繼續響應，新的請求不再響應?
suicide ：自殺，將服務kill掉
a、禁用stonith-enable
[root@BAIYU_173 ~]# crm configure #進入crm命令行模式配置資源等
crm(live)configure# property #切換到property目錄下，可以用兩次tab鍵進行補全和查看
crm(live)configure# help property #不知道用法就help
Set a cluster property
Set the cluster (crm_config) options.
Usage:
property [$id=<set_id>] [rule ...] <option>=<value> [<option>=<value> ...]
Example:
property stonith-enabled=true
property rule date spec years=2014 stonith-enabled=false
crm(live)configure# property stonith-enabled=false #禁用stonith-enabled
crm(live)configure# verify #檢查設置的屬性是否正確
crm(live)configure# commit #檢查沒問題就可以提交了
crm(live)configure# show #查看當前集群的所有配置信息
node BAIYU_173
node BAIYU_175 #兩個節點
primitive webip IPaddr \
params ip=192.168.100.100 nic=eth0 cidr_netmask=24
property cib-bootstrap-options: \
dc-version=1.1.11-97629de \ #DC的版本號
cluster-infrastructure="classic openais (with plugin)" \ #集群的基礎架構，使用的是OpenAIS，插件式的
expected-quorum-votes=2 \ #期望節點的票數
stonith-enabled=false #禁用stonith功能
b、忽略投票規則
crm(live)configure# property no-quorum-policy=ignore
crm(live)configure# verify
crm(live)configure# commit
crm(live)configure# show
node BAIYU_173 \
attributes standby=off
node BAIYU_175
primitive webip IPaddr \
params ip=192.168.100.100 nic=eth0 cidr_netmask=24
property cib-bootstrap-options: \
dc-version=1.1.11-97629de \
cluster-infrastructure="classic openais (with plugin)" \
expected-quorum-votes=2 \
stonith-enabled=false \
no-quorum-policy=ignore
primitive 語法：
primitive <rsc_id> class:provider:ra params param1=value1 param2=value2 op op1 param1=value op op2 parma1=value1
c、添加資源
crm(live)configure# primitive webip ocf:heartbeat:IPaddr params ip=192.168.100.100 nic=eth0 cidr_netmask=24
crm(live)configure# verify # 檢查語法是否有錯
crm(live)configure# commit # 提交給內核
Online: [ BAIYU_173 BAIYU_175 ]
crm(live)# status
Last updated: Mon Oct 26 16:24:53 2015
Last change: Mon Oct 26 16:24:21 2015
Stack: classic openais (with plugin)
Current DC: BAIYU_175 - partition with quorum
Version: 1.1.11-97629de
2 Nodes configured, 2 expected votes
1 Resources configured
Online: [ BAIYU_173 BAIYU_175 ]
webip (ocf::heartbeat:IPaddr): Started BAIYU_173 #可以看到webip 已經在BAIYU_173上運行了
crm(live)#
[root@BAIYU_173 ~]# ip addr # 注意configure查看不到
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:6a:dc:8d brd ff:ff:ff:ff:ff:ff
inet 192.168.100.173/24 brd 192.168.100.255 scope global eth0
inet 192.168.100.100/24 brd 192.168.100.255 scope global secondary eth0 #webip
crm(live)configure# primitive webserver lsb:httpd
crm(live)configure# verify
crm(live)configure# commit
crm(live)configure# cd
crm(live)# status
Last updated: Mon Oct 26 17:52:46 2015
Last change: Mon Oct 26 17:52:34 2015
Stack: classic openais (with plugin)
Current DC: BAIYU_175 - partition with quorum
Version: 1.1.11-97629de
2 Nodes configured, 2 expected votes
2 Resources configured
Online: [ BAIYU_173 BAIYU_175 ]
webip (ocf::heartbeat:IPaddr): Started BAIYU_175
webserver (lsb:httpd): Started BAIYU_173
上面發現2個資源默認是分散在系統節點中
把資源定義在同1個節點有2種方法：定義組或位置約束：
1）新建組，把資源按啟動順序先后加入組
crm(live)configure# group webservice webip webserver #定義組,組中資源在同一個節點并 按順序啟動
crm(live)configure# cd
crm(live)# status
Last updated: Mon Oct 26 18:09:16 2015
Last change: Mon Oct 26 18:07:19 2015
Stack: classic openais (with plugin)
Current DC: BAIYU_175 - partition with quorum
Version: 1.1.11-97629de
2 Nodes configured, 2 expected votes
2 Resources configured
Online: [ BAIYU_173 BAIYU_175 ]
Resource Group: webservice
webip (ocf::heartbeat:IPaddr): Started BAIYU_173
webserver (lsb:httpd): Started BAIYU_173
crm(live)configure# show
node BAIYU_173 \
attributes standby=off
node BAIYU_175 \
attributes standby=off
primitive webip IPaddr \
params ip=192.168.100.100 nic=eth0 cidr_netmask=24
primitive webserver lsb:httpd
group webservice webip webserver
property cib-bootstrap-options: \
dc-version=1.1.11-97629de \
cluster-infrastructure="classic openais (with plugin)" \
expected-quorum-votes=2 \
stonith-enabled=false \
no-quorum-policy=ignore
crm(live)configure# cd ../resource
crm(live)resource# status webservice # 資源在運行時不能刪除，要先停止
resource webservice is running on: BAIYU_173
crm(live)resource# stop webservice
crm(live)resource# status webservice
resource webservice is NOT running
crm(live)resource# cd ../configure
crm(live)configure# delete webservice #刪除組
crm(live)configure# verify
crm(live)configure# commit
crm(live)configure# show
node BAIYU_173 \
attributes standby=off
node BAIYU_175 \
attributes standby=off
primitive webip IPaddr \
params ip=192.168.100.100 nic=eth0 cidr_netmask=24
primitive webserver lsb:httpd
property cib-bootstrap-options: \
dc-version=1.1.11-97629de \
cluster-infrastructure="classic openais (with plugin)" \
expected-quorum-votes=2 \
stonith-enabled=false \
no-quorum-policy=ignore
crm(live)# status
Last updated: Mon Oct 26 18:47:24 2015
Last change: Mon Oct 26 18:46:41 2015
Stack: classic openais (with plugin)
Current DC: BAIYU_175 - partition with quorum
Version: 1.1.11-97629de
2 Nodes configured, 2 expected votes
2 Resources configured
Online: [ BAIYU_173 BAIYU_175 ] #資源又分散在節點中
webip (ocf::heartbeat:IPaddr): Started BAIYU_173
webserver (lsb:httpd): Started BAIYU_175
2）用colocation,ored定義資源排列約束，和順序約束：
crm(live)configure# colocation webserver_with_webip inf: webserver webip
crm(live)configure# verify
crm(live)configure# commit
crm(live)configure# crm status
ERROR: configure.crm: No such command
crm(live)configure# order webip_before_webserver Mandatory: webip webserver
crm(live)configure# verify
crm(live)configure# commit
crm(live)configure# show
node BAIYU_173 \
attributes standby=off
node BAIYU_175 \
attributes standby=off
primitive webip IPaddr \
params ip=192.168.100.100 nic=eth0 cidr_netmask=24
primitive webserver lsb:httpd
colocation webserver_with_webip inf: webserver webip
order webip_before_webserver Mandatory: webip webserver
property cib-bootstrap-options: \
dc-version=1.1.11-97629de \
cluster-infrastructure="classic openais (with plugin)" \
expected-quorum-votes=2 \
stonith-enabled=false \
no-quorum-policy=ignore
crm(live)configure# cd
crm(live)# status
Last updated: Mon Oct 26 18:58:09 2015
Last change: Mon Oct 26 18:57:56 2015
Stack: classic openais (with plugin)
Current DC: BAIYU_175 - partition with quorum
Version: 1.1.11-97629de
2 Nodes configured, 2 expected votes
2 Resources configured
Online: [ BAIYU_173 BAIYU_175 ]
webip (ocf::heartbeat:IPaddr): Started BAIYU_173
webserver (lsb:httpd): Started BAIYU_173
定義位置約束:
crm(live)configure# location webip_on_BAIYU_175 webip rule 50: #uname eq BA
IYU_175
crm(live)configure# show
node BAIYU_173 \
attributes standby=off
node BAIYU_175 \
attributes standby=off
primitive webip IPaddr \
params ip=192.168.100.100 nic=eth0 cidr_netmask=24
primitive webserver lsb:httpd
location webip_on_BAIYU_175 webip \
rule 50: #uname eq BAIYU_175
colocation webserver_with_webip inf: webserver webip
order webip_before_webserver Mandatory: webip webserver
property cib-bootstrap-options: \
dc-version=1.1.11-97629de \
cluster-infrastructure="classic openais (with plugin)" \
expected-quorum-votes=2 \
stonith-enabled=false \
no-quorum-policy=ignore
crm(live)configure# verify
crm(live)configure# commit
crm(live)# status
Last updated: Mon Oct 26 19:27:28 2015
Last change: Mon Oct 26 19:27:10 2015
Stack: classic openais (with plugin)
Current DC: BAIYU_175 - partition with quorum
Version: 1.1.11-97629de
2 Nodes configured, 2 expected votes
2 Resources configured
Online: [ BAIYU_173 BAIYU_175 ]
webip (ocf::heartbeat:IPaddr): Started BAIYU_175
webserver (lsb:httpd): Started BAIYU_175
定義資源對當前節點的粘性：
crm(live)configure# property default-resource-stickiness=50
crm(live)configure# verify
crm(live)configure# commit
注意： 即使當資源在BAIYU_173時，當前有2個資源，粘性：50+50>location:50 資源也不會自動轉到BAIYU_173
因為系統默認只監控節點，當資源被意外關閉時，系統不知道，不會做資源轉移，所以我們要配置系統監控資源：
先把之前定義的資源刪除：
crm(live)# cd ../resource
crm(live)resource# stop webip
crm(live)resource# stop webserver
crm(live)resource# cd ../configure
crm(live)configure# edit
crm(live)configure# edit #只留定義的集群全局屬性
1 node BAIYU_173 \
2 attributes standby=off
3 node BAIYU_175 \
4 attributes standby=off
5 property cib-bootstrap-options: \
6 dc-version=1.1.11-97629de \
7 cluster-infrastructure="classic openais (with plugin)" \
8 expected-quorum-votes=2 \
9 stonith-enabled=false \
10 no-quorum-policy=ignore \
11 default-resource-stickiness=50
12 #vim:set syntax=pcmk
crm(live)configure# verify
crm(live)configure# commit
crm(live)configure# show
node BAIYU_173 \
attributes standby=off
node BAIYU_175 \
attributes standby=off
property cib-bootstrap-options: \
dc-version=1.1.11-97629de \
cluster-infrastructure="classic openais (with plugin)" \
expected-quorum-votes=2 \
stonith-enabled=false \
no-quorum-policy=ignore \
default-resource-stickiness=50
重新定義加入監控選項：
crm(live)configure# primitive webip ocf:heartbeat:IPaddr params ip=192.168.
100.110 nic=eth0 cidr_netmask=24 op monitor interval=10s timeout=20s # op表示加入動作，monitor表示監控 interval 間隔時間，timeout 超時時間
crm(live)configure# verify
crm(live)configure# commit
crm(live)configure# primitive webserver lsb:httpd op monitor interval=10s t
imout=20s
crm(live)configure# group webservice webip webserver
crm(live)configure# verify
crm(live)configure# commit
crm(live)configure# show
node BAIYU_173 \
attributes standby=off
node BAIYU_175 \
attributes standby=off
primitive webip IPaddr \
params ip=192.168.100.110 nic=eth0 cidr_netmask=24 \
op monitor interval=10s timeout=20s
primitive webserver lsb:httpd \
op monitor interval=10s timout=20s
group webservice webip webserver
property cib-bootstrap-options: \
dc-version=1.1.11-97629de \
cluster-infrastructure="classic openais (with plugin)" \
expected-quorum-votes=2 \
stonith-enabled=false \
no-quorum-policy=ignore \
default-resource-stickiness=50
[root@BAIYU_173 ~]# crm
crm(live)# status
Last updated: Mon Oct 26 20:04:41 2015
Last change: Mon Oct 26 20:02:24 2015
Stack: classic openais (with plugin)
Current DC: BAIYU_175 - partition with quorum
Version: 1.1.11-97629de
2 Nodes configured, 2 expected votes
2 Resources configured
Online: [ BAIYU_173 BAIYU_175 ]
Resource Group: webservice
webip (ocf::heartbeat:IPaddr): Started BAIYU_173
webserver (lsb:httpd): Started BAIYU_173
crm(live)#
驗證：
[root@BAIYU_173 ~]# netstat -nlpt|grep 80
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 7186/httpd
[root@BAIYU_173 ~]# killall httpd
[root@BAIYU_173 ~]# netstat -nlpt|grep 80
[root@BAIYU_173 ~]# netstat -nlpt|grep 80
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 8906/httpd
注意：如果資源啟動不了，資源會轉移到其它節點
添加NFS:
[root@BAIYU_173 ~]# crm
crm(live)# resource
crm(live)resource# stop webservice
crm(live)resource# show
Resource Group: webservice
webip (ocf::heartbeat:IPaddr): Stopped
webserver (lsb:httpd): Stopped
crm(live)# ra info ocf:heartbeat:Filesystem #查看RA中對該類型資源的一些默認設置和必須要帶參數
crm(live)configure# primitive webstore ocf:heartbeat:Filesystem params devi
ce="192.168.100.10:/data/myweb" directory="/var/www/html" fstype="nfs" op m
onitor interval=20s timeout=40s op start timeout=60s op stop timeout=60s
crm(live)configure# edit 手動把webstore添加進之前定義的組
crm(live)configure# verify
crm(live)configure# commit
crm(live)configure# show
node BAIYU_173 \
attributes standby=off
node BAIYU_175 \
attributes standby=off
primitive webip IPaddr \
params ip=192.168.100.110 nic=eth0 cidr_netmask=24 \
op monitor interval=10s timeout=20s
primitive webserver lsb:httpd \
op monitor interval=10s timout=20s
primitive webstore Filesystem \
params device="192.168.100.10:/data/myweb" directory="/var/www/html" fstype=nfs \
op monitor interval=20s timeout=40s \
op start timeout=60s interval=0 \
op stop timeout=60s interval=0
group webservice webip webserver webstore \
meta target-role=Stopped
property cib-bootstrap-options: \
dc-version=1.1.11-97629de \
cluster-infrastructure="classic openais (with plugin)" \
expected-quorum-votes=2 \
crm(live)# resource start webservice
crm(live)# resource show
Resource Group: webservice
webip (ocf::heartbeat:IPaddr): Started
webserver (lsb:httpd): Started
webstore (ocf::heartbeat:Filesystem): Started
crm(live)# status
Last updated: Mon Oct 26 20:42:32 2015
Last change: Mon Oct 26 20:42:00 2015
Stack: classic openais (with plugin)
Current DC: BAIYU_175 - partition with quorum
Version: 1.1.11-97629de
2 Nodes configured, 2 expected votes
3 Resources configured
Online: [ BAIYU_173 BAIYU_175 ]
Resource Group: webservice
webip (ocf::heartbeat:IPaddr): Started BAIYU_173
webserver (lsb:httpd): Started BAIYU_173
webstore (ocf::heartbeat:Filesystem): Started BAIYU_173
Failed actions:
webserver_monitor_10000 on BAIYU_173 'not running' (7): call=40, status=complete, last-rc-change='Mon Oct 26 20:07:46 2015', queued=0ms, exec=0ms
清理之前的狀態報錯信息：
crm(live)# resource cleanup webservice #清空相關資源狀態的報錯信息
Cleaning up webip on BAIYU_173
Cleaning up webip on BAIYU_175
Cleaning up webserver on BAIYU_173
Cleaning up webserver on BAIYU_175
Cleaning up webstore on BAIYU_173
Cleaning up webstore on BAIYU_175
Waiting for 6 replies from the CRMd...... OK
將BAIYU_173備用，資源轉移到_175
crm(live)# node standby
crm(live)# status
Last updated: Mon Oct 26 20:53:39 2015
Last change: Mon Oct 26 20:53:35 2015
Stack: classic openais (with plugin)
Current DC: BAIYU_175 - partition with quorum
Version: 1.1.11-97629de
2 Nodes configured, 2 expected votes
3 Resources configured
Node BAIYU_173: standby
Online: [ BAIYU_175 ]
Resource Group: webservice
webip (ocf::heartbeat:IPaddr): Started BAIYU_175
webserver (lsb:httpd): Started BAIYU_175
webstore (ocf::heartbeat:Filesystem): Started BAIYU_175
此時完整的HA httpd配置完成。
轉載于:https://blog.51cto.com/hobowizard/1884305
創作挑戰賽新人創作獎勵來咯，堅持創作打卡瓜分現金大獎總結
以上是生活随笔為你收集整理的HA集群之三：corosync+pacemaker实现httpd服务的高可用的全部內容，希望文章能夠幫你解決所遇到的問題。
- 上一篇： Laravel 在Centos上的安装
- 下一篇： 给 axios 和 redux-axio