Linux 高可用（HA）集群之keepalived
一、keepalived介紹
1、Keepalived 定義
? ?Keepalived 是一個基于VRRP協議來實現的LVS服務高可用方案，可以利用其來避免單點故障。一個LVS服務會有2臺服務器運行Keepalived，一臺為主服務器（MASTER），一臺為備份服務器（BACKUP），但是對外表現為一個虛擬IP，主服務器會發送特定的消息給備份服務器，當備份服務器收不到這個消息的時候，即主服務器宕機的時候， 備份服務器就會接管虛擬IP，繼續提供服務，從而保證了高可用性。Keepalived是VRRP的完美實現，因此在介紹keepalived之前，先介紹一下VRRP的原理。
2、VRRP 協議簡介
? ?在現實的網絡環境中，兩臺需要通信的主機大多數情況下并沒有直接的物理連接。對于這樣的情況，它們之間路由怎樣選擇？主機如何選定到達目的主機的下一跳路由，這個問題通常的解決方法有二種：
在主機上使用動態路由協議(RIP、OSPF等)?
在主機上配置靜態路由?
? ?很明顯，在主機上配置動態路由是非常不切實際的，因為管理、維護成本以及是否支持等諸多問題。配置靜態路由就變得十分流行，但路由器(或者說默認網關default gateway)卻經常成為單點故障。VRRP的目的就是為了解決靜態路由單點故障問題，VRRP通過一競選(election)協議來動態的將路由任務交給LAN中虛擬路由器中的某臺VRRP路由器。
3、VRRP 工作機制
? ?在一個VRRP虛擬路由器中，有多臺物理的VRRP路由器，但是這多臺的物理的機器并不能同時工作，而是由一臺稱為MASTER的負責路由工作，其它的都是BACKUP，MASTER并非一成不變，VRRP讓每個VRRP路由器參與競選，最終獲勝的就是MASTER。MASTER擁有一些特權，比如，擁有虛擬路由器的IP地址，我們的主機就是用這個IP地址作為靜態路由的。擁有特權的MASTER要負責轉發發送給網關地址的包和響應ARP請求。
? ?VRRP通過競選協議來實現虛擬路由器的功能，所有的協議報文都是通過IP多播(multicast)包(多播地址224.0.0.18)形式發送的。虛擬路由器由VRID(范圍0-255)和一組IP地址組成，對外表現為一個周知的MAC地址。所以，在一個虛擬路由 器中，不管誰是MASTER，對外都是相同的MAC和IP(稱之為VIP)。客戶端主機并不需要因為MASTER的改變而修改自己的路由配置，對客戶端來說，這種主從的切換是透明的。
? ?在一個虛擬路由器中，只有作為MASTER的VRRP路由器會一直發送VRRP通告信息(VRRPAdvertisement message)，BACKUP不會搶占MASTER，除非它的優先級(priority)更高。當MASTER不可用時(BACKUP收不到通告信息)， 多臺BACKUP中優先級最高的這臺會被搶占為MASTER。這種搶占是非常快速的(<1s)，以保證服務的連續性。由于安全性考慮，VRRP包使用了加密協議進行加密。
4、VRRP 工作流程
(1).初始化： ? ?
? ?路由器啟動時，如果路由器的優先級是255(最高優先級，路由器擁有路由器地址)，要發送VRRP通告信息，并發送廣播ARP信息通告路由器IP地址對應的MAC地址為路由虛擬MAC，設置通告信息定時器準備定時發送VRRP通告信息，轉為MASTER狀態；否則進入BACKUP狀態，設置定時器檢查定時檢查是否收到MASTER的通告信息。
(2).Master
設置定時通告定時器；?
用VRRP虛擬MAC地址響應路由器IP地址的ARP請求；?
轉發目的MAC是VRRP虛擬MAC的數據包；?
如果是虛擬路由器IP的擁有者，將接受目的地址是虛擬路由器IP的數據包，否則丟棄；?
當收到shutdown的事件時刪除定時通告定時器，發送優先權級為0的通告包，轉初始化狀態；?
如果定時通告定時器超時時，發送VRRP通告信息；?
收到VRRP通告信息時，如果優先權為0，發送VRRP通告信息；否則判斷數據的優先級是否高于本機，或相等而且實際IP地址大于本地實際IP，設置定時通告定時器，復位主機超時定時器，轉BACKUP狀態；否則的話，丟棄該通告包；?
(3).Backup
設置主機超時定時器；?
不能響應針對虛擬路由器IP的ARP請求信息；?
丟棄所有目的MAC地址是虛擬路由器MAC地址的數據包；?
不接受目的是虛擬路由器IP的所有數據包；?
當收到shutdown的事件時刪除主機超時定時器，轉初始化狀態；?
主機超時定時器超時的時候，發送VRRP通告信息，廣播ARP地址信息，轉MASTER狀態；?
收到VRRP通告信息時，如果優先權為0，表示進入MASTER選舉；否則判斷數據的優先級是否高于本機，如果高的話承認MASTER有效，復位主機超時定時器；否則的話，丟棄該通告包；?
5、ARP查詢處理
? ?當內部主機通過ARP查詢虛擬路由器IP地址對應的MAC地址時，MASTER路由器回復的MAC地址為虛擬的VRRP的MAC地址，而不是實際網卡的 MAC地址，這樣在路由器切換時讓內網機器覺察不到；而在路由器重新啟動時，不能主動發送本機網卡的實際MAC地址。如果虛擬路由器開啟的ARP代理 (proxy_arp)功能，代理的ARP回應也回應VRRP虛擬MAC地址；好了VRRP的簡單講解就到這里，我們下來講解一下Keepalived的案例。
6、keepalived組成
? ?keepalived主要有三個模塊，分別是core、check和vrrp。core模塊為keepalived的核心，負責主進程的啟動、維護以及全局配置文件的加載和解析。check負責健康檢查，包括常見的各種檢查方式。vrrp模塊是來實現VRRP協議的。
二、keepalived的配置文件說明
? ?keepalived只有一個配置文件keepalived.conf，里面主要包括以下幾個配置區域，分別是global_defs、static_ipaddress、static_routes、vrrp_script、vrrp_instance和virtual_server。
1、global_defs區域
主要是配置故障發生時的通知對象以及機器標識。
global_defs?{notification_email?{acassen@firewall.locfailover@firewall.locsysadmin@firewall.loc}notification_email_from?Alexandre.Cassen@firewall.locsmtp_server?192.168.200.1smtp_connect_timeout?30enable_trapsrouter_id?LVS_DEVEL }notification_email 故障發生時給誰發郵件通知。?
notification_email_from 通知郵件從哪個地址發出。?
smpt_server 通知郵件的smtp地址。?
smtp_connect_timeout 連接smtp服務器的超時時間。?
enable_traps 開啟SNMP陷阱（Simple Network Management Protocol）。?
router_id 標識本節點的字條串，通常為hostname，但不一定非得是hostname。故障發生時，郵件通知會用到。
2、vrrp_script區域
用來做健康檢查的，當時檢查失敗時會將vrrp_instance的priority減少相應的值。
vrrp_script?chk_http_port?{script?"</dev/tcp/127.0.0.1/80"interval?1weight?-10 }以上意思是如果script中的指令執行失敗，那么相應的vrrp_instance的優先級會減少10個點。
3、vrrp_instance和vrrp_sync_group區域
? ?vrrp_instance用來定義對外提供服務的VIP區域及其相關屬性。?vrrp_rsync_group用來定義vrrp_intance組，使得這個組內成員動作一致。舉個例子來說明一下其功能：?兩個vrrp_instance同屬于一個vrrp_rsync_group，那么其中一個vrrp_instance發生故障切換時，另一個vrrp_instance也會跟著切換（即使這個instance沒有發生故障）。
vrrp_sync_group?VG_1?{group?{inside_network???#?name?of?vrrp_instance?(below)outside_network??#?One?for?each?moveable?IP....}notify_master?/path/to_master.shnotify_backup?/path/to_backup.shnotify_fault?"/path/fault.sh?VG_1"notify?/path/notify.shsmtp_alert } vrrp_instance?VI_1?{state?MASTERinterface?eth0use_vmac?<VMAC_INTERFACE>dont_track_primarytrack_interface?{eth0eth1}mcast_src_ip?<IPADDR>lvs_sync_daemon_interface?eth1garp_master_delay?10virtual_router_id?1priority?100advert_int?1authentication?{auth_type?PASSauth_pass?12345678}virtual_ipaddress?{10.210.214.253/24?brd?10.210.214.255?dev?eth0192.168.1.11/24?brd?192.168.1.255?dev?eth1}virtual_routes?{172.16.0.0/12?via?10.210.214.1192.168.1.0/24?via?192.168.1.1?dev?eth1default?via?202.102.152.1}track_script?{chk_http_port}nopreemptpreempt_delay?300debugnotify_master?<STRING>|<QUOTED-STRING>notify_backup?<STRING>|<QUOTED-STRING>notify_fault?<STRING>|<QUOTED-STRING>notify?<STRING>|<QUOTED-STRING>smtp_alert }notify_master/backup/fault 分別表示切換為主/備/出錯時所執行的腳本。?
notify 表示任何一狀態切換時都會調用該腳本，并且該腳本在以上三個腳本執行完成之后進行調用，keepalived會自動傳遞三個參數（$1 = "GROUP"|"INSTANCE"，$2 = name of group or instance，$3 = target state of transition(MASTER/BACKUP/FAULT)）。?
smtp_alert 表示是否開啟郵件通知（用全局區域的郵件設置來發通知）。?
state 可以是MASTER或BACKUP，不過當其他節點keepalived啟動時會將priority比較大的節點選舉為MASTER，因此該項其實沒有實質用途。?
interface 節點固有IP（非VIP）的網卡，用來發VRRP包。?
use_vmac 是否使用VRRP的虛擬MAC地址。?
dont_track_primary 忽略VRRP網卡錯誤。（默認未設置）?
track_interface 監控以下網卡，如果任何一個不通就會切換到FALT狀態。（可選項）?
mcast_src_ip 修改vrrp組播包的源地址，默認源地址為master的IP。（由于是組播，因此即使修改了源地址，該master還是能收到回應的）?
lvs_sync_daemon_interface 綁定lvs syncd的網卡。?
garp_master_delay 當切為主狀態后多久更新ARP緩存，默認5秒。?
virtual_router_id 取值在0-255之間，用來區分多個instance的VRRP組播。注意： 同一網段中virtual_router_id的值不能重復，否則會出錯。
priority 用來選舉master的，要成為master，那么這個選項的值最好高于其他機器50個點，該項取值范圍是1-255（在此范圍之外會被識別成默認值100）。?
advert_int 發VRRP包的時間間隔，即多久進行一次master選舉（可以認為是健康查檢時間間隔）。?
authentication 認證區域，認證類型有PASS和HA（IPSEC），推薦使用PASS（密碼只識別前8位）。?
virtual_ipaddress vip，不解釋了。?
virtual_routes 虛擬路由，當IP漂過來之后需要添加的路由信息。?
virtual_ipaddress_excluded 發送的VRRP包里不包含的IP地址，為減少回應VRRP包的個數。在網卡上綁定的IP地址比較多的時候用。?
nopreempt 允許一個priority比較低的節點作為master，即使有priority更高的節點啟動。首先nopreemt必須在state為BACKUP的節點上才生效（因為是BACKUP節點決定是否來成為MASTER的），其次要實現類似于關閉auto failback的功能需要將所有節點的state都設置為BACKUP，或者將master節點的priority設置的比BACKUP低。我個人推薦使用將所有節點的state都設置成BACKUP并且都加上nopreempt選項，這樣就完成了關于autofailback功能，當想手動將某節點切換為MASTER時只需去掉該節點的nopreempt選項并且將priority改的比其他節點大，然后重新加載配置文件即可（等MASTER切過來之后再將配置文件改回去再reload一下）。?當使用track_script時可以不用加nopreempt，只需要加上preempt_delay 5，這里的間隔時間要大于vrrp_script中定義的時長。
preempt_delay master啟動多久之后進行接管資源（VIP/Route信息等），并提是沒有nopreempt選項。
4、virtual_server_group和virtual_server區域
virtual_server?IP?Port?{delay_loop?<INT>lb_algo?rr|wrr|lc|wlc|lblc|sh|dhlb_kind?NAT|DR|TUNpersistence_timeout?<INT>persistence_granularity?<NETMASK>protocol?TCPha_suspendvirtualhost?<STRING>alphaomegaquorum?<INT>hysteresis?<INT>quorum_up?<STRING>|<QUOTED-STRING>quorum_down?<STRING>|<QUOTED-STRING>sorry_server?<IPADDR>?<PORT>real_server?<IPADDR>?<PORT>?{weight?<INT>inhibit_on_failurenotify_up?<STRING>|<QUOTED-STRING>notify_down?<STRING>|<QUOTED-STRING>#?HTTP_GET|SSL_GET|TCP_CHECK|SMTP_CHECK|MISC_CHECKHTTP_GET|SSL_GET?{url?{path?<STRING>#?Digest?computed?with?genhashdigest?<STRING>status_code?<INT>}connect_port?<PORT>connect_timeout?<INT>nb_get_retry?<INT>delay_before_retry?<INT>}} }delay_loop 延遲輪詢時間（單位秒）。
lb_algo 后端調試算法（load balancing algorithm）。
lb_kind LVS調度類型NAT/DR/TUN。
virtualhost 用來給HTTP_GET和SSL_GET配置請求header的。
sorry_server 當所有real server宕掉時，sorry server頂替。
real_server 真正提供服務的服務器。
weight 權重。
notify_up/down 當real server宕掉或啟動時執行的腳本。
健康檢查的方式，N多種方式。
path 請求real serserver上的路徑。
digest/status_code 分別表示用genhash算出的結果和http狀態碼。
connect_port 健康檢查，如果端口通則認為服務器正常。
connect_timeout,nb_get_retry,delay_before_retry分別表示超時時長、重試次數，下次重試的時間延遲。
三、keepalived+lvs環境搭建
1、環境介紹
2、同步時間
[root@Master?~]#?ntpdate?202.120.2.101 [root@Slave?~]#?ntpdate?202.120.2.101 [root@node1?~]#?ntpdate?202.120.2.101 [root@node2?~]#?ntpdate?202.120.2.1013、realserver安裝httpd服務及提供測試頁
[root@node1?~]#?rpm?-q?httpd httpd-2.2.15-45.el6.centos.x86_64 [root@node1?~]#?cat?/www/a.com/htdoc/index.html? <h1>This?is?node1?!</h1> [root@node2?~]#?rpm?-q?httpd httpd-2.2.15-45.el6.centos.x86_64 [root@node2?~]#?cat?/www/a.com/htdoc/index.html? <h1>This?is?node2?!</h1>4、各realserver啟動httpd并測試
[root@node1?~]#?service?httpd?start [root@node2?~]#?service?httpd?start [root@Master?~]#?curl?http://192.168.1.9 <h1>This?is?node1?!</h1> [root@Master?~]#?curl?http://192.168.1.10 <h1>This?is?node2?!</h1>5、配置node1節點
[root@node1?~]#?vim?realserver.sh #!/bin/bash?? #?? #?Script?to?start?LVS?DR?real?server.?? #?description:?LVS?DR?real?server?? #?? .??/etc/rc.d/init.d/functions VIP=192.168.1.88?#修改你的VIP?? host=`/bin/hostname` case?"$1"?in?? start)??#?Start?LVS-DR?real?server?on?this?machine.??/sbin/ifconfig?lo?down??/sbin/ifconfig?lo?up??echo?1?>?/proc/sys/net/ipv4/conf/lo/arp_ignore??echo?2?>?/proc/sys/net/ipv4/conf/lo/arp_announce??echo?1?>?/proc/sys/net/ipv4/conf/all/arp_ignore??echo?2?>?/proc/sys/net/ipv4/conf/all/arp_announce/sbin/ifconfig?lo:0?$VIP?broadcast?$VIP?netmask?255.255.255.255?up??/sbin/route?add?-host?$VIP?dev?lo:0 ;;?? stop)#?Stop?LVS-DR?real?server?loopback?device(s).??/sbin/ifconfig?lo:0?down??echo?0?>?/proc/sys/net/ipv4/conf/lo/arp_ignore??echo?0?>?/proc/sys/net/ipv4/conf/lo/arp_announce??echo?0?>?/proc/sys/net/ipv4/conf/all/arp_ignore??echo?0?>?/proc/sys/net/ipv4/conf/all/arp_announce ;;?? status)#?Status?of?LVS-DR?real?server.??islothere=`/sbin/ifconfig?lo:0?|?grep?$VIP`??isrothere=`netstat?-rn?|?grep?"lo:0"?|?grep?$VIP`??if?[?!?"$islothere"?-o?!?"isrothere"?];then??#?Either?the?route?or?the?lo:0?device??#?not?found.??echo?"LVS-DR?real?server?Stopped."??else??echo?"LVS-DR?real?server?Running."??fi?? ;;?? *)??#?Invalid?entry.??echo?"$0:?Usage:?$0?{start|status|stop}"??exit?1?? ;;?? esac??? [root@node1?~]#?chmod?+x?realserver.sh [root@node1?~]#?./realserver.sh?start查看腳本是否執行成功：
[root@node1?~]#?ifconfig? eth0??????Link?encap:Ethernet??HWaddr?00:0C:29:FE:B8:0D??inet?addr:192.168.1.9??Bcast:192.168.1.255??Mask:255.255.255.0inet6?addr:?fe80::20c:29ff:fefe:b80d/64?Scope:LinkUP?BROADCAST?RUNNING?MULTICAST??MTU:1500??Metric:1RX?packets:15783?errors:0?dropped:0?overruns:0?frame:0TX?packets:4866?errors:0?dropped:0?overruns:0?carrier:0collisions:0?txqueuelen:1000?RX?bytes:1396596?(1.3?MiB)??TX?bytes:724790?(707.8?KiB)lo????????Link?encap:Local?Loopback??inet?addr:127.0.0.1??Mask:255.0.0.0inet6?addr:?::1/128?Scope:HostUP?LOOPBACK?RUNNING??MTU:65536??Metric:1RX?packets:0?errors:0?dropped:0?overruns:0?frame:0TX?packets:0?errors:0?dropped:0?overruns:0?carrier:0collisions:0?txqueuelen:0?RX?bytes:0?(0.0?b)??TX?bytes:0?(0.0?b)lo:0??????Link?encap:Local?Loopback??inet?addr:192.168.1.88??Mask:255.255.255.255UP?LOOPBACK?RUNNING??MTU:65536??Metric:16、配置node2
[root@node2?~]#?vim?realserver.sh #!/bin/bash?? #?? #?Script?to?start?LVS?DR?real?server.?? #?description:?LVS?DR?real?server?? #?? .??/etc/rc.d/init.d/functions VIP=192.168.18.200?? host=`/bin/hostname` case?"$1"?in?? start)??#?Start?LVS-DR?real?server?on?this?machine.??/sbin/ifconfig?lo?down??/sbin/ifconfig?lo?up??echo?1?>?/proc/sys/net/ipv4/conf/lo/arp_ignore??echo?2?>?/proc/sys/net/ipv4/conf/lo/arp_announce??echo?1?>?/proc/sys/net/ipv4/conf/all/arp_ignore??echo?2?>?/proc/sys/net/ipv4/conf/all/arp_announce/sbin/ifconfig?lo:0?$VIP?broadcast?$VIP?netmask?255.255.255.255?up??/sbin/route?add?-host?$VIP?dev?lo:0 ;;?? stop)#?Stop?LVS-DR?real?server?loopback?device(s).??/sbin/ifconfig?lo:0?down??echo?0?>?/proc/sys/net/ipv4/conf/lo/arp_ignore??echo?0?>?/proc/sys/net/ipv4/conf/lo/arp_announce??echo?0?>?/proc/sys/net/ipv4/conf/all/arp_ignore??echo?0?>?/proc/sys/net/ipv4/conf/all/arp_announce ;;?? status)#?Status?of?LVS-DR?real?server.??islothere=`/sbin/ifconfig?lo:0?|?grep?$VIP`??isrothere=`netstat?-rn?|?grep?"lo:0"?|?grep?$VIP`??if?[?!?"$islothere"?-o?!?"isrothere"?];then??#?Either?the?route?or?the?lo:0?device??#?not?found.??echo?"LVS-DR?real?server?Stopped."??else??echo?"LVS-DR?real?server?Running."??fi?? ;;?? *)??#?Invalid?entry.??echo?"$0:?Usage:?$0?{start|status|stop}"??exit?1?? ;;?? esac [root@node2?~]#?chmod?+x?realserver.sh [root@node2?~]#?./realserver.sh?start [root@node2?~]#?ifconfig? eth0??????Link?encap:Ethernet??HWaddr?00:0C:29:73:1D:99??inet?addr:192.168.1.10??Bcast:192.168.1.255??Mask:255.255.255.0inet6?addr:?fe80::20c:29ff:fe73:1d99/64?Scope:LinkUP?BROADCAST?RUNNING?MULTICAST??MTU:1500??Metric:1RX?packets:20394?errors:0?dropped:0?overruns:0?frame:0TX?packets:5767?errors:0?dropped:0?overruns:0?carrier:0collisions:0?txqueuelen:1000?RX?bytes:5470619?(5.2?MiB)??TX?bytes:731727?(714.5?KiB)lo????????Link?encap:Local?Loopback??inet?addr:127.0.0.1??Mask:255.0.0.0inet6?addr:?::1/128?Scope:HostUP?LOOPBACK?RUNNING??MTU:65536??Metric:1RX?packets:0?errors:0?dropped:0?overruns:0?frame:0TX?packets:0?errors:0?dropped:0?overruns:0?carrier:0collisions:0?txqueuelen:0?RX?bytes:0?(0.0?b)??TX?bytes:0?(0.0?b)lo:0??????Link?encap:Local?Loopback??inet?addr:192.168.1.88??Mask:255.255.255.255UP?LOOPBACK?RUNNING??MTU:65536??Metric:1現在已經將realserver給配置好了，下面就是配置master與slave。
7、安裝keepalived、ipvsadm
[root@Master?~]#?yum?install?-y?keepalived?ipvsadm [root@Slave?~]#?yum?install?-y?keepalived?ipvsadm8、修改Master配置文件并啟動服務
[root@Master?~]#?cat?/etc/keepalived/keepalived.conf !?Configuration?File?for?keepalived global_defs?{??notification_email?{??? XXXXXXXXX@126.com}???notification_email_from?Master?smtp_server?127.0.0.1????smtp_connect_timeout?30router_id?LVS_DEVEL??? } vrrp_instance?VI_1?{??state?MASTER????interface?eth0???virtual_router_id?51???priority?101????advert_int?1???authentication?{???auth_type?PASS???auth_pass?1111???}???virtual_ipaddress?{???192.168.1.88???}??? } virtual_server?192.168.1.88?80?{??delay_loop?6???lb_algo?rr???lb_kind?DR???nat_mask?255.255.255.0???#persistence_timeout?50???protocol?TCPreal_server?192.168.1.9?80?{?weight?1???HTTP_GET?{????url?{???path?/???status_code?200???}???connect_timeout?2???nb_get_retry?3???delay_before_retry?1???}???}???real_server?192.168.1.10?80?{???weight?1???HTTP_GET?{???url?{???path?/???status_code?200???}???connect_timeout?2???nb_get_retry?3???delay_before_retry?1???}???}??? } [root@Master?~]#?service?keepalived?start 正在啟動?keepalived：??????????????????????????????????????[確定] [root@Master?~]#?ipvsadm?-L?-n IP?Virtual?Server?version?1.2.1?(size=4096) Prot?LocalAddress:Port?Scheduler?Flags->?RemoteAddress:Port???????????Forward?Weight?ActiveConn?InActConn TCP??192.168.1.88:80?rr->?192.168.1.9:80???????????????Route???1??????0??????????0?????????->?192.168.1.10:80??????????????Route???1??????0??????????09、為Slave修改配置文件并啟動服務
[root@Slave?~]#?cat?/etc/keepalived/keepalived.conf !?Configuration?File?for?keepalived global_defs?{??notification_email?{??? XXXXXXXX@126.com??? }???notification_email_from?Slavesmtp_server?127.0.0.1???smtp_connect_timeout?30???router_id?LVS_DEVEL??? } vrrp_instance?VI_1?{??state?BACKUP????interface?eth0???virtual_router_id?51???priority?100????advert_int?1???authentication?{???auth_type?PASS???auth_pass?1111???}???virtual_ipaddress?{???192.168.1.88???}??? } virtual_server?192.168.1.88?80?{??delay_loop?6???lb_algo?rr???lb_kind?DR???nat_mask?255.255.255.0???#persistence_timeout?50???protocol?TCPreal_server?192.168.1.9?80?{???weight?1???HTTP_GET?{???url?{???path?/???status_code?200???}???connect_timeout?2???nb_get_retry?3???delay_before_retry?1???}???}???real_server?192.168.1.10?80?{???weight?1???HTTP_GET?{???url?{???path?/???status_code?200???}???connect_timeout?2???nb_get_retry?3???delay_before_retry?1???}???}??? } [root@Slave?~]#?service?keepalived?start 正在啟動?keepalived：??????????????????????????????????????[確定] [root@Slave?~]#?ipvsadm?-L?-n IP?Virtual?Server?version?1.2.1?(size=4096) Prot?LocalAddress:Port?Scheduler?Flags->?RemoteAddress:Port???????????Forward?Weight?ActiveConn?InActConn TCP??192.168.1.88:80?rr->?192.168.1.9:80???????????????Route???1??????0??????????0?????????->?192.168.1.10:80??????????????Route???1??????0??????????010、瀏覽器測試
11、模擬realserver故障
停止node1：
[root@node1?~]#?service?httpd?stop 停止?httpd：???????????????????????????????????????????????[確定] 查看lvs： [root@Master?~]#?ipvsadm?-L?-n IP?Virtual?Server?version?1.2.1?(size=4096) Prot?LocalAddress:Port?Scheduler?Flags->?RemoteAddress:Port???????????Forward?Weight?ActiveConn?InActConn TCP??192.168.1.88:80?rr->?192.168.1.10:80??????????????Route???1??????0??????????0瀏覽器查看：
收到下線郵件：
node1上線：
[root@node1?~]#?service?httpd?start 正在啟動?httpd：???????????????????????????????????????????[確定] 查看lvs： [root@Master?~]#?ipvsadm?-L?-n IP?Virtual?Server?version?1.2.1?(size=4096) Prot?LocalAddress:Port?Scheduler?Flags->?RemoteAddress:Port???????????Forward?Weight?ActiveConn?InActConn TCP??192.168.1.88:80?rr->?192.168.1.9:80???????????????Route???1??????0??????????0?????????->?192.168.1.10:80??????????????Route???1??????0??????????0收到上線郵件：
12、模擬keepalived節點故障
將Master的 keepalived服務停止：
[root@Master?~]#?service?keepalived?stop 停止?keepalived：??????????????????????????????????????????[確定]查看lvs：
瀏覽器查看，發現服務沒有停止：
? ?大家可以看到，經過上面的演示我們現在LVS的高可用即前端負載均衡的高可用，同時實現對后端realserver監控，也實現后端realserver宕機時會給管理員發送郵件。但還有幾個問題我們還沒有解決，問題如下:
所有realserver都down機，怎么處理？是不是用戶就沒法打開，還是提供一下維護頁面。?
怎么完成維護模式keepalived切換？?
如何在keepalived故障時，發送警告郵件給指定的管理員？
13、為各keepalived提供錯誤頁面
? ?當我們的所以realserver全部都掛掉以后可以在前端的訪問入口，即keepalived+lvs機器上提供個錯誤提示頁，這樣做對用戶有很好的印象，不會是干巴巴的5XX。
先給Master、Slave安裝http服務及提供錯誤頁：
[root@Master?~]#?rpm?-q?httpd httpd-2.2.15-45.el6.centos.x86_64 [root@Master?~]#?cat?/www/a.com/htdoc/index.html <h1>Website?is?currently?under?maintenance,?please?come?back?later!</h1> [root@Slave?~]#?rpm?-q?httpd httpd-2.2.15-47.el6.centos.i686 [root@Slave?~]#?cat?/var/www/html/index.html? <h1>Website?is?currently?under?maintenance,?please?come?back?later!</h1>啟動httpd服務并測試是否能訪問：
[root@Master?~]#?service?httpd?start [root@Slave?~]#?service?httpd?start [root@node1?~]#?curl?http://192.168.1.8 <h1>Website?is?currently?under?maintenance,?please?come?back?later!</h1> [root@node1?~]#?curl?http://192.168.1.22 <h1>Website?is?currently?under?maintenance,?please?come?back?later!</h1>修改Master配置文件：
[root@Master?~]#?cat?/etc/keepalived/keepalived.conf !?Configuration?File?for?keepalived global_defs?{??notification_email?{??? XXXXXXXXX@126.com}???notification_email_from?Master?smtp_server?127.0.0.1????smtp_connect_timeout?30router_id?LVS_DEVEL??? } vrrp_instance?VI_1?{??state?MASTER????interface?eth0???virtual_router_id?51???priority?101????advert_int?1???authentication?{???auth_type?PASS???auth_pass?1111???}???virtual_ipaddress?{???192.168.1.88???}??? } virtual_server?192.168.1.88?80?{??delay_loop?6???lb_algo?rr???lb_kind?DR???nat_mask?255.255.255.0???#persistence_timeout?50???protocol?TCPreal_server?192.168.1.9?80?{?weight?1???HTTP_GET?{????url?{???path?/???status_code?200???}???connect_timeout?2???nb_get_retry?3???delay_before_retry?1???}???}???real_server?192.168.1.10?80?{???weight?1???HTTP_GET?{???url?{???path?/???status_code?200???}???connect_timeout?2???nb_get_retry?3???delay_before_retry?1???}???}???sorry_server?127.0.0.1?80?#增加一行sorry_server? }修改Slave配置文件：
把上面添加的內容sorry_server 127.0.0.1 80，放在Slave的keepalived配置文件相同的位置。
關閉所有的real server并重新啟動一下master與slave的keepalived：
[root@node1?~]#?service?httpd?stop [root@node2?~]#?service?httpd?stop [root@Master?~]#?service?keepalived?restart [root@Slave?~]#?service?keepalived?restart查看lvs：
[root@Slave?~]#?ipvsadm?-L?-n IP?Virtual?Server?version?1.2.1?(size=4096) Prot?LocalAddress:Port?Scheduler?Flags->?RemoteAddress:Port???????????Forward?Weight?ActiveConn?InActConn TCP??192.168.1.88:80?rr->?127.0.0.1:80?????????????????Local???1??????0??????????0瀏覽器測試：
14、為Master和Slave提供狀態檢測
? ?我們一般進行主從切換測試時都是關閉keepalived或關閉網卡接口，有沒有一種方法能實現在不關閉keepalived下或網卡接口來實現維護呢？方法肯定是有的，在keepalived新版本中，支持腳本vrrp_srcipt。
定義腳本說明：
vrrp_srcipt?chk_schedown?{?#定義vrrp執行腳本script?"[?-e?/etc/keepalived/down?]?&&?exit?1?||?exit?0"???#查看是否有down文件，有就進入維護模式?interval?1?#監控間隔??weight?-5?#減小優先級??fall?2?#監控失敗次數??rise?1?#監控成功次數?? }執行腳本：
track_script?{chk_schedown?#執行chk_schedown腳本 }修改Master配置文件：
[root@Master?~]#?cat?/etc/keepalived/keepalived.conf !?Configuration?File?for?keepalived global_defs?{??notification_email?{??? XXXXXXXXX@126.com}???notification_email_from?Master?smtp_server?127.0.0.1????smtp_connect_timeout?30router_id?LVS_DEVEL??? } vrrp_script?chk_schedown?{?script?"[?-e?/etc/keepalived/down?]?&&?exit?1?||?exit?0"???interval?1weight?-5?fall?2??rise?1?? } vrrp_instance?VI_1?{??state?MASTER????interface?eth0???virtual_router_id?51???priority?101????advert_int?1???authentication?{???auth_type?PASS???auth_pass?1111???}???virtual_ipaddress?{???192.168.1.88???}???track_script?{?chk_schedown??}?? } virtual_server?192.168.1.88?80?{??delay_loop?6???lb_algo?rr???lb_kind?DR???nat_mask?255.255.255.0???#persistence_timeout?50???protocol?TCPreal_server?192.168.1.9?80?{?weight?1???HTTP_GET?{????url?{???path?/???status_code?200???}???connect_timeout?2???nb_get_retry?3???delay_before_retry?1???}???}???real_server?192.168.1.10?80?{???weight?1???HTTP_GET?{???url?{???path?/???status_code?200???}???connect_timeout?2???nb_get_retry?3???delay_before_retry?1???}???}sorry_server?127.0.0.1?80??? }Slave上的配置文件在相同的位置也添加上面兩項內容：vrrp_script?chk_schedown和track_script?
測試：
[root@Master?~]#?touch?/etc/keepalived/down查看日志
[root@Master?~]#?tail?-f?/var/log/messages Sep?11?20:12:27?Master?Keepalived_vrrp[2019]:?VRRP_Script(chk_schedown)?failed Sep?11?20:12:28?Master?Keepalived_vrrp[2019]:?VRRP_Instance(VI_1)?Received?higher?prio?advert Sep?11?20:12:28?Master?Keepalived_vrrp[2019]:?VRRP_Instance(VI_1)?Entering?BACKUP?STATE Sep?11?20:12:28?Master?Keepalived_vrrp[2019]:?VRRP_Instance(VI_1)?removing?protocol?VIPs. Sep?11?20:12:28?Master?Keepalived_healthcheckers[2018]:?Netlink?reflector?reports?IP?192.168.1.88?removed查看VIP是否轉移：
[root@Master?~]#?ip?addr?show 1:?lo:?<LOOPBACK,UP,LOWER_UP>?mtu?65536?qdisc?noqueue?state?UNKNOWN?link/loopback?00:00:00:00:00:00?brd?00:00:00:00:00:00inet?127.0.0.1/8?scope?host?loinet6?::1/128?scope?host?valid_lft?forever?preferred_lft?forever 2:?eth0:?<BROADCAST,MULTICAST,UP,LOWER_UP>?mtu?1500?qdisc?pfifo_fast?state?UP?qlen?1000link/ether?00:0c:29:b0:04:27?brd?ff:ff:ff:ff:ff:ffinet?192.168.1.8/24?brd?192.168.1.255?scope?global?eth0inet6?fe80::20c:29ff:feb0:427/64?scope?link?valid_lft?forever?preferred_lft?forever [root@Slave?~]#?ip?addr?show 1:?lo:?<LOOPBACK,UP,LOWER_UP>?mtu?65536?qdisc?noqueue?state?UNKNOWN?link/loopback?00:00:00:00:00:00?brd?00:00:00:00:00:00inet?127.0.0.1/8?scope?host?loinet6?::1/128?scope?host?valid_lft?forever?preferred_lft?forever 2:?eth0:?<BROADCAST,MULTICAST,UP,LOWER_UP>?mtu?1500?qdisc?pfifo_fast?state?UNKNOWN?qlen?1000link/ether?00:0c:29:df:1e:04?brd?ff:ff:ff:ff:ff:ffinet?192.168.1.22/24?brd?192.168.1.255?scope?global?eth0inet?192.168.1.88/32?scope?global?eth0inet6?fe80::20c:29ff:fedf:1e04/64?scope?link?valid_lft?forever?preferred_lft?forever15、在keepalived故障時(或主備切換時)，發送警告郵件給指定的管理員
keepalived通知腳本進階示例：
-s, --service SERVICE,...：指定服務腳本名稱，當狀態切換時可自動啟動、重啟或關閉此服務；?
-a, --address VIP: 指定相關虛擬路由器的VIP地址；?
-m, --mode {mm|mb}：指定虛擬路由的模型，mm表示主主，mb表示主備；它們表示相對于同一種服務而方，其VIP的工作類型；?
-n, --notify {master|backup|fault}：指定通知的類型，即vrrp角色切換的目標角色；?
修改Master配置文件：
[root@Master?~]#?cat?/etc/keepalived/keepalived.conf !?Configuration?File?for?keepalived global_defs?{??notification_email?{??? XXXXXXXX@126.com}???notification_email_from?Master?smtp_server?127.0.0.1????smtp_connect_timeout?30router_id?LVS_DEVEL??? } vrrp_script?chk_schedown?{?#定義vrrp執行腳本??script?"[?-e?/etc/keepalived/down?]?&&?exit?1?||?exit?0"???interval?1weight?-5?fall?2??rise?1?? } vrrp_instance?VI_1?{??state?MASTER????interface?eth0???virtual_router_id?51???priority?101????advert_int?1???authentication?{???auth_type?PASS???auth_pass?1111???}???virtual_ipaddress?{???192.168.1.88???}???track_script?{?chk_schedown??}??#添加如下三行notify_master?"/etc/keepalived/notify.sh?-n?master?-a?192.168.1.88"??notify_backup?"/etc/keepalived/notify.sh?-n?backup?-a?192.168.1.88"notify_fault?"/etc/keepalived/notify.sh?-n?fault?-a?192.168.1.88" } virtual_server?192.168.1.88?80?{??delay_loop?6???lb_algo?rr???lb_kind?DR???nat_mask?255.255.255.0???#persistence_timeout?50???protocol?TCPreal_server?192.168.1.9?80?{?weight?1???HTTP_GET?{????url?{???path?/???status_code?200???}???connect_timeout?2???nb_get_retry?3???delay_before_retry?1???}???}???real_server?192.168.1.10?80?{???weight?1???HTTP_GET?{???url?{???path?/???status_code?200???}???connect_timeout?2???nb_get_retry?3???delay_before_retry?1???}???}sorry_server?127.0.0.1?80??? }修改Slave的配置文件：
在Slave的keepalived配置文件中插入下面三行，位置和Master所放的位置一樣。
??notify_master?"/etc/keepalived/notify.sh?-n?master?-a?192.168.1.88"??notify_backup?"/etc/keepalived/notify.sh?-n?backup?-a?192.168.1.88"notify_fault?"/etc/keepalived/notify.sh?-n?fault?-a?192.168.1.88"在Master和Slave中同時增加notify.sh 腳本：
[root@Master?~]#?cat?/etc/keepalived/notify.sh? #!/bin/bash?? #?Author:?freeloda??? #?description:?An?example?of?notify?script?? #?Usage:?notify.sh?-m|--mode?{mm|mb}?-s|--service?SERVICE1,...?-a|--address?VIP??-n|--notify?{master|backup|falut}?-h|--help contact='XXXXXXX@126.com'? helpflag=0?? serviceflag=0?? modeflag=0?? addressflag=0?? notifyflag=0 Usage()?{?echo?"Usage:?notify.sh?[-m|--mode?{mm|mb}]?[-s|--service?SERVICE1,...]?<-a|--address?VIP>??<-n|--notify?{master|backup|falut}>"???echo?"Usage:?notify.sh?-h|--help"?? } ParseOptions()?{?local?I=1;??if?[?$#?-gt?0?];?then??while?[?$I?-le?$#?];?do??case?$1?in??-s|--service)??[?$#?-lt?2?]?&&?return?3??serviceflag=1??services=(`echo?$2|awk?-F","?'{for(i=1;i<=NF;i++)?print?$i}'`)??shift?2?;;??-h|--help)??helpflag=1??return?0??shift??;;??-a|--address)??[?$#?-lt?2?]?&&?return?3??addressflag=1??vip=$2??shift?2??;;??-m|--mode)??[?$#?-lt?2?]?&&?return?3??mode=$2??shift?2??;;??-n|--notify)??[?$#?-lt?2?]?&&?return?3??notifyflag=1??notify=$2??shift?2??;;??*)??echo?"Wrong?options..."??Usage??return?7??;;??esac??done??return?0??fi?? } #workspace=$(dirname?$0) RestartService()?{?if?[?${#@}?-gt?0?];?then??for?I?in?$@;?do??if?[?-x?/etc/rc.d/init.d/$I?];?then??/etc/rc.d/init.d/$I?restart??else??echo?"$I?is?not?a?valid?service..."??fi??done??fi?? } StopService()?{?if?[?${#@}?-gt?0?];?then??for?I?in?$@;?do??if?[?-x?/etc/rc.d/init.d/$I?];?then??/etc/rc.d/init.d/$I?stop??else??echo?"$I?is?not?a?valid?service..."??fi??done??fi?? } Notify()?{??mailsubject="`hostname`?to?be?$1:?$vip?floating"??mailbody="`date?'+%F?%H:%M:%S'`,?vrrp?transition,?`hostname`?changed?to?be?$1."??echo?$mailbody?|?mail?-s?"$mailsubject"?$contact????##注意此命令需要安裝mailx包。 } #?Main?Function?? ParseOptions?$@?? [?$??-ne?0?]?&&?Usage?&&?exit?5 [?$helpflag?-eq?1?]?&&?Usage?&&?exit?0 if?[?$addressflag?-ne?1?-o?$notifyflag?-ne?1?];?then?Usage??exit?2?? fi mode=${mode:-mb} case?$notify?in? 'master')??if?[?$serviceflag?-eq?1?];?then??RestartService?${services[*]}??fi??Notify?master??;;?? 'backup')??if?[?$serviceflag?-eq?1?];?then??if?[?"$mode"?==?'mb'?];?then??StopService?${services[*]}??else??RestartService?${services[*]}??fi??fi??Notify?backup??;;?? 'fault')??Notify?fault??;;?? *)??Usage??exit?4??;;?? esac [root@Master?~]#?chmod?+x?/etc/keepalived/notify.sh [root@Slave?~]#?chmod?+x?/etc/keepalived/notify.sh從新載入配置文件：
[root@Master?~]#?service?keepalived?restart [root@Slave?~]#?service?keepalived?restart模擬故障：
現將剛才測試文件給刪除。
[root@Master?~]#?rm?-rf?/etc/keepalived/down查看VIP：
[root@Master?~]#?ip?addr?show 1:?lo:?<LOOPBACK,UP,LOWER_UP>?mtu?65536?qdisc?noqueue?state?UNKNOWN?link/loopback?00:00:00:00:00:00?brd?00:00:00:00:00:00inet?127.0.0.1/8?scope?host?loinet6?::1/128?scope?host?valid_lft?forever?preferred_lft?forever 2:?eth0:?<BROADCAST,MULTICAST,UP,LOWER_UP>?mtu?1500?qdisc?pfifo_fast?state?UP?qlen?1000link/ether?00:0c:29:b0:04:27?brd?ff:ff:ff:ff:ff:ffinet?192.168.1.8/24?brd?192.168.1.255?scope?global?eth0inet?192.168.1.88/32?scope?global?eth0inet6?fe80::20c:29ff:feb0:427/64?scope?link?valid_lft?forever?preferred_lft?forever進入維護模式：
[root@Master?~]#?touch?/etc/keepalived/down [root@Master?~]#?ip?addr?show 1:?lo:?<LOOPBACK,UP,LOWER_UP>?mtu?65536?qdisc?noqueue?state?UNKNOWN?link/loopback?00:00:00:00:00:00?brd?00:00:00:00:00:00inet?127.0.0.1/8?scope?host?loinet6?::1/128?scope?host?valid_lft?forever?preferred_lft?forever 2:?eth0:?<BROADCAST,MULTICAST,UP,LOWER_UP>?mtu?1500?qdisc?pfifo_fast?state?UP?qlen?1000link/ether?00:0c:29:b0:04:27?brd?ff:ff:ff:ff:ff:ffinet?192.168.1.8/24?brd?192.168.1.255?scope?global?eth0inet6?fe80::20c:29ff:feb0:427/64?scope?link?valid_lft?forever?preferred_lft?forever [root@Slave?~]#?ip?addr?show 1:?lo:?<LOOPBACK,UP,LOWER_UP>?mtu?65536?qdisc?noqueue?state?UNKNOWN?link/loopback?00:00:00:00:00:00?brd?00:00:00:00:00:00inet?127.0.0.1/8?scope?host?loinet6?::1/128?scope?host?valid_lft?forever?preferred_lft?forever 2:?eth0:?<BROADCAST,MULTICAST,UP,LOWER_UP>?mtu?1500?qdisc?pfifo_fast?state?UNKNOWN?qlen?1000link/ether?00:0c:29:df:1e:04?brd?ff:ff:ff:ff:ff:ffinet?192.168.1.22/24?brd?192.168.1.255?scope?global?eth0inet?192.168.1.88/32?scope?global?eth0inet6?fe80::20c:29ff:fedf:1e04/64?scope?link?valid_lft?forever?preferred_lft?forever查看郵件：
大家可以看到，主備切換時，會發送郵件報警，好了到這里所有演示全部完成。
此文是此篇文章的演練：http://freeloda.blog.51cto.com/2033581/1280962
轉載于:https://blog.51cto.com/cuchadanfan/1696588
總結
以上是生活随笔為你收集整理的Linux 高可用（HA）集群之keepalived的全部內容，希望文章能夠幫你解決所遇到的問題。
- 上一篇： 2016年学习Linux决心书（老男孩教
- 下一篇： ***PHP 去除换行符