keepalived高可用集群實戰項目

---

需要準備17臺虛擬機

目錄

• 客戶端
• 路由器
• powerDNS
• 防火墻
• 時間同步服務器
• keepalived + LVS集群實現web服務高可用
• 配置web_A與web_B
• 實驗：MHA
• • Master主機：192.168.99.114
  • slave主機：192.168.99.115
  • 另一個slave主機：192.168.99.116
  • MHA_Manage主機
• proxySQL高可用方案
• • proxySQL_1
  • proxySQL_2
  • 實現proxySQL高可用集群
• NFS配置與discuz布署
• rsync + inotify實時同步
• • keepalived實現NFS高可用
• NFS的自動掛載

---

客戶端

網卡配置：僅主機模式1

ip配置：

ip：192.168.88.100/24 gateway：192.168.88.254 dns：172.16.23.211

---

路由器

網卡配置：僅主機模式1(eth0) + 橋接模式(eth1)
ip配置：

eth0：192.168.88.254/24 eth1：172.16.23.254/24

路由配置

route add default dev eth1

SNAT配置

iptables -t nat -A POSTROUTING -s 192.168.88.0/24 ! –d 192.168.88.0/24 -j SNAT --to-source 172.16.23.254

ip轉發

echo "net.ipv4.ip_forward = 1" > /etc/sysctl.conf sysctl -p

---

powerDNS

網卡配置：橋接模式

ip配置：

ip: 172.16.23.211/24

安裝http + php + mariadb

yum install -y pdns pdns-backend-mysql httpd php php-mysql php-mbstring mariadb-server# 啟動服務 systemctl start httpd systemctl start mariadb# 配置PowerDNS使用mariadb作為后臺數據存儲 sed -i '/^launch/s/^.*$/launch=gmysql\ngmysql-host=localhost\ngmysql-port=3306\ngmysql-dbname=powerdns\ngmysql-user=powerdns\ngmysql-password=123/' /etc/pdns/pdns.conf# 創建數據庫和帳號給powerDNS用 mysqlcreate database powerdns; grant all privileges on powerdns.* to powerdns@localhost identified by "123";# 還有powerdns的數據庫 USE powerdns; CREATE TABLE domains (id INT AUTO_INCREMENT,name VARCHAR(255) NOT NULL,master VARCHAR(128) DEFAULT NULL,last_check INT DEFAULT NULL,type VARCHAR(6) NOT NULL,notified_serial INT DEFAULT NULL,account VARCHAR(40) DEFAULT NULL,PRIMARY KEY (id) ) Engine=InnoDB;CREATE UNIQUE INDEX name_index ON domains(name);CREATE TABLE records (id BIGINT AUTO_INCREMENT,domain_id INT DEFAULT NULL,name VARCHAR(255) DEFAULT NULL,type VARCHAR(10) DEFAULT NULL,content VARCHAR(64000) DEFAULT NULL,ttl INT DEFAULT NULL,prio INT DEFAULT NULL,change_date INT DEFAULT NULL,disabled TINYINT(1) DEFAULT 0,ordername VARCHAR(255) BINARY DEFAULT NULL,auth TINYINT(1) DEFAULT 1,PRIMARY KEY (id) ) Engine=InnoDB;CREATE INDEX nametype_index ON records(name,type); CREATE INDEX domain_id ON records(domain_id); CREATE INDEX recordorder ON records (domain_id, ordername);CREATE TABLE supermasters (ip VARCHAR(64) NOT NULL,nameserver VARCHAR(255) NOT NULL,account VARCHAR(40) NOT NULL,PRIMARY KEY (ip, nameserver) ) Engine=InnoDB;CREATE TABLE comments (id INT AUTO_INCREMENT,domain_id INT NOT NULL,name VARCHAR(255) NOT NULL,type VARCHAR(10) NOT NULL,modified_at INT NOT NULL,account VARCHAR(40) NOT NULL,comment VARCHAR(64000) NOT NULL,PRIMARY KEY (id) ) Engine=InnoDB;CREATE INDEX comments_domain_id_idx ON comments (domain_id); CREATE INDEX comments_name_type_idx ON comments (name, type); CREATE INDEX comments_order_idx ON comments (domain_id, modified_at);CREATE TABLE domainmetadata (id INT AUTO_INCREMENT,domain_id INT NOT NULL,kind VARCHAR(32),content TEXT,PRIMARY KEY (id) ) Engine=InnoDB;CREATE INDEX domainmetadata_idx ON domainmetadata (domain_id, kind);CREATE TABLE cryptokeys (id INT AUTO_INCREMENT,domain_id INT NOT NULL,flags INT NOT NULL,active BOOL,content TEXT,PRIMARY KEY(id) ) Engine=InnoDB;CREATE INDEX domainidindex ON cryptokeys(domain_id);CREATE TABLE tsigkeys (id INT AUTO_INCREMENT,name VARCHAR(255),algorithm VARCHAR(50),secret VARCHAR(255),PRIMARY KEY (id) ) Engine=InnoDB;CREATE UNIQUE INDEX namealgoindex ON tsigkeys(name, algorithm);# 啟動服務 systemctl start pdns systemctl enable pdns# 安裝httpd和php相關包 yum -y install php-devel php-gd php-mcrypt php-imap php-ldap php-odbc php-pear php-xml php-xmlrpc php-mcrypt php-mhash gettext# 啟動服務 systemctl restart httpd# 下載poweradmin程序， cd /var/www/html wget http://downloads.sourceforge.net/project/poweradmin/poweradmin-2.1.7.tgz# 解壓縮到相應目錄 tar xvf poweradmin-2.1.7.tgz mv poweradmin-2.1.7 poweradmin# 設置下權限 setfacl -Rm u:apache:rwx poweradmin

訪問網頁安裝向導地址：
http://172.16.23.211/poweradmin/install/
下一步。下一步

---

根據你前面寫的帳號密碼來

---

如圖

---

按照下面頁面說明，在數據庫中創建用戶并授權，然后再下一步

---

按下面頁面說明，修改config.in.php文件內容,要先把原來的改名

mv /var/www/html/poweradmin/inc/config-me.inc.php /var/www/html/poweradmin/inc/config.inc.php

---

安裝完畢后，刪除install目錄

rm -rf /var/www/html/poweradmin/install/

---

登錄http://172.16.23.211/poweradmin/

username：admin password：123

---

來添加個master zone

---

準備給這個zone添加記錄

---

添加A記錄，指向172.16.23.200

用前面的客戶端測試下,如果不同就要檢查下你的DNS有沒有指向172.16.23.211這個powerDNS了

ping bbs.jibill.com

---

防火墻

網卡配置：僅主機模式2(eth0) + 橋接模式(eth1)

ip配置：
僅主機模式2

#這里注意了，192.168.99.254就是右邊所有主機的網關 eth0：192.168.99.254/24 eth1：172.16.23.200/24

ip轉發

echo "net.ipv4.ip_forward = 1" > /etc/sysctl.conf sysctl -p

配置DNAT

iptables -t nat -A PREROUTING -d 172.16.23.200/24 -p tcp --dport 80 -j DNAT --to-destination 192.168.0.100

配置SNAT

iptables -t nat -A POSTROUTING -s 192.168.99.0/24 ! –d 192.168.99.0/24 -j SNAT --to-source 172.16.23.200

添加VIP路由

route add -net 192.168.0.100/24 dev eth0

---

時間同步服務器

ip配置：

ip：192.168.99.105 gateway: 192.168.99.254

安裝與啟動chrony

yum install chrony systemctl start chronyd systemctl enable chronyd

配置時間同步源

cat /etc/chrony.conf# l國內大學的時間源3 server s1a.time.edu.cn iburst4 server s1b.time.edu.cn iburst5 server s1c.time.edu.cn iburst ... 25 allow 192.168.99.0/24 ... 28 local stratum 10

重啟服務生效

systemctl restart chronyd

顯示當前chronyd正在訪問的時間源的信息

chronyc sources -v

---

keepalived + LVS集群實現web服務高可用

ip配置

# ka1配置 ip：192.168.99.106 gateway: 192.168.99.254# ka2配置 ip：192.168.99.107 gateway: 192.168.99.254

ka1和ka2配置時間同步

yum -y install chrony sed -i -e '1i\server 192.168.99.105 iburst' -e '/^server/d' /etc/chrony.conf systemctl restart chronyd

在ka1和ka2上都安裝ipvsadm

yum -y install ipvsadm keepalived mailx

設置主機名

# 192.168.99.106設置為ka1 hostnamectl set-hostname "ka1"# 192.168.99.107設置為ka2 hostnamectl set-hostname "ka2"

ka1和ka2的郵箱配置

echo 'set from=417060833@qq.com' >> /etc/mail.rc echo 'set smtp=smtp.qq.com' >> /etc/mail.rc echo 'set smtp-auth=login' >> /etc/mail.rc echo 'set smtp-auth-user=你的QQ郵箱@qq.com' >> /etc/mail.rc echo 'set smtp-auth-password=你的授權碼' >> /etc/mail.rc

測試郵箱的配置

echo "內容" | mail -v -s "標題" 你的郵箱

在ka1創建腳本

[106]$ cat notify.sh #!/bin/bash # #*********************************************************** #Author: Jibill Chen #QQ: 417060833 #Date: 2019-08-09 #FileName： notify.sh #URL: http://www.jibiao.work #Description： The test script #********************************************************** vip="192.168.0.100" EMAIL="417060833@qq.com"notify() {mailsubject="`hostname` to be $1: $vip floating"mailbody="`date '+%F %H:%M:%S'`: vrrp transition, `hostname` changed to be $1"echo $mailbody | mail -s "$mailsubject" $EMAIL } rs_notify() {mailsubject="Real server: $2 to be $1"mailbody="`date '+%F %H:%M:%S'`: Real server: $2 changed to be $1"echo $mailbody | mail -s "$mailsubject" $EMAIL}case "$1" inmaster)notify masterexit 0;;backup)notify backupexit 0;;rsup)rs_notify up $2exit 0;;rsdown)rs_notify down $2exit 0;;*)echo 'Usage: `basename $0` {master|backup|fault}'exit 1;; esac

設置好權限，傳給ka2

chmod +x /etc/keepalived/notify.sh scp /etc/keepalived/notify.sh 192.168.99.107:/etc/keepalived/

ka2的keepalived配置

cat /etc/keepalived/keepalived.conf #配置如下 ! Configuration File for keepalivedglobal_defs {notification_email {root@localhost}notification_email_from root@localhostsmtp_server 127.0.0.1smtp_connect_timeout 30router_id ka1vrrp_skip_check_adv_addr # vrrp_strictvrrp_garp_interval 0vrrp_gna_interval 0 }vrrp_instance VI_1 {state MASTERinterface eth0virtual_router_id 11priority 100advert_int 1authentication {auth_type PASSauth_pass 1111}unicast_src_ip 192.168.99.106 # ka1的ipunicast_peer {192.168.99.107 # ka2的ip}virtual_ipaddress {192.168.0.100 dev eth0 label eth0:1 # vip}notify_master "/etc/keepalived/notify.sh master" #切換為主時腳本notify_backup "/etc/keepalived/notify.sh backup" #切換為backup時腳本 } virtual_server 192.168.0.100 80 { # vip + portdelay_loop 3lb_algo wrrlb_kind DRprotocol TCPreal_server 192.168.99.108 80 { # web_Anotify_up "/etc/keepalived/notify.sh rsup web_A" #RS上線通知腳本notify_down "/etc/keepalived/notify.sh rsdown web_A" #RS下線通知腳本weight 1TCP_CHECK {connect_port 80connect_timeout 4retry 3delay_before_retry 3}}real_server 192.168.99.109 80 { # web_Bnotify_up "/etc/keepalived/notify.sh rsup web_B" #RS上線通知腳本notify_down "/etc/keepalived/notify.sh rsdown web_B" #RS下線通知腳本weight 1TCP_CHECK {connect_port 80connect_timeout 50retry 3delay_before_retry 3}} }

ka2的配置

cat /etc/keepalived/keepalived.conf #配置如下 ! Configuration File for keepalivedglobal_defs {notification_email {root@localhost}notification_email_from root@localhostsmtp_server 127.0.0.1smtp_connect_timeout 30router_id ka2vrrp_skip_check_adv_addr # vrrp_strictvrrp_garp_interval 0vrrp_gna_interval 0 }vrrp_instance VI_1 {state BACKUPinterface eth0virtual_router_id 11priority 80advert_int 1authentication {auth_type PASSauth_pass 1111}unicast_src_ip 192.168.99.107 # ka2的ipunicast_peer {192.168.99.106 # ka1的ip}virtual_ipaddress {192.168.0.100 dev eth0 label eth0:1 # vip}notify_master "/etc/keepalived/notify.sh master" notify_backup "/etc/keepalived/notify.sh backup" } virtual_server 192.168.0.100 80 { # vip + portdelay_loop 3lb_algo wrrlb_kind DRprotocol TCPreal_server 192.168.99.108 80 { # web_Anotify_up "/etc/keepalived/notify.sh rsup web_A"notify_down "/etc/keepalived/notify.sh rsdown web_A" weight 1TCP_CHECK {connect_port 80connect_timeout 4retry 3delay_before_retry 3}}real_server 192.168.99.109 80 { # web_Bnotify_up "/etc/keepalived/notify.sh rsup web_B"notify_down "/etc/keepalived/notify.sh rsdown web_B" weight 1TCP_CHECK {connect_port 80connect_timeout 50retry 3delay_before_retry 3}} }

啟動ka1與ka2

systemctl restart keepalived

---

配置web_A與web_B

ip配置

# web_A: ip：192.168.99.108 gateway: 192.168.99.254#web_B: ip：192.168.99.109 gateway: 192.168.99.254

ka1和ka2配置時間同步

yum -y install chrony sed -i -e '1i\server 192.168.99.105 iburst' -e '/^server/d' /etc/chrony.conf systemctl restart chronyd

在web_A與web_B安裝httpd與LAMP環境并啟動

yum -y install httpd mariadb php php-mysql php-mbstring systemctl restart httpd

測試頁面

#web_A echo "testA" > /var/www/html/a.html#web_B echo "testB" > /var/www/html/a.html

設置arp

echo "net.ipv4.conf.lo.arp_ignore = 1" >> /etc/sysctl.conf echo "net.ipv4.conf.all.arp_ignore = 1" >> /etc/sysctl.conf echo "net.ipv4.conf.lo.arp_announce = 2" >> /etc/sysctl.conf echo "net.ipv4.conf.all.arp_announce = 2" >> /etc/sysctl.conf sysctl -p

設置vip

cd /etc/sysconfig/network-scripts/vim ifcfg-lo:0 #添加下面內容 DEVICE=lo:0 BOOTPROTO=static BROADCAST=192.168.0.255 NETWORK=192.168.0.0 IPADDR=192.168.0.100 #這個是VIP NETMASK=255.255.255.255 ONBOOT=yes TYPE=Ethernet#重啟 systemctl restart network

好了，測試下吧。用客戶端。如果失敗了你就回去檢查吧

[Client]$ curl 172.16.23.200/a.html testA[Client]$ curl 172.16.23.200/a.html testB

別忘了web_B也要配置一樣的

---

實驗：MHA

實驗環境：
1 臺MHA_Manage主機：192.168.99.117
1 臺Master主機：192.168.99.114
2 臺slave主機：192.168.99.115~116

Master主機：192.168.99.114

網絡配置

ip: 192.168.99.114 gateway: 192.168.99.254

時間同步

yum -y install chrony sed -i -e '1i\server 192.168.99.105 iburst' -e '/^server/d' /etc/chrony.conf systemctl restart chronyd

安裝mariadb-server并修改mariadb配置文件

yum -y install mariadb-server sed -i '/\[mysqld\]/a\log-bin\nserver_id=114\nskip_name_resolve=1' /etc/my.cnf#啟動 systemctl restart mariadb

創建連接需要的帳號

mysql#創建復制用帳號 MariaDB [(none)]> grant replication slave on *.* to repluser@'%' identified by '123';#創建管理用帳號 MariaDB [(none)]> grant all on *.* to mhauser@'192.168.99.%'identified by'123';#proxySQL監控的帳號 MariaDB [(none)]> grant replication client on *.* to monitor@'192.168.99.%' identified by'123'; #proxySQL訪問的帳號 MariaDB [(none)]> grant all on *.* to sqluser@'%' identified by '123';#discuz帳號和數據庫 MariaDB [(none)]> create database dzdb; MariaDB [(none)]> grant all privileges on dzdb.* to dzuser@'192.168.99.%' identified by "123";

---

slave主機：192.168.99.115

網絡配置

ip: 192.168.99.115 gateway: 192.168.99.254

時間同步

yum -y install chrony sed -i -e '1i\server 192.168.99.105 iburst' -e '/^server/d' /etc/chrony.conf systemctl restart chronyd

安裝修改mariadb配置文件

yum -y install mariadb-server sed -i '/\[mysqld\]/a\log-bin\nserver_id=115\nread_only\nskip_name_resolve=1\nrelay_log_purge=0' /etc/my.cnf #這里：關閉`relay_log_purge`是為了不讓mysql自動清除中繼日志 #啟動 systemctl restart mariadb

連接到主服務器

mysqlMariaDB [(none)]> CHANGE MASTER TO MASTER_HOST='192.168.99.114', MASTER_PORT=3306, MASTER_USER='repluser', MASTER_PASSWORD='123', MASTER_LOG_FILE='mariadb-bin.000001', MASTER_LOG_POS=245;#啟動 MariaDB [(none)]> start slave ;

---

另一個slave主機：192.168.99.116

網絡配置

ip: 192.168.99.116 gateway: 192.168.99.254

時間同步

yum -y install chrony sed -i -e '1i\server 192.168.99.105 iburst' -e '/^server/d' /etc/chrony.conf systemctl restart chronyd

安裝并修改mariadb配置文件

yum -y install mariadb-server sed -i '/\[mysqld\]/a\log-bin\nserver_id=116\nread_only\nskip_name_resolve=1\nrelay_log_purge=0' /etc/my.cnf#啟動 systemctl restart mariadb

連接到主服務器

mysqlMariaDB [(none)]> CHANGE MASTER TO MASTER_HOST='192.168.99.114', MASTER_PORT=3306, MASTER_USER='repluser', MASTER_PASSWORD='123', MASTER_LOG_FILE='mariadb-bin.000001', MASTER_LOG_POS=245;#啟動 MariaDB [(none)]> start slave ;

---

MHA_Manage主機

網絡配置

ip: 192.168.99.117 gateway: 192.168.99.254

時間同步

yum -y install chrony sed -i -e '1i\server 192.168.99.105 iburst' -e '/^server/d' /etc/chrony.conf systemctl restart chronyd

準備2個安裝包
mha4mysql-manager 和 mha4mysql-node

鏈接：https://pan.baidu.com/s/1lu0HPQDanJRotSZoVoPlHw
提取碼：pvt4

在Manager主機（192.168.99.101）節點上安裝兩個包，注意，yum源需要EPEL

yum -y localinstall mha4mysql-node-0.56-0.el6.noarch.rpm yum -y localinstall mha4mysql-manager-0.56-0.el6.noarch.rpm

在被管理節點(Master與2臺slave)安裝，注意，yum源需要EPEL

yum -y localinstall mha4mysql-node-0.56-0.el6.noarch.rpm

在管理節點建立配置文件

#新建目錄，用于存放配置文件 mkdir /etc/mastermha/ #創建mha的工作目錄 mkdir -p /data/mastermha/app1#配置文件可能不存在，直接新建,注意把注釋去了 vim /etc/mastermha/app1.cnf[server default]user=mhauser #管理帳號password=123 #密碼manager_workdir=/data/mastermha/app1/ #本地工作目錄manager_log=/data/mastermha/app1/manager.log #本地的日志remote_workdir=/data/mastermha/app1/ #遠程工作目錄ssh_user=root #SSH帳號repl_user=repluser #復制用帳號repl_password=123 #密碼ping_interval=1 #檢測周期[server1] #被管理的節點hostname=192.168.8.17 #被管理節點的IPcandidate_master=1 #可以當主服務器的優先級[server2]hostname=192.168.8.27candidate_master=1[server3]hostname=192.168.8.37

基于key的ssh驗證

#生成密鑰 ssh-keygen#復制給自己 ssh-copy-id 192.168.99.117#拷貝給其它服務器 scp -r .ssh 192.168.99.114:/root/ scp -r .ssh 192.168.99.115:/root/ scp -r .ssh 192.168.99.116:/root/

檢查連接

masterha_check_ssh --conf=/etc/mastermha/app1.cnf ... Tue Jul 16 09:54:35 2019 - [debug] ok. Tue Jul 16 09:54:36 2019 - [info] All SSH connection tests passed successfully.

檢查復制

masterha_check_repl --conf=/etc/mastermha/app1.cnf ... MySQL Replication Health is OK.

啟動，開始監控

masterha_manager --conf=/etc/mastermha/app1.cnf &Tue Jul 16 09:55:10 2019 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping. Tue Jul 16 09:55:10 2019 - [info] Reading application default configuration from /etc/mastermha/app1.cnf.. Tue Jul 16 09:55:10 2019 - [info] Reading server configuration from /etc/mastermha/app1.cnf..

Manager的監控是一次性的，當提升完新的主節點后，就完成了使命，程序就自動退出了。

---

proxySQL高可用方案

proxySQL_1

網絡配置

ip: 192.168.99.112 gateway: 192.168.99.254

時間同步

yum -y install chrony sed -i -e '1i\server 192.168.99.105 iburst' -e '/^server/d' /etc/chrony.conf systemctl restart chronyd

安裝前還得配置下官方的yum源
要不就自行下載安裝：https://github.com/sysown/proxysql/releases

vim /etc/yum.repos.d/proxysql.repo[proxysql_repo] name= ProxySQL YUM repository baseurl=http://repo.proxysql.com/ProxySQL/proxysql-1.4.x/centos/\$releasever gpgcheck=1 gpgkey=http://repo.proxysql.com/ProxySQL/repo_pub_key

安裝proxySQL

yum clean all yum -y install proxysql mariadb

啟動proxySQL

systemctl start proxysql

登錄到proxysql試試

mysql -uadmin -padmin -P6032 -h127.0.0.1#添加你的3臺MySQL主機 MySQL > insert into mysql_servers(hostgroup_id,hostname,port) values(10,'192.168.99.114',3306);MySQL > insert into mysql_servers(hostgroup_id,hostname,port) values(10,'192.168.99.115',3306);MySQL > insert into mysql_servers(hostgroup_id,hostname,port) values(10,'192.168.99.116',3306);

添加上了，可以看看

MySQL > select * from mysql_servers;

當然，這2步少不了。加載并保存到磁盤

MySQL > load mysql servers to runtime; MySQL > save mysql servers to disk;

添加監控后端節點的用戶。后面ProxySQL通過每個節點的read_only值來自動調整它們是屬于讀組還是寫組

MySQL [(none)]> set mysql-monitor_username='monitor'; MySQL [(none)]> set mysql-monitor_password='123';

加載到RUNTIME，并保存到disk

MySQL [(none)]> load mysql variables to runtime; MySQL [(none)]> save mysql variables to disk;

查看監控連接是否正常的 (對connect指標的監控)：(如果connect_error的結果為NULL則表示正常,看最后幾條)

MySQL [(none)]> select * from mysql_server_connect_log;

設置分組信息

指定寫組的id為10，讀組的id為20

MySQL> insert into mysql_replication_hostgroups values(10,20,"test");

加載到RUNTIME生效并保存

MySQL> load mysql servers to runtime; MySQL> save mysql servers to disk;

Monitor模塊監控后端的read_only值，按照read_only的值將節點自動移動到讀/寫組

MySQL> select hostgroup_id,hostname,port,status,weight from mysql_servers; +--------------+----------------+------+--------+--------+ | hostgroup_id | hostname | port | status | weight | +--------------+----------------+------+--------+--------+ | 10 | 192.168.99.114 | 3306 | ONLINE | 1 | | 20 | 192.168.99.115 | 3306 | ONLINE | 1 | | 20 | 192.168.99.116 | 3306 | ONLINE | 1 | +--------------+----------------+------+--------+--------+

在ProxySQL配置，將用戶sqluser添加到mysql_users表中， default_hostgroup默認組設置為寫組10，當讀寫分離的路由規則不符合時，會訪問默認組的數據庫

MySQL> insert into mysql_users(username,password,default_hostgroup) values('sqluser','123',10);MySQL> insert into mysql_users(username,password,default_hostgroup) values('dzuser','123',10);

保存生效

MySQL> load mysql users to runtime; MySQL> save mysql users to disk;

在proxysql上配置路由規則，實現讀寫分離

MySQL> insert into mysql_query_rules (rule_id,active,match_digest,destination_hostgroup,apply)VALUES (1,1,'^SELECT.*FOR UPDATE$',10,1),(2,1,'^SELECT',20,1);

保存生效

MySQL> load mysql query rules to runtime; MySQL> save mysql query rules to disk;

---

proxySQL_2

網絡配置

ip: 192.168.99.113 gateway: 192.168.99.254

時間同步

yum -y install chrony sed -i -e '1i\server 192.168.99.105 iburst' -e '/^server/d' /etc/chrony.conf systemctl restart chronyd

步驟同上了，不重復了。

---

實現proxySQL高可用集群

回到原來的ka1與ka2：192.168.99.106與107

ka1上追加配置，注意，是追加

vim /etc/keepalived/keepalived.conf #配置成雙主模式，在后面追加 vrrp_instance VI_2 {state BACKUPinterface eth0virtual_router_id 22priority 80advert_int 1authentication {auth_type PASSauth_pass 456}unicast_src_ip 192.168.99.106unicast_peer {192.168.99.107}virtual_ipaddress {192.168.0.200 dev eth0 label eth0:2} } virtual_server 192.168.0.200 6033 {delay_loop 3lb_algo wrrlb_kind DRprotocol TCPreal_server 192.168.99.112 6033 {weight 1TCP_CHECK {connect_port 6033connect_timeout 5retry 3delay_before_retry 3}}real_server 192.168.99.113 6033 {weight 1TCP_CHECK {connect_port 6033connect_timeout 5retry 3delay_before_retry 3}} }

在ka2上配置，

vim /etc/keepalived/keepalived.conf #配置成雙主模式，在后面追加 vrrp_instance VI_2 {state MASTERinterface eth0virtual_router_id 22priority 100advert_int 1authentication {auth_type PASSauth_pass 456}unicast_src_ip 192.168.99.107unicast_peer {192.168.99.106}virtual_ipaddress {192.168.0.200 dev eth0 label eth0:2} } virtual_server 192.168.0.200 6033 {delay_loop 3lb_algo wrrlb_kind DRprotocol TCPreal_server 192.168.99.112 6033 {weight 1TCP_CHECK {connect_port 6033connect_timeout 5retry 3delay_before_retry 3}}real_server 192.168.99.113 6033 {weight 1TCP_CHECK {connect_port 6033connect_timeout 5retry 3delay_before_retry 3}} }

再到proxySQL設置VIP
proxySQL_1: 192.168.99.112
3. 設置ARP

echo "net.ipv4.conf.lo.arp_ignore = 1" >> /etc/sysctl.conf echo "net.ipv4.conf.all.arp_ignore = 1" >> /etc/sysctl.conf echo "net.ipv4.conf.lo.arp_announce = 2" >> /etc/sysctl.conf echo "net.ipv4.conf.all.arp_announce = 2" >> /etc/sysctl.conf sysctl -p

添加VIP

cd /etc/sysconfig/network-scripts/vim ifcfg-lo:0 #添加下面內容 DEVICE=lo:0 BOOTPROTO=static BROADCAST=192.168.0.255 NETWORK=192.168.0.0 IPADDR=192.168.0.200 NETMASK=255.255.255.255 ONBOOT=yes TYPE=Ethernet#重啟 systemctl restart network

proxySQL_2: 192.168.99.113
5. 設置ARP

echo "net.ipv4.conf.lo.arp_ignore = 1" >> /etc/sysctl.conf echo "net.ipv4.conf.all.arp_ignore = 1" >> /etc/sysctl.conf echo "net.ipv4.conf.lo.arp_announce = 2" >> /etc/sysctl.conf echo "net.ipv4.conf.all.arp_announce = 2" >> /etc/sysctl.conf sysctl -p

添加VIP

cd /etc/sysconfig/network-scripts/vim ifcfg-lo:0 #添加下面內容 DEVICE=lo:0 BOOTPROTO=static BROADCAST=192.168.0.255 NETWORK=192.168.0.0 IPADDR=192.168.0.200 NETMASK=255.255.255.255 ONBOOT=yes TYPE=Ethernet#重啟 systemctl restart network

---

NFS配置與discuz布署

ip配置

ip: 192.168.99.110 gateway: 192.168.99.254

時間同步

yum -y install chrony sed -i -e '1i\server 192.168.99.105 iburst' -e '/^server/d' /etc/chrony.conf systemctl restart chronyd

安裝nfs服務

yum install -y nfs-utils

配置一個共享目錄

mkdir /data/bbs#配置文件 vim /etc/exports/data/bbs 192.168.99.0/24(rw,all_squash,anonuid=997,anongid=995)

加載服務

exportfs -r #重啟服務 systemctl restart nfs-server

新建用戶

groupadd -g 995 apache useradd -r -u 997 -g 995 -s /sbin/nologin apache

下載discuz

wget http://download.comsenz.com/DiscuzX/3.3/Discuz_X3.3_SC_UTF8.zip

解壓&設置權限

#如果沒有unzip工具：yum -y install unzip unzip Discuz_X3.3_SC_UTF8.zip mv upload/* /data/bbs/ chown -R apache.apache /data/bbs

---

rsync + inotify實時同步

在NFS同步服務器上配置：192.168.99.111
0. ip配置

ip: 192.168.99.111 gateway: 192.168.99.254

時間同步

yum -y install chrony sed -i -e '1i\server 192.168.99.105 iburst' -e '/^server/d' /etc/chrony.conf systemctl restart chronyd

先安裝：

yum -y install rsync

改配置

vim /etc/rsyncd.confuid = rootgid = rootuse chroot = nomax connections = 0ignore errorsexclude = lost+found/log file = /var/log/rsyncd.logpid file = /var/run/rsyncd.pidlock file = /var/run/rsyncd.lockreverse lookup = nohosts allow = 192.168.99.0/24 [backup]path = /data/comment = backupread only = noauth users = rsyncusersecrets file = /etc/rsync.pass

服務器端生成驗證文件

echo "rsyncuser:123" > /etc/rsync.pass chmod 600 /etc/rsync.pass

服務器端啟動rsync服務

rsync --daemon systemctl start rsyncd

部署NFS服務，下載nfs-utils

yum -y install nfs-utils

準備備份的NFS服務，這樣利用rsync+inotify實時同步，讓本機做為NFS的備份服務器，當主NFS掛掉之后，起用本機。要實現還需要在web服務器上加一個監控腳本，腳本在后續。

mkdir /data/bbs# 新建用戶 groupadd -g 995 apache useradd -r -u 997 -g 995 -s /sbin/nologin apache# 給目錄權限 setfacl -R -m u:apache:rwx /data/bbs#配置文件 vim /etc/exports/data/bbs 192.168.99.0/24(rw,all_squash,anonuid=997,anongid=995)# 加載服務 exportfs -r# 重啟服務 systemctl restart nfs-server

---

NFS配置：192.168.99.110

密碼文件

echo "123" > /etc/rsync.pass chmod 600 /etc/rsync.pass

客戶端測試同步數據

yum -y install rsync inotify-tools rsync -avz --password-file=/etc/rsync.pass /data/ rsyncuser@192.168.99.111::backup

客戶端創建NFS_rsync.sh腳本，用來實時同步

#!/bin/bash SRC='/data/' DEST='rsyncuser@192.168.99.111::backup'inotifywait -mrq --timefmt '%Y-%m-%d %H:%M' --format '%T %w %f' -e create,delete,moved_to,close_write,attrib ${SRC} |while read DATE TIME DIR FILE;doFILEPATH=${DIR}${FILE}rsync -az --delete --password-file=/etc/rsync.pass $SRC $DEST && echo "At ${TIME} on ${DATE}, file $FILEPATH was backuped up via rsync" >> /var/log/changelist.logdone

后臺運行

chmod +x NFS_rsync.sh ./NFS_rsync.sh &

---

keepalived實現NFS高可用

VIP為192.168.99.99

在NFS：192.168.99.110

#安裝keepalived yum -y install keepalived

配置文件

vim /etc/keepalived/keepalived.conf! Configuration File for keepalivedglobal_defs {notification_email {root@localhost}notification_email_from root@localhostsmtp_server 127.0.0.1smtp_connect_timeout 30router_id NFS1vrrp_skip_check_adv_addr # vrrp_strictvrrp_garp_interval 0vrrp_gna_interval 0 }vrrp_instance VI_1 {state MASTERinterface eth0virtual_router_id 33priority 100advert_int 1authentication {auth_type PASSauth_pass 789}unicast_src_ip 192.168.99.110unicast_peer {192.168.99.111}virtual_ipaddress {192.168.99.99 dev eth0 label eth0:1} }

啟動

systemctl restart keepalived

在NFS同步服務器上：192.168.99.111

#安裝keepalived yum -y install keepalived

配置文件

vim /etc/keepalived/keepalived.conf! Configuration File for keepalivedglobal_defs {notification_email {root@localhost}notification_email_from root@localhostsmtp_server 127.0.0.1smtp_connect_timeout 30router_id NFS2vrrp_skip_check_adv_addr # vrrp_strictvrrp_garp_interval 0vrrp_gna_interval 0 }vrrp_instance VI_1 {state BACKUPinterface eth0virtual_router_id 33priority 80advert_int 1authentication {auth_type PASSauth_pass 789}unicast_src_ip 192.168.99.111unicast_peer {192.168.99.110}virtual_ipaddress {192.168.99.99 dev eth0 label eth0:1} }

啟動

systemctl restart keepalived

---

NFS的自動掛載

回到web_A: 192.168.99.108(web_B也要這么配置)

安裝nfs工具

yum -y install nfs-utils

掛載nfs

mkdir /data/bbs mount 192.168.99.99:/data/bbs /data/bbs

到這里，我們就成功的把web服務的文件存放在NFS服務器上，但NFS服務器存在單點故障問題，所以在上面我們利用keepalived可以實現高可用，當主NFS服務器故障時，可實現VIP漂移，但這樣會有個問題，主備切換后，web服務器需要重新掛載，無法到達雙機熱備的效果。

所以還需要寫個腳本來實現自動重載，auto_nfs.sh腳本如下：

#!/bin/bash # #*********************************************************** #Author: Jibill Chen #QQ: 417060833 #Date: 2019-08-10 #FileName： auto_nfs.sh #URL: https://thson.blog.csdn.net/ #Description： The test script #********************************************************** vip_dir="192.168.99.99:/data/bbs" nfs_dir="/data/bbs"n=0 while : dostat $nfs_dir &> /dev/null[ $? -ne 0 ] && let n+=1if [ $n -gt 3 ] ;thenumount $nfs_dirsleep 1mount $vip_dir $nfs_dirbreakfisleep 3 done

后臺運行

bash auto_nfs.sh &

傳給web_B：192.168.99.109

scp auto_nfs.sh 192.168.99.109:/root

在web_B上也配置NFS

#安裝nfs工具 yum -y install nfs-utils#掛載nfs mkdir /data/bbs mount 192.168.99.99:/data/bbs /data/bbs#后臺運行腳本 scp auto_nfs.sh 192.168.99.109:/root

還有discuz的虛擬主機還沒有完成。修改下配置文件
web_A與web_B都要配置

vim /etc/httpd/conf.d/discuz.conf<virtualhost *:80>documentroot /data/bbsservername bbs.jibill.com<directory /data/bbs>require all granted</directory> </virtualhost>

重啟服務

systemctl restart httpd

---

到此，我們完成了整個keepalived高可用集群的配置，現在來配置個discuz來驗證proxySQL的高可用。

注意：安裝discuz的過程中，先把web_A或者web_B關了一臺。否則在安裝的過程的會因為LVS調度而出現問題

總結

以上是生活随笔為你收集整理的keepalived高可用集群实战项目的全部內容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網站內容還不錯，歡迎將生活随笔推薦給好友。