目錄

• 1, 環(huán)境
• 2, Redis簡(jiǎn)介
• 3, Gearman簡(jiǎn)介
• 4, MySQL - Redis配合使用方案
• 4, 軟件安裝
• 5, MySQL UDF + Trigger同步數(shù)據(jù)到Gearman
• 6, 參考資料

1, 環(huán)境

CentOS, MySQL, Redis, Nodejs

2, Redis簡(jiǎn)介

Redis是一個(gè)開源的K-V內(nèi)存數(shù)據(jù)庫(kù)，它的key可以是string/set/hash/list/...，因?yàn)槭腔趦?nèi)存的，所在訪問速度相當(dāng)快。

3, Gearman簡(jiǎn)介

Gearman是一個(gè)開源的Map/Reduce分布式計(jì)算框架，具有豐富的client sdk，而且它支持MySQL UDF。

Gearman工作圖

Gearman調(diào)用流程

Gearman集群

從圖中可以看出貌似Gearman的集群功能比較弱，Job Server之間沒啥關(guān)系好像。

4, MySQL - Redis配合使用方案

首先我們以MySQL數(shù)據(jù)為主，將insert/update/delete交給MySQL，而select交給redis；
當(dāng)有數(shù)據(jù)發(fā)生變化時(shí)，通過MySQL Trigger實(shí)時(shí)異步調(diào)用Gearman的UDF提交一個(gè)job給Job Server，當(dāng)job執(zhí)行的時(shí)候會(huì)去更新redis；從而保證redis與MySQL中的數(shù)據(jù)是同步的。

4, 軟件安裝

安裝gearman

//安裝依賴 $ yum install -y boost-devel gperf libevent-devel libuuid-devel//下載gearman $ wget https://launchpad.net/gearmand/1.2/1.1.12/+download/gearmand-1.1.12.tar.gz $ tar zxvf gearmand-1.1.12.tar.gz//編譯安裝，指定mysqlclient的鏈接路徑 $ ./configure LDFLAGS=-L/usr/lib64/mysql/ $ make $ make install//啟動(dòng)gearmand服務(wù)端 $ /usr/local/sbin/gearmand -l /var/log/gearmand.log -d//查看是否安裝成功，查看gearman版本 $ /usr/local/sbin/gearmand -V

gearman client

gearman client有很多種：PHP, Perl, Python, Ruby, Nodejs, Go, Java, C#...，我們選擇GearmaNode(Nodejs庫(kù))作為實(shí)驗(yàn)用的gearman client

Nodejs的安裝省略

GearmaNode的安裝及測(cè)試

安裝GearmaNode

//安裝GearmaNode $ npm install gearmanode

啟動(dòng)Gearman Job Server

//確保gearman的job server已經(jīng)啟動(dòng), job server默認(rèn)監(jiān)聽4730端口 $ netstat -nplt | grep 4730//如果沒有，則啟動(dòng)job server（-d表示啟動(dòng), -l指定log的位置） $ /usr/local/sbin/gearmand -l /var/log/gearmand.log -d

worker.js (Gearman Job Worker)

var gearmanode = require('gearmanode'); var worker = gearmanode.worker(); //對(duì)job client傳遞過來的字符串執(zhí)行reverse倒序操作 worker.addFunction('reverse', function (job) {job.sendWorkData(job.payload); // mirror input as partial resultjob.workComplete(job.payload.toString().split("").reverse().join("")); }); //啟動(dòng)job worker $ node worker.js

client.js (Gearman Job Client)

var gearmanode = require('gearmanode'); var client = gearmanode.client(); //提交job var job = client.submitJob('reverse', 'hello world!'); job.on('workData', function(data) {console.log('WORK_DATA >>> ' + data); }); //結(jié)果回調(diào)函數(shù) job.on('complete', function() {console.log('RESULT >>> ' + job.response);client.close(); }); //執(zhí)行client.js $ node client.js //查看控制臺(tái)打印輸出

WORK_DATA >>> hello world!
RESULT >>> !dlrow olleh

可以看到gearman安裝成功，運(yùn)行正常。

5, MySQL UDF + Trigger同步數(shù)據(jù)到Gearman

安裝lib_mysqludf_json

lib_mysqludf_json可以把MySQL表的數(shù)據(jù)以json數(shù)據(jù)格式輸出

//下載lib_mysqludf_json $ git clone https://github.com/mysqludf/lib_mysqludf_json.git $ cd lib_mysqludf_json//編譯 $ gcc $(mysql_config --cflags) -shared -fPIC -o lib_mysqludf_json.so lib_mysqludf_json.c//拷貝lib_mysqludf_json.so到MySQL的plugin目錄 //可以登陸MySQL，輸入命令"show variables like '%plugin%'"查看plugin位置 $ cp lib_mysqludf_json.so /usr/lib64/mysql/plugin///演示lib_mysqludf_json功能 $ mysql -uname -hhost -ppwd //首先注冊(cè)UDF函數(shù) mysql> CREATE FUNCTION json_object RETURNS STRING SONAME "lib_mysqludf_json.so"; //json_array|json_members|json_values函數(shù)注冊(cè)方式與json_object一樣. mysql> use test; mysql> select * from user_list; +------+----------+ | NAME | PASSWORD | +------+----------+ | troy | pwd | +------+----------+ mysql> select json_object(name,password) as user from user_list; +----------------------------------+ | user | +----------------------------------+ | {"name":"troy","password":"pwd"} | +----------------------------------+

安裝gearman-mysql-udf

//下載 $ wget https://launchpad.net/gearman-mysql-udf/trunk/0.6/+download/gearman-mysql-udf-0.6.tar.gz $ tar zxvf gearman-mysql-udf-0.6.tar.gz $ cd gearman-mysql-udf-0.6//安裝libgearman-devel $ yum install libgearman-devel//編譯安裝 $ ./configure --with-mysql=/usr/bin/mysql_config --libdir=/usr/lib64/mysql/plugin/ $ make && make install//登錄MySQL注冊(cè)UDF函數(shù) mysql> CREATE FUNCTION gman_do_background RETURNS STRINGSONAME "libgearman_mysql_udf.so"; mysql> CREATE FUNCTION gman_servers_set RETURNS STRINGSONAME "libgearman_mysql_udf.so"; //函數(shù)gman_do|gman_do_high|gman_do_low|gman_do_high_background|gman_do_low_background|gman_sum注冊(cè)方式類似，請(qǐng)參考gearman-mysql-udf-0.6/README//指定gearman job server地址 mysql> SELECT gman_servers_set('127.0.0.1:4730');

如果出現(xiàn)異常信息：
ERROR 1126 (HY000): Can't open shared library 'libgearman_mysql_udf.so' (errno: 11 libgearman.so.8: cannot open shared object file: No such file or directory)
表示系統(tǒng)找不到 libgearman.so 文件，一般so都在/usr/local/lib目錄下，修改配置文件/etc/ld.so.conf，將/usr/local/lib目錄加入進(jìn)去即可:

$ cat /etc/ld.so.conf include ld.so.conf.d/*.conf /usr/local/lib $ /sbin/ldconfig -v | grep gearman*

MySQL Trigger調(diào)用Gearman UDF實(shí)現(xiàn)同步

DELIMITER $$ CREATE TRIGGER user_list_data_to_redis AFTER UPDATE ON user_listFOR EACH ROW BEGINSET @ret=gman_do_background('syncToRedis', json_object(NEW.name as 'name', NEW.password as 'password')); END$$ DELIMITER ;

Gearman Nodejs Worker將數(shù)據(jù)異步復(fù)制到redis

redis的安裝步驟省略
啟動(dòng)redis-server

$ redis-server

安裝redis的Nodejs client:

npm install hiredis redis

redis-worker.js (Gearman Nodejs worker)

var redis = require("redis"); var gearmanode = require('gearmanode'); var worker = gearmanode.worker();//添加gearman函數(shù)syncToRedis //當(dāng)MySQL表記錄更改時(shí)，此函數(shù)會(huì)被調(diào)用 worker.addFunction('syncToRedis', function (job) {job.sendWorkData(job.payload);console.log("-------job.payload: " + job.payload.toString());//將字符串轉(zhuǎn)換成json object, 然后調(diào)用更新redisupdateRedis(eval('(' + job.payload.toString() + ')'));job.workComplete("Successed!"); });//些函數(shù)只是簡(jiǎn)單的將MySQL表中的一行的記錄按單個(gè)字段更新到redis中。可根據(jù)實(shí)際情況自行擴(kuò)展 function updateRedis(json) {var client = redis.createClient();client.on("error", function (err) {console.log("Error " + err);});for(var key in json){client.set(key, json[key], redis.print);}client.quit(); }

啟動(dòng)worder:

node redis-worker.js

更新MySQL

mysql> select * from user_list; +------+----------+ | NAME | PASSWORD | +------+----------+ | troy | jane | +------+----------+ mysql> update user_list set name='hello', password='world';

redis-worker控制臺(tái)輸出：

-------job.payload: {"name":"hello","password":"world"}

查看redis數(shù)據(jù)

$ redis-cli 127.0.0.1:6379> keys * 1) "password" 2) "name" 127.0.0.1:6379> get name "hello" 127.0.0.1:6379> get password "world"

可以看到只要MySQL中user_list表的數(shù)據(jù)被update了，redis中的數(shù)據(jù)也會(huì)被更新。

6, 參考資料

Gearman+PHP+MySQL UDF的組合異步實(shí)現(xiàn)MySQL到Redis的數(shù)據(jù)復(fù)制

轉(zhuǎn)載于:https://www.cnblogs.com/java-koma/p/4280769.html

總結(jié)

以上是生活随笔為你收集整理的Gearman + Nodejs + MySQL UDF异步实现 MySQL 到 Redis 的数据同步的全部?jī)?nèi)容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網(wǎng)站內(nèi)容還不錯(cuò)，歡迎將生活随笔推薦給好友。