當(dāng)前位置：首頁(yè) > 编程资源 > 编程问答 >内容正文

编程问答

Memcached - In Action

發(fā)布時(shí)間：2025/3/18 编程问答 14 豆豆

生活随笔收集整理的這篇文章主要介紹了 Memcached - In Action 小編覺(jué)得挺不錯(cuò)的,現(xiàn)在分享給大家,幫大家做個(gè)參考.

Memcached

標(biāo)簽： Java與NoSQL

With Java

比較知名的Java Memcached客戶(hù)端有三款:Java-Memcached-Client、XMemcached以及Spymemcached, 其中以XMemcached性能最好, 且維護(hù)較穩(wěn)定/版本較新:

<dependency><groupId>com.googlecode.xmemcached</groupId><artifactId>xmemcached</artifactId><version>2.0.0</version> </dependency>

XMemcached以及其他兩款Memcached客戶(hù)端的詳細(xì)信息可參考博客XMemcached-一個(gè)新的開(kāi)源Java memcached客戶(hù)端、Java幾個(gè)Memcached連接客戶(hù)端對(duì)比選擇.

實(shí)踐

任何技術(shù)都有其最適用的場(chǎng)景,只有在合適的場(chǎng)景下,才能發(fā)揮最好的效果.Memcached使用內(nèi)存讀寫(xiě)數(shù)據(jù),速度比DB和文件系統(tǒng)快得多, 因此,Memcached的常用場(chǎng)景有:

緩存DB查詢(xún)數(shù)據(jù): 作為緩存“保護(hù)”數(shù)據(jù)庫(kù), 防止頻繁的讀寫(xiě)帶給DB過(guò)大的壓力;
中繼MySQL主從延遲: 利用其“讀寫(xiě)快”特點(diǎn)實(shí)現(xiàn)主從數(shù)據(jù)庫(kù)的消息同步.

緩存DB查詢(xún)數(shù)據(jù)

通過(guò)Memcached緩存數(shù)據(jù)庫(kù)查詢(xún)結(jié)果,減少DB訪(fǎng)問(wèn)次數(shù),以提高動(dòng)態(tài)Web應(yīng)用響應(yīng)速度:

JDBC模擬Memcached緩存DB數(shù)據(jù):

/*** @author jifang.* @since 2016/6/13 20:08.*/ public class MemcachedDAO {private static final int _1M = 60 * 1000;private static final DataSource dataSource;private static final MemcachedClient mc;static {Properties properties = new Properties();try {properties.load(ClassLoader.getSystemResourceAsStream("db.properties"));} catch (IOException ignored) {}/** 初始化連接池 **/HikariConfig config = new HikariConfig();config.setDriverClassName(properties.getProperty("mysql.driver.class"));config.setJdbcUrl(properties.getProperty("mysql.url"));config.setUsername(properties.getProperty("mysql.user"));config.setPassword(properties.getProperty("mysql.password"));config.setMaximumPoolSize(Integer.valueOf(properties.getProperty("pool.max.size")));config.setMinimumIdle(Integer.valueOf(properties.getProperty("pool.min.size")));config.setIdleTimeout(Integer.valueOf(properties.getProperty("pool.max.idle_time")));config.setMaxLifetime(Integer.valueOf(properties.getProperty("pool.max.life_time")));dataSource = new HikariDataSource(config);/** 初始化Memcached **/try {mc = new XMemcachedClientBuilder(properties.getProperty("memcached.servers")).build();} catch (IOException e) {throw new RuntimeException(e);}}public List<Map<String, Object>> executeQuery(String sql) {List<Map<String, Object>> result;try {/** 首先請(qǐng)求MC **/String key = sql.replace(' ', '-');result = mc.get(key);// 如果key未命中, 再請(qǐng)求DBif (result == null || result.isEmpty()) {ResultSet resultSet = dataSource.getConnection().createStatement().executeQuery(sql);/** 獲得列數(shù)/列名 **/ResultSetMetaData meta = resultSet.getMetaData();int columnCount = meta.getColumnCount();List<String> columnName = new ArrayList<>();for (int i = 1; i <= columnCount; ++i) {columnName.add(meta.getColumnName(i));}/** 填充實(shí)體 **/result = new ArrayList<>();while (resultSet.next()) {Map<String, Object> entity = new HashMap<>(columnCount);for (String name : columnName) {entity.put(name, resultSet.getObject(name));}result.add(entity);}/** 寫(xiě)入MC **/mc.set(key, _1M, result);}} catch (TimeoutException | InterruptedException | MemcachedException | SQLException e) {throw new RuntimeException(e);}return result;}public static void main(String[] args) {MemcachedDAO dao = new MemcachedDAO();List<Map<String, Object>> execute = dao.executeQuery("select * from orders");System.out.println(execute);} }

注: 代碼僅供展示DB緩存思想,因?yàn)橐话沩?xiàng)目很少會(huì)直接使用JDBC操作DB,而是會(huì)選用像MyBatis之類(lèi)的ORM框架代替之,而這類(lèi)框架框架一般也會(huì)開(kāi)放接口出來(lái)實(shí)現(xiàn)與緩存產(chǎn)品的整合(如MyBatis開(kāi)放出一個(gè)org.apache.ibatis.cache.Cache接口,通過(guò)實(shí)現(xiàn)該接口,可將Memcached與MyBatis整合, 細(xì)節(jié)可參考博客MyBatis與Memcached集成.

中繼MySQL主從延遲

MySQL在做replication時(shí),主從復(fù)制時(shí)會(huì)由一段時(shí)間延遲,尤其是主從服務(wù)器分處于異地機(jī)房時(shí),這種情況更加明顯.FaceBook官方的一篇技術(shù)文章提到:其加州的數(shù)據(jù)中心到弗吉尼亞州數(shù)據(jù)中心的主從同步延遲達(dá)到70MS. 考慮以下場(chǎng)景:

用戶(hù)U購(gòu)買(mǎi)電子書(shū)B:insert into Master (U,B);
用戶(hù)U觀(guān)看電子書(shū)B:select 購(gòu)買(mǎi)記錄 [user='A',book='B'] from Slave.
由于主從延遲的存在,第②步中無(wú)記錄,用戶(hù)無(wú)權(quán)觀(guān)看該書(shū).

此時(shí)可以利用Memcached在Master與Slave之間做過(guò)渡:

用戶(hù)U購(gòu)買(mǎi)電子書(shū)B:memcached->add('U:B',true);
主數(shù)據(jù)庫(kù): insert into Master (U,B);
用戶(hù)U觀(guān)看電子書(shū)B: select 購(gòu)買(mǎi)記錄 [user='U',book='B'] from Slave;
如果沒(méi)查詢(xún)到,則memcached->get('U:B'),查到則說(shuō)明已購(gòu)買(mǎi)但有主從延遲.
如果Memcached中也沒(méi)查詢(xún)到,用戶(hù)無(wú)權(quán)觀(guān)看該書(shū).

分布式緩存

Memcached雖然名義上是分布式緩存,但其自身并未實(shí)現(xiàn)分布式算法.當(dāng)一個(gè)請(qǐng)求到達(dá)時(shí),需要由客戶(hù)端實(shí)現(xiàn)的分布式算法將不同的key路由到不同的Memcached服務(wù)器中.而分布式取模算法有著致命的缺陷(詳細(xì)可參考分布式之取模算法的缺陷), 因此Memcached客戶(hù)端一般采用一致性Hash算法來(lái)保證分布式.

目標(biāo):
- key的分布盡量均勻;
- 增/減服務(wù)器節(jié)點(diǎn)對(duì)于其他節(jié)點(diǎn)的影響盡量小.

一致性Hash算法

首先開(kāi)辟一塊非常大的空間(如圖中:0~2³²),然后將所有的數(shù)據(jù)使用hash函數(shù)(如MD5、Ketama等)映射到這個(gè)空間內(nèi),形成一個(gè)Hash環(huán). 當(dāng)有數(shù)據(jù)需要存儲(chǔ)時(shí),先得到一個(gè)hash值對(duì)應(yīng)到hash環(huán)上的具體位置(如k1),然后沿順時(shí)針?lè)较蛘业揭慌_(tái)機(jī)器(如B),將k1存儲(chǔ)到B這個(gè)節(jié)點(diǎn)中:
如果B節(jié)點(diǎn)宕機(jī),則B上的所有負(fù)載就會(huì)落到C節(jié)點(diǎn)上：
這樣,只會(huì)影響C節(jié)點(diǎn)，對(duì)其他的節(jié)點(diǎn)如A、D的數(shù)據(jù)都不會(huì)造成影響. 然而,這樣又會(huì)帶來(lái)一定的風(fēng)險(xiǎn),由于B節(jié)點(diǎn)的負(fù)載全部由C節(jié)點(diǎn)承擔(dān),C節(jié)點(diǎn)的負(fù)載會(huì)變得很高,因此C節(jié)點(diǎn)又會(huì)很容易宕機(jī),依次下去會(huì)造成整個(gè)集群的不穩(wěn)定.
理想的情況下是當(dāng)B節(jié)點(diǎn)宕機(jī)時(shí),將原先B節(jié)點(diǎn)上的負(fù)載平均的分擔(dān)到其他的各個(gè)節(jié)點(diǎn)上. 為此,又引入了“虛擬節(jié)點(diǎn)”的概念: 想象在這個(gè)環(huán)上有很多“虛擬節(jié)點(diǎn)”,數(shù)據(jù)的存儲(chǔ)是沿著環(huán)的順時(shí)針?lè)较蛘乙粋€(gè)虛擬節(jié)點(diǎn),每個(gè)虛擬節(jié)點(diǎn)都會(huì)關(guān)聯(lián)到一個(gè)真實(shí)節(jié)點(diǎn),但一個(gè)真實(shí)節(jié)點(diǎn)會(huì)對(duì)應(yīng)多個(gè)虛擬節(jié)點(diǎn),且不同真實(shí)節(jié)點(diǎn)的多個(gè)虛擬節(jié)點(diǎn)是交差分布的:

圖中A1、A2、B1、B2、C1、C2、D1、D2 都是“虛擬節(jié)點(diǎn)”,機(jī)器A負(fù)責(zé)存儲(chǔ)A1、A2的數(shù)據(jù), 機(jī)器B負(fù)責(zé)存儲(chǔ)B1、B2的數(shù)據(jù)… 只要虛擬節(jié)點(diǎn)數(shù)量足夠多且分布均勻,當(dāng)其中一臺(tái)機(jī)器宕機(jī)之后,原先機(jī)器上的負(fù)載就會(huì)平均分配到其他所有機(jī)器上(如圖中節(jié)點(diǎn)B宕機(jī),其負(fù)載會(huì)分擔(dān)到節(jié)點(diǎn)A和節(jié)點(diǎn)D上).

Java實(shí)現(xiàn)

/*** @author jifang.* @since 2016/6/5 11:55.*/ public class ConsistentHash<Node> {/*** 虛擬節(jié)點(diǎn)-真實(shí)節(jié)點(diǎn)Map*/public SortedMap<Long, Node> VRNodesMap = new TreeMap<>();/*** 虛擬節(jié)點(diǎn)數(shù)目*/private int vCount = 50;/*** 真實(shí)節(jié)點(diǎn)數(shù)目*/private int rCount = 0;public ConsistentHash() {}public ConsistentHash(int vCount) {this.vCount = vCount;}public ConsistentHash(List<Node> rNodes) {init(rNodes);}public ConsistentHash(List<Node> rNodes, int vCount) {this.vCount = vCount;init(rNodes);}private void init(List<Node> rNodes) {if (rNodes != null) {for (Node node : rNodes) {add(rCount, node);++rCount;}}}public void addRNode(Node rNode) {add(rCount, rNode);++rCount;}public void rmRNode(Node rNode) {--rCount;remove(rCount, rNode);}public Node getRNode(String key) {// 沿環(huán)的順時(shí)針找到一個(gè)虛擬節(jié)點(diǎn)SortedMap<Long, Node> tailMap = VRNodesMap.tailMap(hash(key));if (tailMap.size() == 0) {return VRNodesMap.get(VRNodesMap.firstKey());}return tailMap.get(tailMap.firstKey());}private void add(int rIndex, Node rNode) {for (int j = 0; j < vCount; ++j) {VRNodesMap.put(hash(String.format("RNode-%s-VNode-%s", rIndex, j)), rNode);}}private void remove(int rIndex, Node rNode) {for (int j = 0; j < vCount; ++j) {VRNodesMap.remove(hash(String.format("RNode-%s-VNode-%s", rIndex, j)));}}/*** MurMurHash算法，是非加密HASH算法，性能很高，* 比傳統(tǒng)的CRC32,MD5，SHA-1（這兩個(gè)算法都是加密HASH算法，復(fù)雜度本身就很高，帶來(lái)的性能上的損害也不可避免）* 等HASH算法要快很多，而且據(jù)說(shuō)這個(gè)算法的碰撞率很低.* http://murmurhash.googlepages.com/*/private Long hash(String key) {ByteBuffer buf = ByteBuffer.wrap(key.getBytes());int seed = 0x1234ABCD;ByteOrder byteOrder = buf.order();buf.order(ByteOrder.LITTLE_ENDIAN);long m = 0xc6a4a7935bd1e995L;int r = 47;long h = seed ^ (buf.remaining() * m);long k;while (buf.remaining() >= 8) {k = buf.getLong();k *= m;k ^= k >>> r;k *= m;h ^= k;h *= m;}if (buf.remaining() > 0) {ByteBuffer finish = ByteBuffer.allocate(8).order(ByteOrder.LITTLE_ENDIAN);// for big-endian version, do this first:// finish.position(8-buf.remaining());finish.put(buf).rewind();h ^= finish.getLong();h *= m;}h ^= h >>> r;h *= m;h ^= h >>> r;buf.order(byteOrder);return h;} }

測(cè)試

public class ConsistentHashMain {private static final int KEY_COUNT = 1000;@Testpublic void test() {ConsistentHash<String> nodes = new ConsistentHash<>(new ArrayList<String>(), 50);nodes.addRNode("10.45.156.11");nodes.addRNode("10.45.156.12");nodes.addRNode("10.45.156.13");nodes.addRNode("10.45.156.14");nodes.addRNode("10.45.156.15");nodes.addRNode("10.45.156.16");nodes.addRNode("10.45.156.17");nodes.addRNode("10.45.156.18");nodes.addRNode("10.45.156.19");nodes.addRNode("10.45.156.10");Map<String, String> map = new HashMap<>();initMap(map, nodes);// 刪除節(jié)點(diǎn)nodes.rmRNode("10.45.156.19");// 增加節(jié)點(diǎn)nodes.addRNode("10.45.156.20");int mis = 0;for (Map.Entry<String, String> entry : map.entrySet()) {String key = entry.getKey();String value = entry.getValue();if (!nodes.getRNode(key).equals(value)) {++mis;}}System.out.println(String.format("當(dāng)前命中率為：%s%%", (KEY_COUNT - mis) * 100.0 / KEY_COUNT));}private void initMap(Map<String, String> map, ConsistentHash<String> nodes) {for (int i = 0; i < KEY_COUNT; ++i) {String key = String.format("key-%s", i);map.put(key, nodes.getRNode(key));}} }

經(jīng)過(guò)實(shí)際測(cè)試: 當(dāng)有十臺(tái)真實(shí)節(jié)點(diǎn),而每個(gè)真實(shí)節(jié)點(diǎn)有50個(gè)虛擬節(jié)點(diǎn)時(shí),在發(fā)生一臺(tái)實(shí)際節(jié)點(diǎn)宕機(jī)/新增一臺(tái)節(jié)點(diǎn)的情況時(shí),命中率仍然能夠達(dá)到90%左右.對(duì)比簡(jiǎn)單取模Hash算法:

當(dāng)節(jié)點(diǎn)從N到N-1時(shí),緩存的命中率直線(xiàn)下降為1/N(N越大,命中率越低);一致性Hash的表現(xiàn)就優(yōu)秀多了:

命中率只下降為原先的 (N-1)/N ,且服務(wù)器節(jié)點(diǎn)越多,性能越好.因此一致性Hash算法可以最大限度地減小服務(wù)器增減時(shí)的緩存重新分布帶來(lái)的壓力.

XMemcached實(shí)現(xiàn)

實(shí)際上XMemcached客戶(hù)端自身實(shí)現(xiàn)了很多一致性Hash算法(KetamaMemcachedSessionLocator/PHPMemcacheSessionLocator), 因此在開(kāi)發(fā)中沒(méi)有必要自己去實(shí)現(xiàn):

示例: 支持分布式的MemcachedFilter:

/*** @author jifang.* @since 2016/5/21 15:50.*/ public class MemcachedFilter implements Filter {private MemcachedClient memcached;private static final int _1MIN = 60;@Overridepublic void init(FilterConfig filterConfig) throws ServletException {try {MemcachedClientBuilder builder = new XMemcachedClientBuilder(AddrUtil.getAddresses("10.45.156.11:11211" +"10.45.156.12:11211" +"10.45.156.13:11211"));builder.setSessionLocator(new KetamaMemcachedSessionLocator());memcached = builder.build();} catch (IOException e) {throw new RuntimeException(e);}}@Overridepublic void doFilter(ServletRequest req, ServletResponse response, FilterChain chain) throws IOException, ServletException {// 對(duì)PrintWriter包裝MemcachedWriter mWriter = new MemcachedWriter(response.getWriter());chain.doFilter(req, new MemcachedResponse((HttpServletResponse) response, mWriter));HttpServletRequest request = (HttpServletRequest) req;String key = request.getRequestURI();Enumeration<String> names = request.getParameterNames();if (names.hasMoreElements()) {String name = names.nextElement();StringBuilder sb = new StringBuilder(key).append("?").append(name).append("=").append(request.getParameter(name));while (names.hasMoreElements()) {name = names.nextElement();sb.append("&").append(name).append("=").append(request.getParameter(name));}key = sb.toString();}try {String rspContent = mWriter.getRspContent();memcached.set(key, _1MIN, rspContent);} catch (TimeoutException | InterruptedException | MemcachedException e) {throw new RuntimeException(e);}}@Overridepublic void destroy() {}private static class MemcachedWriter extends PrintWriter {private StringBuilder sb = new StringBuilder();private PrintWriter writer;public MemcachedWriter(PrintWriter out) {super(out);this.writer = out;}@Overridepublic void print(String s) {sb.append(s);this.writer.print(s);}public String getRspContent() {return sb.toString();}}private static class MemcachedResponse extends HttpServletResponseWrapper {private PrintWriter writer;public MemcachedResponse(HttpServletResponse response, PrintWriter writer) {super(response);this.writer = writer;}@Overridepublic PrintWriter getWriter() throws IOException {return this.writer;}} }

以上代碼最好有Nginx的如下配置支持:

Nginx以前端請(qǐng)求的"URI+Args"作為key去請(qǐng)求Memcached,如果key命中,則直接由Nginx從緩存中取出數(shù)據(jù)響應(yīng)前端;未命中,則產(chǎn)生404異常,Nginx捕獲之并將request提交后端服務(wù)器.在后端服務(wù)器中,request被MemcachedFilter攔截, 待業(yè)務(wù)邏輯執(zhí)行完, 該Filter會(huì)將Response的數(shù)據(jù)拿到并寫(xiě)入Memcached, 以備下次直接響應(yīng).

參考:

緩存系統(tǒng)MemCached的Java客戶(hù)端優(yōu)化歷程

memcached Java客戶(hù)端spymemcached的一致性Hash算法

一致性哈希算法及其在分布式系統(tǒng)中的應(yīng)用

陌生但默默一統(tǒng)江湖的MurmurHash

Hash 函數(shù)概覽

總結(jié)

以上是生活随笔為你收集整理的Memcached - In Action的全部?jī)?nèi)容，希望文章能夠幫你解決所遇到的問(wèn)題。

如果覺(jué)得生活随笔網(wǎng)站內(nèi)容還不錯(cuò)，歡迎將生活随笔推薦給好友。

上一篇：飞机游戏感悟
下一篇： OSChina 周三乱弹 ——相亲妹子说