簡介: 最新的Cloudera CDH5.0.0 beta版本已經支持RM的HA, 筆者為此簡單驗證了RM HA的功能. 后續將繼續分析其HA的原理,以及其與社區RM HA的區別.

集群部屬與RM failover功能性驗證

硬件準備

四臺機器, bj1, bj3, bj4, bj5 準備好相應的環境(包括ssh互通, java環境).

角色說明, bj1為rm1, bj3為rm2, bj4和bj4為slave.

Zookeeper部屬在bj1上.

Hadoop版本準備http://archive.cloudera.com/cdh5/cdh/5/ 下載相應的CDH5版本hadoop-2.2.0-cdh5.0.0-beta-1.tar.gz(包括部屬包和原代碼),然后部屬到每臺slave中.

Zookeeper安裝在bj1, 下載最新Zookeeper, 解壓后配置 conf/zoo.cfg文件, 然后啟動.

[yuling.sh@v125050024 ~]$ cd zookeeper-3.4.3/

[yuling.sh@v125050024 zookeeper-3.4.3]$ cp conf/zoo_sample.cfg conf/zoo.cfg

[yuling.sh@v125050024 zookeeper-3.4.3]$ bin/zkServer.sh start

配置文件準備,見附錄, 也可以參考(https://www.cloudera.com/content/cloudera-content/cloudera-docs/CDH5/latest/CDH5-High-Availability-Guide/cdh5hag_cfg_RM_HA.html).

?

首先啟動HDFS

bin/hadoop namenode –format

sbin/start-dfs.sh

網頁上查看Namenode:　 http://bj1:50070/dfshealth.jsp

啟動Yarn

rm1上啟動resourcemanager

sbin/yarn-daemon.sh start resourcemanager

rm2上啟動resourcemanager

sbin/yarn-daemon.sh start resourcemanager

?

slave啟動NodeManager

????sbin/yarn-daemons.sh start nodemanager

查看rm1和mr2的網頁. http://bj1:23188/cluster 和 http://bj3:23188/cluster 其中active RM的網頁可以查看, stanby的RM無法查看網頁.

注: 如果yarn.resourcemanager.ha.automatic-failover.enabled設置為false, 則需要手動設置其中一個RM為active,負責兩個RM都為standby.

提交一個sleep作業測試

bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.2.0-cdh5.0.0-beta-1.jar sleep -m 1000

然后可以到網頁上查看作業運行情況

在作業運行過程中kill掉active的RM進程, 這時候打開standby RM的網頁,可以看到剛才提交的作業繼續運行.

[yuling.sh@v125050024 hadoop-2.2.0-cdh5.0.0-beta-1]$ jps

31333 ResourceManager

31671 Jps

29502 NameNode

25375 QuorumPeerMain

[yuling.sh@v125050024 hadoop-2.2.0-cdh5.0.0-beta-1]$ kill 31333

?

結論: 上述幾步簡單驗證了Cloudera RM Auto Failover的功能。

?

附錄1

配置參考

etc/hadoop/slaves

bj4

bj5

etc/hadoop/hdfs-site.xml

<property>

<name>fs.default.name</name>

<value>hdfs://bj1:9000</value>

</property>

etc/hadoop/mapred-site.xml

<property>

<name>mapreduce.framework.name</name>

<value>yarn</value>

</property>

etc/hadoop/yarn-site.xml配置如下

除了yarn.resourcemanager.ha.id需要稍作修改外, 其它配置都可以一樣.

<!-- Resource Manager Configs -->

<property>

<name>yarn.resourcemanager.connect.retry-interval.ms</name>

<value>2000</value>

</property>

<property>

<name>yarn.resourcemanager.ha.enabled</name>

<value>true</value>

</property>

<property>

<name>yarn.resourcemanager.ha.automatic-failover.enabled</name>

<value>true</value>

</property>

<property>

<name>yarn.resourcemanager.ha.rm-ids</name>

<value>rm1,rm2</value>

</property>

<property>

<name>yarn.resourcemanager.ha.id</name>

<value>rm2</value> <!—注釋, rm1上配置為rm1, rm2上配置rm2-->

</property>

<property>

<name>yarn.resourcemanager.store.class</name>

<value>org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore</value>

</property>

<property>

<name>yarn.resourcemanager.zk.state-store.address</name>

<value>bj1:2181</value>

</property>

<property>

<name>ha.zookeeper.quorum</name>

<value>bj1:2181</value>

</property>

<property>

<name>yarn.resourcemanager.recovery.enabled</name>

<value>true</value>

</property>

<property>

<name>yarn.app.mapreduce.am.scheduler.connection.wait.interval-ms</name>

<value>5000</value>

</property>

<!-- RM1 configs -->

<property>

<name>yarn.resourcemanager.address.rm1</name>

<value>bj1:23140</value>

</property>

<property>

<name>yarn.resourcemanager.scheduler.address.rm1</name>

<value>bj1:23130</value>

</property>

<property>

<name>yarn.resourcemanager.webapp.address.rm1</name>

<value>bj1:23188</value>

</property>

<property>

<name>yarn.resourcemanager.resource-tracker.address.rm1</name>

<value>bj1:23125</value>

</property>

<property>

<name>yarn.resourcemanager.admin.address.rm1</name>

<value>bj1:23141</value>

</property>

<property>

<name>yarn.resourcemanager.ha.admin.address.rm1</name>

<value>bj1:23142</value>

</property>

<!-- RM2 configs -->

<property>

<name>yarn.resourcemanager.address.rm2</name>

<value>bj3:23140</value>

</property>

<property>

<name>yarn.resourcemanager.scheduler.address.rm2</name>

<value>bj3:23130</value>

</property>

<property>

<name>yarn.resourcemanager.webapp.address.rm2</name>

<value>bj3:23188</value>

</property>

<property>

<name>yarn.resourcemanager.resource-tracker.address.rm2</name>

<value>bj3:23125</value>

</property>

<property>

<name>yarn.resourcemanager.admin.address.rm2</name>

<value>bj3:23141</value>

</property>

<property>

<name>yarn.resourcemanager.ha.admin.address.rm2</name>

<value>bj3:23142</value>

</property>

<!-- Node Manager Configs -->

<property>

<description>Address where the localizer IPC is.</description>

<name>yarn.nodemanager.localizer.address</name>

<value>0.0.0.0:23344</value>

</property>

<property>

<description>NM Webapp address.</description>

<name>yarn.nodemanager.webapp.address</name>

<value>0.0.0.0:23999</value>

</property>

<property>

<name>yarn.nodemanager.aux-services</name>

<value>mapreduce_shuffle</value>

</property>

<property>

<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>

<value>org.apache.hadoop.mapred.ShuffleHandler</value>

</property>

<property>

<name>yarn.nodemanager.local-dirs</name>

<value>/tmp/pseudo-dist/yarn/local</value>

</property>

<property>

<name>yarn.nodemanager.log-dirs</name>

<value>/tmp/pseudo-dist/yarn/log</value>

</property>

<property>

<name>mapreduce.shuffle.port</name>

<value>23080</value>

</property>

轉載于:https://www.cnblogs.com/shenh062326/p/3529267.html

總結

以上是生活随笔為你收集整理的[RM HA 1] Cloudera CDH5 RM HA功能验证的全部內容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網站內容還不錯，歡迎將生活随笔推薦給好友。