六、YARN資源管理與調(diào)度策略

6.1、YARN資源管理

MapReduce任務(wù)/資源流程：

---

由5.1章節(jié)我們可知，當(dāng)把yarn.scheduler.capacity.maximum-am-resource-percent參數(shù)調(diào)整為0.2 時，我們可以同時運(yùn)行2個任務(wù)，此時Application Master最大資源為5120M（每個任務(wù)啟動的AppMaster資源為2048M）

當(dāng)參數(shù)調(diào)整為0.3時，可同時運(yùn)行4個任務(wù)

6.2、YARN調(diào)度策略

Yarn的資源調(diào)度器是可配置的，Yarn定義了一套接口規(guī)范(接口ResourceScheduler)，用戶可以按照自己的需求實(shí)現(xiàn)這個接口中的方法。

Yarn自身自帶了三種調(diào)度器，分別是：FIFO Scheduler、Capacity Scheduler和Fair Scheduler。

Yarn的資源調(diào)度器默認(rèn)使用Capacity Scheduler，但是默認(rèn)只有一個root.default隊列。

調(diào)整Yarn資源隊列：

添加dev,prod 隊列，刪除default隊列；

修改default隊列配置為dev隊列，并修改其capacity為40 ， maxium-capacity 為60 ；

復(fù)制dev隊列屬性，并修改為prod隊列，修改其capacity為60 ， maxium-capacity 為80 ；

重啟YARN集群；

cd /opt/hadoop-3.1.4/etc/hadoop vi capacity-scheduler.xml

修改如下：

修改完后，配置prod，在dev末尾添加：

<!-- config prod queue --><property><name>yarn.scheduler.capacity.root.prod.capacity</name><value>60</value><description>Default queue target capacity.</description></property><property><name>yarn.scheduler.capacity.root.prod.user-limit-factor</name><value>1</value><description>Default queue user limit a percentage from 0.0 to 1.0.</description></property><property><name>yarn.scheduler.capacity.root.prod.maximum-capacity</name><value>80</value><description>The maximum capacity of the default queue. </description></property><property><name>yarn.scheduler.capacity.root.prod.state</name><value>RUNNING</value><description>The state of the default queue. State can be one of RUNNING or STOPPED.</description></property><property><name>yarn.scheduler.capacity.root.prod.acl_submit_applications</name><value>*</value><description>The ACL of who can submit jobs to the default queue.</description></property><property><name>yarn.scheduler.capacity.root.prod.acl_administer_queue</name><value>*</value><description>The ACL of who can administer jobs on the default queue.</description></property><property><name>yarn.scheduler.capacity.root.prod.acl_application_max_priority</name><value>*</value><description>The ACL of who can submit applications with configured priority.For e.g, [user={name} group={name} max_priority={priority} default_priority={priority}]</description></property><property><name>yarn.scheduler.capacity.root.prod.maximum-application-lifetime</name><value>-1</value><description>Maximum lifetime of an application which is submitted to a queuein seconds. Any value less than or equal to zero will be considered asdisabled.This will be a hard time limit for all applications in thisqueue. If positive value is configured then any application submittedto this queue will be killed after exceeds the configured lifetime.User can also specify lifetime per application basis inapplication submission context. But user lifetime will beoverridden if it exceeds queue maximum lifetime. It is point-in-timeconfiguration.Note : Configuring too low value will result in killing applicationsooner. This feature is applicable only for leaf queue.</description></property><property><name>yarn.scheduler.capacity.root.prod.default-application-lifetime</name><value>-1</value><description>Default lifetime of an application which is submitted to a queuein seconds. Any value less than or equal to zero will be considered asdisabled.If the user has not submitted application with lifetime value then thisvalue will be taken. It is point-in-time configuration.Note : Default lifetime can't exceed maximum lifetime. This feature isapplicable only for leaf queue.</description></property>

配置prod完后，保存、退出，拷貝到node1、node2、node3下：

scp capacity-scheduler.xml node1:/opt/hadoop-3.1.4/etc/hadoop/ scp capacity-scheduler.xml node2:/opt/hadoop-3.1.4/etc/hadoop/ scp capacity-scheduler.xml node3:/opt/hadoop-3.1.4/etc/hadoop/

最后，重啟下集群(進(jìn)入sbin目錄)

./stop-yarn.sh ./start-yarn.sh

啟動成功

最后，我們查看下網(wǎng)頁的情況，瀏覽器網(wǎng)址輸入：

master:8088

OK，我們配置成功了！

測試：
接下來，我們來測試一下：

① 我們提交到dev隊列

在網(wǎng)頁上我們可以看到任務(wù)已經(jīng)提交了

在網(wǎng)頁上，我們可以看占用資源查看：

② 提交到prod隊列

通過提交到dev和prod ，我們可以得出：不管是dev還是prod queue都不會占用Max_Capacity

總結(jié)

以上是生活随笔為你收集整理的学习笔记Hadoop（十二）—— Hadoop基础操作（4）—— YARN资源管理与调度策略的全部內(nèi)容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網(wǎng)站內(nèi)容還不錯，歡迎將生活随笔推薦給好友。