六、YARN資源管理與調度策略

6.1、YARN資源管理

MapReduce任務/資源流程：

---

由5.1章節我們可知，當把yarn.scheduler.capacity.maximum-am-resource-percent參數調整為0.2 時，我們可以同時運行2個任務，此時Application Master最大資源為5120M（每個任務啟動的AppMaster資源為2048M）

當參數調整為0.3時，可同時運行4個任務

6.2、YARN調度策略

Yarn的資源調度器是可配置的，Yarn定義了一套接口規范(接口ResourceScheduler)，用戶可以按照自己的需求實現這個接口中的方法。

Yarn自身自帶了三種調度器，分別是：FIFO Scheduler、Capacity Scheduler和Fair Scheduler。

Yarn的資源調度器默認使用Capacity Scheduler，但是默認只有一個root.default隊列。

調整Yarn資源隊列：

添加dev,prod 隊列，刪除default隊列；

修改default隊列配置為dev隊列，并修改其capacity為40 ， maxium-capacity 為60 ；

復制dev隊列屬性，并修改為prod隊列，修改其capacity為60 ， maxium-capacity 為80 ；

重啟YARN集群；

cd /opt/hadoop-3.1.4/etc/hadoop vi capacity-scheduler.xml

修改如下：

修改完后，配置prod，在dev末尾添加：

<!-- config prod queue --><property><name>yarn.scheduler.capacity.root.prod.capacity</name><value>60</value><description>Default queue target capacity.</description></property><property><name>yarn.scheduler.capacity.root.prod.user-limit-factor</name><value>1</value><description>Default queue user limit a percentage from 0.0 to 1.0.</description></property><property><name>yarn.scheduler.capacity.root.prod.maximum-capacity</name><value>80</value><description>The maximum capacity of the default queue. </description></property><property><name>yarn.scheduler.capacity.root.prod.state</name><value>RUNNING</value><description>The state of the default queue. State can be one of RUNNING or STOPPED.</description></property><property><name>yarn.scheduler.capacity.root.prod.acl_submit_applications</name><value>*</value><description>The ACL of who can submit jobs to the default queue.</description></property><property><name>yarn.scheduler.capacity.root.prod.acl_administer_queue</name><value>*</value><description>The ACL of who can administer jobs on the default queue.</description></property><property><name>yarn.scheduler.capacity.root.prod.acl_application_max_priority</name><value>*</value><description>The ACL of who can submit applications with configured priority.For e.g, [user={name} group={name} max_priority={priority} default_priority={priority}]</description></property><property><name>yarn.scheduler.capacity.root.prod.maximum-application-lifetime</name><value>-1</value><description>Maximum lifetime of an application which is submitted to a queuein seconds. Any value less than or equal to zero will be considered asdisabled.This will be a hard time limit for all applications in thisqueue. If positive value is configured then any application submittedto this queue will be killed after exceeds the configured lifetime.User can also specify lifetime per application basis inapplication submission context. But user lifetime will beoverridden if it exceeds queue maximum lifetime. It is point-in-timeconfiguration.Note : Configuring too low value will result in killing applicationsooner. This feature is applicable only for leaf queue.</description></property><property><name>yarn.scheduler.capacity.root.prod.default-application-lifetime</name><value>-1</value><description>Default lifetime of an application which is submitted to a queuein seconds. Any value less than or equal to zero will be considered asdisabled.If the user has not submitted application with lifetime value then thisvalue will be taken. It is point-in-time configuration.Note : Default lifetime can't exceed maximum lifetime. This feature isapplicable only for leaf queue.</description></property>

配置prod完后，保存、退出，拷貝到node1、node2、node3下：

scp capacity-scheduler.xml node1:/opt/hadoop-3.1.4/etc/hadoop/ scp capacity-scheduler.xml node2:/opt/hadoop-3.1.4/etc/hadoop/ scp capacity-scheduler.xml node3:/opt/hadoop-3.1.4/etc/hadoop/

最后，重啟下集群(進入sbin目錄)

./stop-yarn.sh ./start-yarn.sh

啟動成功

最后，我們查看下網頁的情況，瀏覽器網址輸入：

master:8088

OK，我們配置成功了！

測試：
接下來，我們來測試一下：

① 我們提交到dev隊列

在網頁上我們可以看到任務已經提交了

在網頁上，我們可以看占用資源查看：

② 提交到prod隊列

通過提交到dev和prod ，我們可以得出：不管是dev還是prod queue都不會占用Max_Capacity

總結

以上是生活随笔為你收集整理的学习笔记Hadoop（十二）—— Hadoop基础操作（4）—— YARN资源管理与调度策略的全部內容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網站內容還不錯，歡迎將生活随笔推薦給好友。