Yarn資源分配示例

1 簡介

此手冊應用于yarn資源按照隊列分配的情景。

2 準備

2.1 環境說明

hadoop版本: hadoop-2.5.0-cdh5.3.3

3 配置

3.1 配置說明

1)? Yarn-site.xml里配置yanr能用到的資源總數以及公平調度策略所在路徑

#在虛擬機關機情況下，選中后點擊設置，在控制器SATA處點擊右邊的加號以創建虛擬硬盤。

?<property>

<name>yarn.nodemanager.resource.memory-mb</name>

<value>32760</value>

</property>

?

<property>

<name>yarn.scheduler.minimum-allocation-mb</name>

<value>2048</value>

</property>

?

<property>

<name>yarn.scheduler.maximum-allocation-mb</name>

<value>16384</value>

</property>

?

<property>

<name>yarn.nodemanager.resource.cpu-vcores</name>

<value>21</value>

</property>

?

?

?

<property>

???????? <name>yarn.scheduler.fair.allow-undeclared-pools</name>

???????? <value>false</value>

???????? <description>如果這是true,新的隊列可以在提交申請時被創建,falsessh使用默認隊列</description>

</property>

<property>

???????? <name>yarn.scheduler.fair.user-as-default-queue</name>

???????? <value>false</value>

???????? <description>使用跟用戶名相同的隊列名,如果設置為false或者未設置,則所有未知隊列的應用程序將被提交到default隊列中，默認值為true</description>

</property>

<property>

???????? <name>yarn.resourcemanager.scheduler.class</name>

???????? <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler</value>

???????? <description>公平配置隊列配置在fair-scheduler</description>

</property>

<property>

???????? <name>yarn.scheduler.fair.allocation.file</name>

???????? <value>/home/hadoop/hadoop-2.7.1/etc/hadoop/fair-allocation.xml</value>

???????? <description>公平配置文件路徑</description>

</property>

<property>

???????? <name>yarn.scheduler.fair.preemption</name>

???????? <value>true</value>

???????? <description>是否支持搶占,默認值為false</description>

</property>

<property>

???????? <name>yarn.scheduler.fair.sizebasedweight</name>

???????? <value>false</value>

???????? <description>是否啟用按應用程序資源需求分配資源,默認值為false即采用公平輪詢的方法分配資源</description>

</property>

<property>

???????? <name>yarn.scheduler.increment-allocation-mb</name>

???????? <value>1024</value>

???????? <description>僅fair有效,內存規整化單位,墨認值1024.(示例一個container請求1.5G,則調度器規整化為2G)</description>

</property>

2)? 配置隊列的資源限額及應用數量

#隊列可指定到用戶或則用戶組,如果是用戶組則需要在aclSubmitApps參數前加空格。

<allocations>

???????? <queuename="root">

???????? <aclSubmitApps>hadoop,dpi</aclSubmitApps>

???????? <aclAdministerApps>hadoop</aclAdministerApps>

?????????????????? <queuename="default">

??????????????????????????? <minResources>2000mb,1vcores</minResources>

??????????????????????????? <maxResources>8000mb,8vcores</maxResources>

??????????????????????????? <maxRunningApps>40</maxRunningApps>

??????????????????????????? <schedulingMode>fair</schedulingMode>

??????????????????????????? <aclSubmitApps>*</aclSubmitApps>

??????????????????????????? <aclAdministerApps>hadoop</aclAdministerApps>

?????????????????? </queue>

?????????????????? <queuename="test">

??????????????????????????? <minResources>2000mb,10vcores</minResources>

??????????????????????????? <maxResources>8000mb,8vcores</maxResources>

??????????????????????????? <maxRunningApps>4</maxRunningApps>

??????????????????????????? <schedulingMode>fair</schedulingMode>

??????????????????????????? <aclSubmitApps>test,hadoop</aclSubmitApps>

??????????????????????????? <aclAdministerApps>hadoop</aclAdministerApps>

?????????????????? </queue>

?????????????????? <queuename="hadoop">

??????????????????????????? <minResources>2000mb,1vcores</minResources>

??????????????????????????? <maxResources>8000mb,8vcores</maxResources>

??????????????????????????? <maxRunningApps>4</maxRunningApps>

??????????????????????????? <schedulingMode>fair</schedulingMode>

??????????????????????????? <aclSubmitApps>hadoop</aclSubmitApps>

??????????????????????????? <aclAdministerApps>hadoop</aclAdministerApps>

?????????????????? </queue>

???????? </queue>

???????? <username="hadoop"> <!-- 對于特定用戶的配置 -->

???????? ?<maxRunningApps>100</maxRunningApps>?

???????? </user>?

???????? <userMaxAppsDefault>50</userMaxAppsDefault><!--默認的用戶最多可以同時運行的任務 -->

???????? <queuePlacementPolicy>

?????????????????? <rulename="user" create="false" />

?????????????????? <rulename="specified" />

?????????????????? <rulename="default"/>

???????? </queuePlacementPolicy>

</allocations>

3.2配置問題

暫無.

#修改隊列信息后會動態刷新，以yarn 8088端口里的資源為準。

4 使用

set mapreduce.job.queuename為某個隊列到8088端口查看任務是否提交到該隊列.

5 總結

暫無.

總結

以上是生活随笔為你收集整理的Yarn资源分配示例的全部內容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網站內容還不錯，歡迎將生活随笔推薦給好友。