昨天QQ群里提了一個hadoop運行效率分配的問題，總結一下，寫個文章。集群使用hadoop-1.0.3
有些hadoop集群在運行的時候，不完全是絕對平均的分配，不過需要盡可能平均的分配任務，避免某一臺或者某幾臺服務器任務過重，其他服務器無事可做。這個，一方面是需要用到balancer，一個就是機架感知了。
通常，balancer是自動啟動的。而機架感知則需要單獨配置和編寫腳本。不過，機架感知，不是說是感知哪個服務器壞了，是根據機架位置的拓撲結構來選取服務器進行任務的權重分配。
機架感知需要自己寫一個腳本，然后放到hadoop的core-site.xml配置文件中，用namenode和jobtracker進行調用。
python代碼摘自竹葉青的博客
#!/usr/bin/python
#-*-coding:UTF-8 -*-
import sys

rack = {"hadoop-node-31":"rack1",
????????????????????????????????"hadoop-node-32":"rack1",
????????????????????????????????"hadoop-node-33":"rack1",
????????????????????????????????"hadoop-node-34":"rack1",
????????????????????????????????"hadoop-node-49":"rack2",
????????????????????????????????"hadoop-node-50":"rack2",
????????????????????????????????"hadoop-node-51":"rack2",
????????????????????????????????"hadoop-node-52":"rack2",
????????????????????????????????"hadoop-node-53":"rack2",
????????????????????????????????"hadoop-node-54":"rack2",
????????????????????????????????"192.168.1.31":"rack1",
????????????????????????????????"192.168.1.32":"rack1",
????????????????????????????????"192.168.1.33":"rack1",
????????????????????????????????"192.168.1.34":"rack1",
????????????????????????????????"192.168.1.49":"rack2",
????????????????????????????????"192.168.1.50":"rack2",
????????????????????????????????"192.168.1.51":"rack2",
????????????????????????????????"192.168.1.52":"rack2",
????????????????????????????????"192.168.1.53":"rack2",
????????????????????????????????"192.168.1.54":"rack2",
????????????????????????????????}

if __name__=="__main__":
????????print "/" + rack.get(sys.argv[1],"rack0")
按照老趙的博客，由于hadoop沒有明確的說明機架感知是感知IP還是感知主機名，所以就都寫上了。然后將腳本賦予可執(zhí)行權限chmod +x RackAware.py，并放到bin/目錄下。
然后打開conf/core-site.html放入一下代碼
<property>
????????<name>topology.script.file.name</name>
????????<value>/opt/modules/hadoop/hadoop-1.0.3/bin/RackAware.py</value>
<!--機架感知腳本路徑-->
????</property>
????<property>
????????<name>topology.script.number.args</name>
????????<value>20</value>
<!--機架服務器數量，由于我寫了20個，所以這里寫20-->
????</property>

然后重啟hadoop的namenode和jobtracker，可以在logs里看下namenode和jobtracker的日志，看到機架感知功能已經啟用了。
namenode日志
2012-06-08 14:42:19,174 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* NameSystem.registerDatanode: node registration from 192.168.1.49:50010 storage DS-1155827498-192.168.1.49-50010-1338289368956
2012-06-08 14:42:19,204 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /rack2/192.168.1.49:50010
2012-06-08 14:42:19,205 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* NameSystem.registerDatanode: node registration from 192.168.1.53:50010 storage DS-1773813988-192.168.1.53-50010-1338289405131
2012-06-08 14:42:19,226 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /rack2/192.168.1.53:50010
2012-06-08 14:42:19,226 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* NameSystem.registerDatanode: node registration from 192.168.1.34:50010 storage DS-2024494948-127.0.0.1-50010-1338289438983
2012-06-08 14:42:19,242 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /rack1/192.168.1.34:50010
2012-06-08 14:42:19,242 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* NameSystem.registerDatanode: node registration from 192.168.1.54:50010 storage DS-767528606-192.168.1.54-50010-1338289412267

......


2012-06-08 14:42:49,492 INFO org.apache.hadoop.hdfs.StateChange: STATE* Network topology has 2 racks and 10 datanodes
2012-06-08 14:42:49,492 INFO org.apache.hadoop.hdfs.StateChange: STATE* UnderReplicatedBlocks has 0 blocks
2012-06-08 14:42:49,642 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: ReplicateQueue QueueProcessingStatistics: First cycle completed 0 blocks in 0 msec
2012-06-08 14:42:49,642 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: ReplicateQueue QueueProcessingStatistics: Queue flush completed 0 blocks in 0 msec processing time, 0 msec clock time, 1 cycles

jobtracker 日志
2012-06-08 14:50:42,728 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /rack2/hadoop-node-52
2012-06-08 14:50:42,731 INFO org.apache.hadoop.mapred.JobTracker: Adding tracker tracker_hadoop-node-52:localhost.localdomain/127.0.0.1:33999 to host hadoop-node-52
2012-06-08 14:50:42,753 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /rack1/hadoop-node-34
2012-06-08 14:50:42,754 INFO org.apache.hadoop.mapred.JobTracker: Adding tracker tracker_hadoop-node-34:localhost.localdomain/127.0.0.1:37025 to host hadoop-node-34
2012-06-08 14:50:42,773 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /rack2/hadoop-node-51

總結

以上是生活随笔為你收集整理的Hadoop配置机架感知(python脚本)的全部內容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網站內容還不錯，歡迎將生活随笔推薦給好友。