?

一、hadoop下載

使用2.7.6版本，因為公司生產環境是這個版本

cd /opt wget http://mirrors.hust.edu.cn/apache/hadoop/common/hadoop-2.7.6/hadoop-2.7.6.tar.gz

二、配置文件

參考文檔：https://hadoop.apache.org/docs/r2.7.6

?在$HADOOP_HOME/etc/hadoop目錄下需要配置7個文件

1.core-site.xml

<?xml version="1.0" encoding="UTF-8"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <configuration><property><name>fs.defaultFS</name><value>hdfs://pangu10:9000</value><description>NameNode URI,hdfs處理對外端口</description></property><property><name>hadoop.tmp.dir</name><value>/opt/hdfs/tmp</value><description>hdfs重新格式化時(如新增了一個datenode)需要刪除這個臨時目錄</description></property> </configuration> View Code

?2.hdfs-site.xml

<?xml version="1.0" encoding="UTF-8"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <configuration><property><name>dfs.namenode.name.dir</name><value>file:/opt/hdfs/name</value><description>namenode上存儲hdfs名字空間元數據 </description></property><property><name>dfs.datanode.data.dir</name><value>file:/opt/hdfs/data</value><description>datanode上數據塊的物理存儲位置</description></property><property><name>dfs.replication</name><value>1</value><description>設置dfs副本數，不設置默認是3個</description></property><property><name>dfs.namenode.secondary.http-address</name><value>pangu11:50090</value><description>設置secondname的端口</description></property> </configuration> View Code

3.yarn-site.xml

<?xml version="1.0"?> <configuration><property><name>yarn.resourcemanager.hostname</name><value>pangu10</value><description>指定resourcemanager所在的hostname</description></property><property><name>yarn.nodemanager.aux-services</name><value>mapreduce_shuffle</value><description>NodeManager上運行的附屬服務,需配置成mapreduce_shuffle，才可運行MapReduce程序</description></property><property><name>yarn.nodemanager.pmem-check-enabled</name><value>false</value></property><property><name>yarn.nodemanager.vmem-check-enabled</name><value>false</value></property> </configuration> View Code

4.mapred-site.xml

<?xml version="1.0"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <configuration><property><name>mapreduce.framework.name</name><value>yarn</value><description>指定mapreduce使用yarn框架</description></property> </configuration> View Code

5.slaves

pangu10 pangu11 pangu12

?6.yarn-env.sh

找到第23行

# export JAVA_HOME=/home/y/libexec/jdk1.6.0/

替換成

export JAVA_HOME=/opt/jdk1.8.0_181/

7.hadoop-env.sh

找到25行

export JAVA_HOME=${JAVA_HOME}

替換成

export JAVA_HOME=/opt/jdk1.8.0_181/

三、復制到slave

?

四、hdfs格式化

shell執行如下命令?

hdfs namenode -format

如果出現下面紅色的日志內容則格式化成功

18/10/12 12:38:33 INFO util.GSet: capacity = 2^15 = 32768 entries 18/10/12 12:38:33 INFO namenode.FSImage: Allocated new BlockPoolId: BP-1164998719-192.168.56.10-1539362313584 18/10/12 12:38:33 INFO common.Storage: Storage directory /opt/hdfs/name has been successfully formatted. 18/10/12 12:38:33 INFO namenode.FSImageFormatProtobuf: Saving image file /opt/hdfs/name/current/fsimage.ckpt_0000000000000000000 using no compression 18/10/12 12:38:33 INFO namenode.FSImageFormatProtobuf: Image file /opt/hdfs/name/current/fsimage.ckpt_0000000000000000000 of size 320 bytes saved in 0 seconds. 18/10/12 12:38:33 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0 18/10/12 12:38:33 INFO util.ExitUtil: Exiting with status 0 18/10/12 12:38:33 INFO namenode.NameNode: SHUTDOWN_MSG: /************************************************************ SHUTDOWN_MSG: Shutting down NameNode at pangu10/192.168.56.10 ************************************************************/

?

五、啟動hadoop

?

cd $HADOOP_HOME/sbin

./start-all.sh

說明：不要使用sh start-all.sh模式

?

http://pangu10:8088/cluster

轉載于:https://www.cnblogs.com/Netsharp/p/9780971.html

總結

以上是生活随笔為你收集整理的SPARK安装二：HADOOP集群部署的全部內容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網站內容還不錯，歡迎將生活随笔推薦給好友。