當前位置：首頁 > 编程资源 > 编程问答 >内容正文

编程问答

CentOS7下Hadoop集群搭建

發布時間：2025/3/19 编程问答 24 豆豆

生活随笔收集整理的這篇文章主要介紹了 CentOS7下Hadoop集群搭建小編覺得挺不錯的,現在分享給大家,幫大家做個參考.

文章目錄

1、概念
- 1.1、主從結構
- 1.2、Hadoop集群角色名稱
2、安裝前準備
- 2.1、安裝軟件準備
- 2.2、Hadoop集群服務器規劃
3、安裝
- 3.1 解壓文件
- 3.2 修改配置文件
- - 3.2.1.修改`hadoop-env.sh`配置
  - 3.2.2.修改`core-site.xml`配置
  - 3.2.3.修改`hdfs-site.xml`配置
  - 3.2.4.修改`mapred-site.xml`配置
  - 3.2.5.修改`yarn-site.xml`配置
  - 3.2.6.修改`slaves`配置
- 3.3 增加Hadoop環境變量
- 3.4 將Hadoop主機復制
- 3.5 Hadoop初始化
- 3.6 Hadoop啟動
- - 3.6.1 啟動HDFS
  - 3.6.2 啟動YARN
  - 3.6.2 查看服務啟動情況
4、啟動HDFS和YARN的web管理頁面
5、Hadoop常用命令

1、概念

1.1、主從結構

主從結構：集群中有部分節點充當主服務器，其他節點相對的充當從服務器的結構。主從結構一般分為：一主多從、多主多從。
Hadoop中的HDFS和YARN都是主從結構，主從結構有多種叫法：主節點-從節點、master-slave、管理者-工作者、leader-follower

1.2、Hadoop集群角色名稱

Hadoop集群中各個角色的名稱：

服務主節點從節點

HDFS	NameNode	DataNode
YARN	ResourceManager	NodeManager

2、安裝前準備

2.1、安裝軟件準備

1、準備一臺CentOS7的虛擬機
2、虛擬機中已安裝JDK，如果沒有，請參考《CentOS7安裝JDK1.8簡單體驗（java開發必備）》
3、準備好Hadoop的安裝文件，本次使用的版本是hadoop-2.7.3.tar.gz，可以點擊進行下載。

2.2、Hadoop集群服務器規劃

服務主機ip地址主機名稱用戶HDFSYARN

Hadoop-Msater	192.192.168.223.131	hadoop-master	root	NameNode,DataNode	ResourceManager,NodeManager
Hadoop-Slave1	192.192.168.223.128	hadoop-slave1	root	SecondaryNameNode,DataNode	NodeManager
Hadoop-Slave2	192.192.168.223.129	hadoop-slave2	root	DataNode	NodeManager
Hadoop-Slave3	192.192.168.223.130	hadoop-slave3	root	DataNode	NodeManager

規劃安裝目錄：/usr/local/hadoop/apps
規劃數據目錄：/usr/local/hadoop/data
說明：請自行創建或者自定義安裝目錄

3、安裝

3.1 解壓文件

首先將壓縮文包拷貝到/usr/local/hadoop/apps下，然后執行解壓程序

[root@hadoop-master apps]# cd /usr/local/hadoop/apps/ [root@hadoop-master apps]# ls hadoop-2.7.3.tar.gz [root@hadoop-master apps]# tar zxvf hadoop-2.7.3.tar.gz

3.2 修改配置文件

配置文件的目錄在：/usr/local/hadoop/apps/hadoop-2.7.3/etc/hadoop/

[root@hadoop-master apps]# cd /usr/local/hadoop/apps/hadoop-2.7.3/etc/hadoop/ [root@hadoop-master hadoop]# ls capacity-scheduler.xml hadoop-env.sh httpfs-env.sh kms-env.sh mapred-env.sh ssl-server.xml.example configuration.xsl hadoop-metrics2.properties httpfs-log4j.properties kms-log4j.properties mapred-queues.xml.template yarn-env.cmd container-executor.cfg hadoop-metrics.properties httpfs-signature.secret kms-site.xml mapred-site.xml.template yarn-env.sh core-site.xml hadoop-policy.xml httpfs-site.xml log4j.properties slaves yarn-site.xml hadoop-env.cmd hdfs-site.xml

3.2.1.修改hadoop-env.sh配置

打開hadoop-env.sh配置文件，修改JDK環境變量

[root@hadoop-master hadoop]# vi hadoop-env.sh

默認的環境變量取系統環境變量${JAVA_HOME}，當然也可以自己修改成。

# The java implementation to use. export JAVA_HOME=${JAVA_HOME}

因為我本機配置過JDK環境變量，因此此處不做修改。

[root@hadoop-master hadoop]# echo $JAVA_HOME /usr/local/jdk/jdk1.8.0_261

注意：我上面的示范太想當然了，這里必須配置jdk絕對路徑，不然啟動會報錯。
正確的配置是按照如下修改：

[root@hadoop-master hadoop]# vi hadoop-env.sh # The java implementation to use. export JAVA_HOME=/usr/local/jdk/jdk1.8.0_261

3.2.2.修改core-site.xml配置

首先打開core-site.xml配置文件

[root@hadoop-master hadoop]# vim core-site.xml

配置參數介紹：
fs.defaultFS ：這個屬性用來指定namenode的hdfs協議的文件系統通信地址，可以指定一個主機+端口，也可以指定為一個namenode服務（這個服務內部可以有多臺namenode實現ha的namenode服務
hadoop.tmp.dir : hadoop集群在工作的時候存儲的一些臨時文件的目錄

<configuration><property><name>fs.defaultFS</name><value>hdfs://hadoop-master:9000</value></property><property><name>hadoop.tmp.dir</name><value>/usr/local/hadoop/data/hadoopdata</value></property> </configuration>

3.2.3.修改hdfs-site.xml配置

[root@hadoop-master hadoop]# vi hdfs-site.xml

dfs.namenode.name.dir：namenode數據的存放地點。也就是namenode元數據存放的地方，記錄了hdfs系統中文件的元數據。
dfs.datanode.data.dir： datanode數據的存放地點。也就是block塊存放的目錄了。
dfs.replication：hdfs的副本數設置。也就是上傳一個文件，其分割為block塊后，每個block的冗余副本個數，默認配置是3。
dfs.secondary.http.address：secondarynamenode 運行節點的信息，和 namenode 不同節點

<configuration><property><name>dfs.namenode.name.dir</name><value>/usr/local/hadoop/data/hadoopdata/name</value><description>為了保證元數據的安全一般配置多個不同目錄</description></property><property><name>dfs.datanode.data.dir</name><value>/usr/local/hadoop/data/hadoopdata/data</value><description>datanode 的數據存儲目錄</description></property><property><name>dfs.replication</name><value>2</value><description>HDFS 的數據塊的副本存儲個數, 默認是3</description></property><property><name>dfs.secondary.http.address</name><value>hadoop-slave1:50090</value><description>secondarynamenode 運行節點的信息，和 namenode 不同節點</description></property> </configuration>

3.2.4.修改mapred-site.xml配置

[root@hadoop-master hadoop]# cp mapred-site.xml.template mapred-site.xml [root@hadoop-master hadoop]# vi mapred-site.xml

mapreduce.framework.name：指定mr框架為yarn方式,Hadoop二代MP也基于資源管理系統Yarn來運行。

<configuration><property><name>mapreduce.framework.name</name><value>yarn</value></property> </configuration>

3.2.5.修改yarn-site.xml配置

[root@hadoop-master hadoop]# vi yarn-site.xml

yarn.resourcemanager.hostname：yarn總管理器的IPC通訊地址
yarn.nodemanager.aux-services：NodeManager上運行的附屬服務。需配置成mapreduce_shuffle，才可運行MapReduce程序，默認""

<configuration> <property><name>yarn.resourcemanager.hostname</name><value>hadoop-master</value></property><property><name>yarn.nodemanager.aux-services</name><value>mapreduce_shuffle</value><description>YARN 集群為 MapReduce 程序提供的 shuffle 服務</description></property> </configuration>

3.2.6.修改slaves配置

[root@hadoop-master hadoop]# vi slaves hadoop-master hadoop-slave1 hadoop-slave2 hadoop-slave3

3.3 增加Hadoop環境變量

增加Hadoop環境變量

如果root用戶進行安裝。 vi /etc/profile

如果普通用戶進行安裝。 vi ~/.bashrc

將光標調整到文件尾部，并添加以下內容

export HADOOP_HOME=/usr/local/hadoop/apps/hadoop-2.7.3 export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:

更新配置文件，并查看hadoop版本

[root@hadoop-master hadoop]# source /etc/profile [root@hadoop-master hadoop]# hadoop version Hadoop 2.7.3 Subversion https://git-wip-us.apache.org/repos/asf/hadoop.git -r baa91f7c6bc9cb92be5982de4719c1c8af91ccff Compiled by root on 2016-08-18T01:41Z Compiled with protoc 2.5.0 From source with checksum 2e4ce5f957ea4db193bce3734ff29ff4 This command was run using /usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/common/hadoop-common-2.7.3.jar

3.4 將Hadoop主機復制

將Hadoop-Master克隆成三份：

Hadoop-Slave1 Hadoop-Slave2 Hadoop-Slave3

修改主機名：
Hadoop-Slave1主機名修改為：hadoop-slave1

[root@hadoop-master ~]# hostnamectl set-hostname hadoop-slave1 [root@hadoop-master ~]# hostname hadoop-slave1

Hadoop-Slave2主機名修改為：hadoop-slave2

[root@hadoop-master ~]# hostnamectl set-hostname hadoop-slave2 [root@hadoop-master ~]# hostname hadoop-slave2

Hadoop-Slave3主機名修改為：hadoop-slave3

[root@hadoop-master ~]# hostnamectl set-hostname hadoop-slave3 [root@hadoop-master ~]# hostname hadoop-slave3

并且在每一臺的主機里添加ip地址：

[root@hadoop-master ~]# vi /etc/hosts192.168.223.131 hadoop-master 192.168.223.128 hadoop-slave1 192.168.223.129 hadoop-slave2 192.168.223.130 hadoop-slave3

主機之間設置免密登錄可以參考《CentOS7虛擬機之間設置免密登錄》

3.5 Hadoop初始化

注意：HDFS初始化只能在主節點上進行

[root@hadoop-master hadoop]# hadoop namenode -format

初始化結果：

[root@hadoop-master hadoop]# hadoop namenode -format DEPRECATED: Use of this script to execute hdfs command is deprecated. Instead use the hdfs command for it.20/08/08 13:23:37 INFO namenode.NameNode: STARTUP_MSG: /************************************************************ STARTUP_MSG: Starting NameNode STARTUP_MSG: host = hadoop-master/192.168.223.131 STARTUP_MSG: args = [-format] STARTUP_MSG: version = 2.7.3 STARTUP_MSG: classpath = /usr/local/hadoop/apps/hadoop-2.7.3/etc/hadoop:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/common/lib/jaxb-api-2.2.2.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/common/lib/stax-api-1.0-2.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/common/lib/activation-1.1.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/common/lib/jackson-core-asl-1.9.13.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/common/lib/jackson-mapper-asl-1.9.13.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/common/lib/jackson-jaxrs-1.9.13.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/common/lib/jackson-xc-1.9.13.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/common/lib/jersey-server-1.9.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/common/lib/asm-3.2.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/common/lib/log4j-1.2.17.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/common/lib/jets3t-0.9.0.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/common/lib/httpclient-4.2.5.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/common/lib/httpcore-4.2.5.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/common/lib/java-xmlbuilder-0.4.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/common/lib/commons-lang-2.6.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/common/lib/commons-configuration-1.6.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/common/lib/commons-digester-1.8.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/common/lib/commons-beanutils-1.7.0.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/common/lib/slf4j-api-1.7.10.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/common/lib/avro-1.7.4.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/common/lib/paranamer-2.3.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/common/lib/snappy-java-1.0.4.1.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/common/lib/commons-compress-1.4.1.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/common/lib/xz-1.0.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/common/lib/protobuf-java-2.5.0.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/common/lib/gson-2.2.4.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/common/lib/hadoop-auth-2.7.3.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/common/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/common/lib/apacheds-i18n-2.0.0-M15.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/common/lib/api-asn1-api-1.0.0-M20.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/common/lib/api-util-1.0.0-M20.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/common/lib/zookeeper-3.4.6.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/common/lib/netty-3.6.2.Final.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/common/lib/curator-framework-2.7.1.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/common/lib/curator-client-2.7.1.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/common/lib/jsch-0.1.42.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/common/lib/curator-recipes-2.7.1.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/common/lib/htrace-core-3.1.0-incubating.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/common/lib/junit-4.11.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/common/lib/hamcrest-core-1.3.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/common/lib/mockito-all-1.8.5.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/common/lib/hadoop-annotations-2.7.3.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/common/lib/guava-11.0.2.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/common/lib/jsr305-3.0.0.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/common/lib/commons-cli-1.2.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/common/lib/commons-math3-3.1.1.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/common/lib/xmlenc-0.52.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/common/lib/commons-httpclient-3.1.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/common/lib/commons-logging-1.1.3.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/common/lib/commons-codec-1.4.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/common/lib/commons-io-2.4.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/common/lib/commons-net-3.1.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/common/lib/commons-collections-3.2.2.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/common/lib/servlet-api-2.5.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/common/lib/jetty-6.1.26.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/common/lib/jetty-util-6.1.26.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/common/lib/jsp-api-2.1.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/common/lib/jersey-core-1.9.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/common/lib/jersey-json-1.9.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/common/lib/jettison-1.1.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/common/hadoop-common-2.7.3.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/common/hadoop-common-2.7.3-tests.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/common/hadoop-nfs-2.7.3.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/hdfs:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/hdfs/lib/commons-codec-1.4.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/hdfs/lib/commons-logging-1.1.3.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/hdfs/lib/netty-3.6.2.Final.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/hdfs/lib/guava-11.0.2.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/hdfs/lib/jsr305-3.0.0.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/hdfs/lib/xmlenc-0.52.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/hdfs/lib/commons-io-2.4.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/hdfs/lib/servlet-api-2.5.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/hdfs/lib/jetty-6.1.26.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/hdfs/lib/jetty-util-6.1.26.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/hdfs/lib/jersey-core-1.9.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/hdfs/lib/jackson-core-asl-1.9.13.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/hdfs/lib/jackson-mapper-asl-1.9.13.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/hdfs/lib/jersey-server-1.9.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/hdfs/lib/asm-3.2.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/hdfs/lib/commons-lang-2.6.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/hdfs/lib/protobuf-java-2.5.0.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/hdfs/lib/htrace-core-3.1.0-incubating.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/hdfs/lib/netty-all-4.0.23.Final.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/hdfs/lib/xercesImpl-2.9.1.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/hdfs/lib/xml-apis-1.3.04.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/hdfs/lib/leveldbjni-all-1.8.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/hdfs/hadoop-hdfs-2.7.3.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/hdfs/hadoop-hdfs-2.7.3-tests.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/hdfs/hadoop-hdfs-nfs-2.7.3.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/yarn/lib/zookeeper-3.4.6-tests.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/yarn/lib/commons-lang-2.6.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/yarn/lib/guava-11.0.2.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/yarn/lib/jsr305-3.0.0.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/yarn/lib/commons-logging-1.1.3.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/yarn/lib/protobuf-java-2.5.0.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/yarn/lib/commons-cli-1.2.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/yarn/lib/log4j-1.2.17.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/yarn/lib/jaxb-api-2.2.2.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/yarn/lib/stax-api-1.0-2.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/yarn/lib/activation-1.1.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/yarn/lib/commons-compress-1.4.1.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/yarn/lib/xz-1.0.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/yarn/lib/servlet-api-2.5.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/yarn/lib/commons-codec-1.4.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/yarn/lib/jetty-util-6.1.26.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/yarn/lib/jersey-core-1.9.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/yarn/lib/jersey-client-1.9.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/yarn/lib/jackson-core-asl-1.9.13.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/yarn/lib/jackson-mapper-asl-1.9.13.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/yarn/lib/jackson-jaxrs-1.9.13.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/yarn/lib/jackson-xc-1.9.13.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/yarn/lib/guice-servlet-3.0.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/yarn/lib/guice-3.0.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/yarn/lib/javax.inject-1.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/yarn/lib/aopalliance-1.0.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/yarn/lib/commons-io-2.4.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/yarn/lib/jersey-server-1.9.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/yarn/lib/asm-3.2.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/yarn/lib/jersey-json-1.9.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/yarn/lib/jettison-1.1.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/yarn/lib/jaxb-impl-2.2.3-1.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/yarn/lib/jersey-guice-1.9.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/yarn/lib/zookeeper-3.4.6.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/yarn/lib/netty-3.6.2.Final.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/yarn/lib/leveldbjni-all-1.8.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/yarn/lib/commons-collections-3.2.2.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/yarn/lib/jetty-6.1.26.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/yarn/hadoop-yarn-api-2.7.3.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/yarn/hadoop-yarn-common-2.7.3.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/yarn/hadoop-yarn-server-common-2.7.3.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.7.3.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/yarn/hadoop-yarn-server-web-proxy-2.7.3.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/yarn/hadoop-yarn-server-applicationhistoryservice-2.7.3.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-2.7.3.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/yarn/hadoop-yarn-server-tests-2.7.3.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/yarn/hadoop-yarn-client-2.7.3.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/yarn/hadoop-yarn-server-sharedcachemanager-2.7.3.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.3.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.7.3.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/yarn/hadoop-yarn-registry-2.7.3.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/mapreduce/lib/protobuf-java-2.5.0.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/mapreduce/lib/avro-1.7.4.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/mapreduce/lib/jackson-core-asl-1.9.13.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/mapreduce/lib/jackson-mapper-asl-1.9.13.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/mapreduce/lib/paranamer-2.3.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/mapreduce/lib/snappy-java-1.0.4.1.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/mapreduce/lib/commons-compress-1.4.1.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/mapreduce/lib/xz-1.0.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/mapreduce/lib/hadoop-annotations-2.7.3.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/mapreduce/lib/commons-io-2.4.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/mapreduce/lib/jersey-core-1.9.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/mapreduce/lib/jersey-server-1.9.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/mapreduce/lib/asm-3.2.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/mapreduce/lib/log4j-1.2.17.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/mapreduce/lib/netty-3.6.2.Final.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/mapreduce/lib/leveldbjni-all-1.8.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/mapreduce/lib/guice-3.0.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/mapreduce/lib/javax.inject-1.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/mapreduce/lib/aopalliance-1.0.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/mapreduce/lib/jersey-guice-1.9.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/mapreduce/lib/guice-servlet-3.0.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/mapreduce/lib/junit-4.11.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/mapreduce/lib/hamcrest-core-1.3.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.7.3.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/mapreduce/hadoop-mapreduce-client-common-2.7.3.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-2.7.3.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/mapreduce/hadoop-mapreduce-client-app-2.7.3.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-2.7.3.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.7.3.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-plugins-2.7.3.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.3.jar:/usr/local/hadoop/apps/hadoop-2.7.3/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.7.3-tests.jar:/usr/local/hadoop/apps/hadoop-2.7.3/contrib/capacity-scheduler/*.jar:/usr/local/hadoop/apps/hadoop-2.7.3/contrib/capacity-scheduler/*.jar STARTUP_MSG: build = https://git-wip-us.apache.org/repos/asf/hadoop.git -r baa91f7c6bc9cb92be5982de4719c1c8af91ccff; compiled by 'root' on 2016-08-18T01:41Z STARTUP_MSG: java = 1.8.0_261 ************************************************************/ 20/08/08 13:23:37 INFO namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT] 20/08/08 13:23:37 INFO namenode.NameNode: createNameNode [-format] 20/08/08 13:23:38 WARN common.Util: Path /usr/local/hadoop/data/hadoopdata/name should be specified as a URI in configuration files. Please update hdfs configuration. 20/08/08 13:23:38 WARN common.Util: Path /usr/local/hadoop/data/hadoopdata/name should be specified as a URI in configuration files. Please update hdfs configuration. Formatting using clusterid: CID-fce3113f-05f1-4de1-bc28-6927ce8d03ca 20/08/08 13:23:38 INFO namenode.FSNamesystem: No KeyProvider found. 20/08/08 13:23:38 INFO namenode.FSNamesystem: fsLock is fair:true 20/08/08 13:23:38 INFO blockmanagement.DatanodeManager: dfs.block.invalidate.limit=1000 20/08/08 13:23:38 INFO blockmanagement.DatanodeManager: dfs.namenode.datanode.registration.ip-hostname-check=true 20/08/08 13:23:38 INFO blockmanagement.BlockManager: dfs.namenode.startup.delay.block.deletion.sec is set to 000:00:00:00.000 20/08/08 13:23:38 INFO blockmanagement.BlockManager: The block deletion will start around 2020 八月 08 13:23:38 20/08/08 13:23:38 INFO util.GSet: Computing capacity for map BlocksMap 20/08/08 13:23:38 INFO util.GSet: VM type = 64-bit 20/08/08 13:23:38 INFO util.GSet: 2.0% max memory 966.7 MB = 19.3 MB 20/08/08 13:23:38 INFO util.GSet: capacity = 2^21 = 2097152 entries 20/08/08 13:23:38 INFO blockmanagement.BlockManager: dfs.block.access.token.enable=false 20/08/08 13:23:38 INFO blockmanagement.BlockManager: defaultReplication = 2 20/08/08 13:23:38 INFO blockmanagement.BlockManager: maxReplication = 512 20/08/08 13:23:38 INFO blockmanagement.BlockManager: minReplication = 1 20/08/08 13:23:38 INFO blockmanagement.BlockManager: maxReplicationStreams = 2 20/08/08 13:23:38 INFO blockmanagement.BlockManager: replicationRecheckInterval = 3000 20/08/08 13:23:38 INFO blockmanagement.BlockManager: encryptDataTransfer = false 20/08/08 13:23:38 INFO blockmanagement.BlockManager: maxNumBlocksToLog = 1000 20/08/08 13:23:38 INFO namenode.FSNamesystem: fsOwner = root (auth:SIMPLE) 20/08/08 13:23:38 INFO namenode.FSNamesystem: supergroup = supergroup 20/08/08 13:23:38 INFO namenode.FSNamesystem: isPermissionEnabled = true 20/08/08 13:23:38 INFO namenode.FSNamesystem: HA Enabled: false 20/08/08 13:23:38 INFO namenode.FSNamesystem: Append Enabled: true 20/08/08 13:23:39 INFO util.GSet: Computing capacity for map INodeMap 20/08/08 13:23:39 INFO util.GSet: VM type = 64-bit 20/08/08 13:23:39 INFO util.GSet: 1.0% max memory 966.7 MB = 9.7 MB 20/08/08 13:23:39 INFO util.GSet: capacity = 2^20 = 1048576 entries 20/08/08 13:23:39 INFO namenode.FSDirectory: ACLs enabled? false 20/08/08 13:23:39 INFO namenode.FSDirectory: XAttrs enabled? true 20/08/08 13:23:39 INFO namenode.FSDirectory: Maximum size of an xattr: 16384 20/08/08 13:23:39 INFO namenode.NameNode: Caching file names occuring more than 10 times 20/08/08 13:23:39 INFO util.GSet: Computing capacity for map cachedBlocks 20/08/08 13:23:39 INFO util.GSet: VM type = 64-bit 20/08/08 13:23:39 INFO util.GSet: 0.25% max memory 966.7 MB = 2.4 MB 20/08/08 13:23:39 INFO util.GSet: capacity = 2^18 = 262144 entries 20/08/08 13:23:39 INFO namenode.FSNamesystem: dfs.namenode.safemode.threshold-pct = 0.9990000128746033 20/08/08 13:23:39 INFO namenode.FSNamesystem: dfs.namenode.safemode.min.datanodes = 0 20/08/08 13:23:39 INFO namenode.FSNamesystem: dfs.namenode.safemode.extension = 30000 20/08/08 13:23:39 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.window.num.buckets = 10 20/08/08 13:23:39 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.num.users = 10 20/08/08 13:23:39 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.windows.minutes = 1,5,25 20/08/08 13:23:39 INFO namenode.FSNamesystem: Retry cache on namenode is enabled 20/08/08 13:23:39 INFO namenode.FSNamesystem: Retry cache will use 0.03 of total heap and retry cache entry expiry time is 600000 millis 20/08/08 13:23:39 INFO util.GSet: Computing capacity for map NameNodeRetryCache 20/08/08 13:23:39 INFO util.GSet: VM type = 64-bit 20/08/08 13:23:39 INFO util.GSet: 0.029999999329447746% max memory 966.7 MB = 297.0 KB 20/08/08 13:23:39 INFO util.GSet: capacity = 2^15 = 32768 entries 20/08/08 13:23:39 INFO namenode.FSImage: Allocated new BlockPoolId: BP-1166270090-192.168.223.131-1596864219273 20/08/08 13:23:39 INFO common.Storage: Storage directory /usr/local/hadoop/data/hadoopdata/name has been successfully formatted. 20/08/08 13:23:39 INFO namenode.FSImageFormatProtobuf: Saving image file /usr/local/hadoop/data/hadoopdata/name/current/fsimage.ckpt_0000000000000000000 using no compression 20/08/08 13:23:39 INFO namenode.FSImageFormatProtobuf: Image file /usr/local/hadoop/data/hadoopdata/name/current/fsimage.ckpt_0000000000000000000 of size 351 bytes saved in 0 seconds. 20/08/08 13:23:39 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0 20/08/08 13:23:39 INFO util.ExitUtil: Exiting with status 0 20/08/08 13:23:39 INFO namenode.NameNode: SHUTDOWN_MSG: /************************************************************ SHUTDOWN_MSG: Shutting down NameNode at hadoop-master/192.168.223.131 ************************************************************/

3.6 Hadoop啟動

3.6.1 啟動HDFS

啟動HDFS，不管在集群哪個節點都可以

[root@hadoop-master ~]# start-dfs.sh Starting namenodes on [hadoop-master] root@hadoop-master's password: hadoop-master: starting namenode, logging to /usr/local/hadoop/apps/hadoop-2.7.3/logs/hadoop-root-namenode-hadoop-master.out root@hadoop-master's password: hadoop-slave1: starting datanode, logging to /usr/local/hadoop/apps/hadoop-2.7.3/logs/hadoop-root-datanode-hadoop-slave1.out hadoop-slave2: starting datanode, logging to /usr/local/hadoop/apps/hadoop-2.7.3/logs/hadoop-root-datanode-hadoop-slave2.out hadoop-slave3: starting datanode, logging to /usr/local/hadoop/apps/hadoop-2.7.3/logs/hadoop-root-datanode-hadoop-slave3.out hadoop-master: starting datanode, logging to /usr/local/hadoop/apps/hadoop-2.7.3/logs/hadoop-root-datanode-hadoop-master.out Starting secondary namenodes [hadoop-slave1] hadoop-slave1: starting secondarynamenode, logging to /usr/local/hadoop/apps/hadoop-2.7.3/logs/hadoop-root-secondarynamenode-hadoop-slave1.out [root@hadoop-master ~]#

歷盡艱難終于啟動成功了，期間遇到了輸入密碼兩次，剛開始不知道于是乎傻傻的等，然后殺進程，重復了好幾次操作。下面介紹一個快速殺進程的命令：

ps -ef | grep hadoop | grep -v grep | cut -c 9-15 | xargs kill -s 9

3.6.2 啟動YARN

注意：這個只能在主節點進行啟動。

[root@hadoop-master ~]# start-yarn.sh starting yarn daemons starting resourcemanager, logging to /usr/local/hadoop/apps/hadoop-2.7.3/logs/yarn-root-resourcemanager-hadoop-master.out root@hadoop-master's password: hadoop-slave2: starting nodemanager, logging to /usr/local/hadoop/apps/hadoop-2.7.3/logs/yarn-root-nodemanager-hadoop-slave2.out hadoop-slave3: starting nodemanager, logging to /usr/local/hadoop/apps/hadoop-2.7.3/logs/yarn-root-nodemanager-hadoop-slave3.out hadoop-slave1: starting nodemanager, logging to /usr/local/hadoop/apps/hadoop-2.7.3/logs/yarn-root-nodemanager-hadoop-slave1.out hadoop-master: starting nodemanager, logging to /usr/local/hadoop/apps/hadoop-2.7.3/logs/yarn-root-nodemanager-hadoop-master.out [root@hadoop-master ~]#

3.6.2 查看服務啟動情況

hadoop-master

[root@hadoop-master ~]# jps 8737 Jps 8003 DataNode 8317 ResourceManager 7870 NameNode 8606 NodeManager

hadoop-slave1

[root@hadoop-slave1 hadoop]# jps 6055 NodeManager 5913 SecondaryNameNode 6185 Jps 5818 DataNode

hadoop-slave2

[root@hadoop-slave2 hadoop]# jps 4922 NodeManager 5068 Jps 4751 DataNode

hadoop-slave3

[root@hadoop-slave3 hadoop]# jps 5184 Jps 4868 DataNode 5038 NodeManager

4、啟動HDFS和YARN的web管理頁面

HDFS：http://192.168.223.131:50070 YARN：http://192.168.223.131:8088

這里我只看到了一個節點：

網上查詢原因，大意是：直接克隆的機器storageID一樣，導致訪問總是出現錯誤。
可以參考：hadoop集群通過web管理界面只顯示一個節點，datanode只啟動一個

但是，但是，但是我幾乎試遍了所有的方式，耗時3個多小時都沒有解決，幾乎要放棄了，突然想到一個問題，我虛擬機的防火墻還開著，集群之間需要通信是否被防火墻所阻擋了？？？

嘗試關閉虛擬機防火墻：

systemctl stop firewalld.service

然后奇跡出現了：

看到這個結果，也不知道想哭還是想笑，以后一定要引以為戒。

5、Hadoop常用命令

1. start-all.sh 啟動所有的Hadoop守護進程。包括NameNode、 Secondary NameNode、DataNode、JobTracker、 TaskTrack 2. stop-all.sh 停止所有的Hadoop守護進程。包括NameNode、 Secondary NameNode、DataNode、JobTracker、 TaskTrack 3. start-dfs.sh 啟動Hadoop HDFS守護進程NameNode、SecondaryNameNode和DataNode 4. stop-dfs.sh 停止Hadoop HDFS守護進程NameNode、SecondaryNameNode和DataNode 5. hadoop-daemons.sh start namenode 單獨啟動NameNode守護進程 6. hadoop-daemons.sh stop namenode 單獨停止NameNode守護進程 7. hadoop-daemons.sh start datanode 單獨啟動DataNode守護進程 8. hadoop-daemons.sh stop datanode 單獨停止DataNode守護進程 9. hadoop-daemons.sh start secondarynamenode 單獨啟動SecondaryNameNode守護進程 10. hadoop-daemons.sh stop secondarynamenode 單獨停止SecondaryNameNode守護進程 11. start-mapred.sh 啟動Hadoop MapReduce守護進程JobTracker和TaskTracker 12. stop-mapred.sh 停止Hadoop MapReduce守護進程JobTracker和TaskTracker 13. hadoop-daemons.sh start jobtracker 單獨啟動JobTracker守護進程 14. hadoop-daemons.sh stop jobtracker 單獨停止JobTracker守護進程 15. hadoop-daemons.sh start tasktracker 單獨啟動TaskTracker守護進程 16. hadoop-daemons.sh stop tasktracker 單獨啟動TaskTracker守護進程

總結

以上是生活随笔為你收集整理的CentOS7下Hadoop集群搭建的全部內容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網站內容還不錯，歡迎將生活随笔推薦給好友。

上一篇： CentOS7虚拟机之间设置免密登录
下一篇：每天学一点儿shell：shell字符串