hadoop 2.7.3偽分布式安裝

hadoop集群的偽分布式部署由于只需要一臺(tái)服務(wù)器，在測(cè)試，開(kāi)發(fā)過(guò)程中還是很方便實(shí)用的，有必要將搭建偽分布式的過(guò)程記錄下來(lái)，好記性不如爛筆頭。
hadoop 2.7.3
JDK 1.8.91

到Apache的官網(wǎng)下載hadoop的二進(jìn)制安裝包。

cd /home/fuxin.zhao/soft
tar -czvf hadoop 2.7.3.tar.gz
cd hadoop-2.7.3
cd etc/hadoop/
pwd

1. 建立本機(jī)到本機(jī)的免密登錄

ssh-keygen -t rsa -P "" cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys ssh localhost

1. 修改hadoop的配置文件

位于$HADOOP_HOME/conf目錄下的修改四個(gè)配置文件：slaves、core-site.xml
hdfs-site.xml 、mapred-site.xml 、 yarn-site.xml

vi etc/hadoop/yarn-env.sh

export JAVA_HOME=/usr/local/jdk

vi etc/hadoop/hadoop-env.sh

export JAVA_HOME=/usr/local/jdk

vi slaves

##加入本機(jī)的hostname fuxin.zhao@ubuntuServer01:~/soft/hadoop-2.7.3/etc/hadoop$ vi slaves ubuntuServer01

vi core-site.xml

<configuration><property><name>fs.defaultFS</name><value>hdfs://ubuntuServer01:9000</value></property><property><name>hadoop.tmp.dir</name><value>file:/home/fuxin.zhao/hadoop/tmp</value><description>Abase for other temporary directories.</description></property> </configuration>

vi hdfs-site.xml：

<configuration><property><name>dfs.replication</name><value>1</value></property><property><name>dfs.namenode.name.dir</name><value>file:/home/fuxin.zhao/hadoop/tmp/dfs/name</value></property><property><name>dfs.datanode.data.dir</name><value>file:/home/fuxin.zhao/hadoop/tmp/dfs/data</value></property><property><name>dfs.block.size</name><value>67108864</value></property> </configuration>

vi yarn-site.xml

<configuration> <property><name>yarn.nodemanager.aux-services</name><value>mapreduce_shuffle</value> </property> <property><name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name><value>org.apache.hadoop.mapred.ShuffleHandler</value> </property> <property><name>yarn.scheduler.minimum-allocation-mb</name><value>512</value> </property> <property><name>yarn.scheduler.maximum-allocation-mb</name><value>2048</value> </property> <property><name>yarn.scheduler.minimum-allocation-vcores</name><value>1</value> </property> <property><name>yarn.scheduler.maximum-allocation-vcores</name><value>2</value> </property> </configuration>

vi mapred-site.xml

<configuration> <property><name>mapreduce.framework.name</name><value>yarn</value> </property> <property><name>yarn.app.mapreduce.am.resource.mb</name><value>512</value> </property> <property><name>mapreduce.map.memory.mb</name><value>512</value> </property> <property><name>mapreduce.reduce.memory.mb</name><value>512</value> </property> </configuration>

vi .bashrc

export JAVA_HOME=/usr/local/jdk export HADOOP_HOME=/home/fuxin.zhao/soft/hadoop-2.7.3 export PATH=${HADOOP_HOME}/bin:${HADOOP_HOME}/sbin:$PATH

配置完成后，執(zhí)行 NameNode 的格式化:

./bin/hdfs namenode -format
./sbin/start-dfs.sh
./sbin/start-yarn.sh
mr-jobhistory-daemon.sh start historyserver

查看hdfs的web頁(yè)面：
http://ubuntuserver01:50070/
http://ubuntuserver01:8088/

hadoop fs -ls /
hadoop fs -mkdir /user
hadoop fs -mkdir /user/fuxin.zhao
hadoop fs -touchz textFile

運(yùn)行官方自帶的測(cè)試job（teragen and terasort）：

測(cè)試job（teragen and terasort） #在/tmp/terasort/1000000下生成100M數(shù)據(jù) hadoop jar $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.3.jar teragen 1000000 /tmp/terasort/1000000-input#排序，輸出到/tmp/terasort/1000000-output hadoop jar $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.3.jar terasort /tmp/terasort/1000000-input /tmp/terasort/1000000-output#刪除臨時(shí)文件 hadoop fs -rm -r /tmp/terasort/1000000-input hadoop fs -rm -r /tmp/terasort/1000000-output

轉(zhuǎn)載于:https://www.cnblogs.com/honeybee/p/6400709.html

總結(jié)

以上是生活随笔為你收集整理的hadoop 2.7.3伪分布式安装的全部?jī)?nèi)容，希望文章能夠幫你解決所遇到的問(wèn)題。

如果覺(jué)得生活随笔網(wǎng)站內(nèi)容還不錯(cuò)，歡迎將生活随笔推薦給好友。