linux下hadoop 环境搭建
準(zhǔn)備 3臺linux 分別hostname為: node-1,node-2,node-3
vi /etc/sysconfig/network HOSTNAME=yourhostname#保存 hostname yourhostname hostname安裝 JDK 并配置 環(huán)境變量
通過xshell等ssh 工具 查看3臺機(jī)子的時間是否一致(需通過成一致的時間)
xshell 查看-撰寫-撰寫窗口-全部會話(可以在敲一遍命名 在所有窗口生效)
Vi /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4::1 localhost localhost.localdomain localhost6 localhost6.localdomain6192.168.1.220 node-1192.168.1.222 node-2192.168.1.223 node-3免密登入配置:
ssh-keygen -t rsa(4個空格) 生成 id_rsa(公鑰) id_rsa..pub(公鑰)cd ~/.ssh/scp id_rsa.pub root@ip:/usr/local/cd /usr/local/cat id_rsa.pub >> ~/.ssh/authorized_keys解壓hadoop安裝包tar -zxvf hadoop-3.3.1.tar.gz
vi /usr/local/hadoop-3.3.1/etc/hadoop/hadoop-env.xml
#配置jdk JAVA_HOME=/usr/local/jdk1.8.0_301
vi /usr/local/hadoop-3.3.1/etc/hadoop/core-site.xml
<configuration> <property> <name>fs.default.name</name> <value>hdfs://node-1:9000</value> </property> ##臨時文件存放位置<property> <name>hadoop.tmp.dir</name> <value>/home/hadoop/tmp</value> </property> </configuration>vi /usr/local/hadoop-3.3.1/etc/hadoop/hdfs-site.xml
<configuration><property> <name>dfs.replication</name> <value>2</value></property> <property> <name>dfs.namenode.secondary.http-address</name> <value>node-2:50090</value> </property><property> <name>dfs.datanode.handler.count</name> <value>50</value></property> </configuration>vi /usr/local/hadoop-3.3.1/etc/hadoop/mapred-site.xml
<configuration><property> <name>mapreduce.framework.name</name> <value>yarn</value> </property> </configuration>vi /usr/local/hadoop-3.3.1/etc/hadoop/yarn-site.xml
<configuration><property> <name>yarn.resourcemanager.hostname</name> <value>node-1</value> </property> <property><name>yarn.log-aggregation-enable</name><value>true</value></property><property><name>yarn.log-aggregation.retain-seconds</name><value>604800</value></property><property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> </property></configuration>vi /usr/local/hadoop-3.3.1/etc/hadoop/workers
node-1 node-2 node-3配置hadoop環(huán)境變量(vi /etc/profile)
export HADOOP_HOME=/usr/local/hadoop-3.3.1 export PATH=$PATH:$HADOOP_HOME/bin: $HADOOP_HOME/sbin復(fù)制: scp -r /usr/local/hadoop-3.3.1/ root@node-2:/usr/local/
scp -r /usr/local/hadoop-3.3.1/ root@node-3:/usr/local/
復(fù)制: scp –r /etc/profile root@node-2:/etc/
scp –r /etc/profile root@node-3:/etc/
將start-dfs.sh，stop-dfs.sh兩個文件頂部添加以下參數(shù)
#!/usr/bin/env bash HDFS_DATANODE_USER=root HADOOP_SECURE_DN_USER=hdfs HDFS_NAMENODE_USER=root HDFS_SECONDARYNAMENODE_USER=rootstart-yarn.sh，stop-yarn.sh
#!/usr/bin/env bash YARN_RESOURCEMANAGER_USER=root HADOOP_SECURE_DN_USER=yarn YARN_NODEMANAGER_USER=root如果啟動后沒有datenode節(jié)點要刪除 core-site.xml中的路徑在執(zhí)行格式化
通過瀏覽器http://ip:8080訪問頁面
java操作HDFS
//連接hdfs configuration conf =new Configuration(); conf.set(fs.defaultFs,"hdfs://node-1:9000")//設(shè)置用戶身份 方式1:System.setProperty("HADOOP_USER_NAME","root") 方式2:FileSystem fs=FileSystem.get(new URI("hdfs://node-1:9000"),conf,"root"); FileSystem fs=FileSystem.get(conf); //操作 fs.xxxx() fs.close() ---------------------------獲取遍歷文件-------- RemoteInterator<LocatedFIleStatus> list=fs.listFIles(path,true); while(list.hasnext()){LocatedFIleStatus f=list.next()}; --------------------------上傳文件流操作--------------------------------------------- FSDataOutputStream out=fs.create(new Path(文件),true); FIleInputStream in=new FIleInputstream("D://文件"); IOUtils.copy(in, out ); ---------------------------------------------------------------------------------總結(jié)
以上是生活随笔為你收集整理的linux下hadoop 环境搭建的全部內(nèi)容，希望文章能夠幫你解決所遇到的問題。
- 上一篇： Binder远程转本地
- 下一篇： Linux kernel同步机制