CentOS7搭建部署Ambari 2.6.2.0最新版(HDP-UTILS、HDP-GPL)大数据平台
生活随笔
收集整理的這篇文章主要介紹了
CentOS7搭建部署Ambari 2.6.2.0最新版(HDP-UTILS、HDP-GPL)大数据平台
小編覺得挺不錯的,現在分享給大家,幫大家做個參考.
如上述博客有任何錯誤或者疑問，請加VX：1755337994，及時告知！萬分感激！?
注：本文基于root用戶操作
一、安裝環境準備
操作系統 centos7.5
hdc-data1：192.168.163.51
hdc-data2：192.168.163.52
hdc-data3：192.168.163.53
【安裝環境準備每臺集群機器一樣的操作，或者使用scp遠程拷貝】
1、主機名IP映射配置
FQDN：(Fully Qualified Domain Name)全限定域名：同時帶有主機名和域名的名稱。（通過符號“.”）
例如：主機名是bigserver,域名是mycompany.com,那么FQDN就是bigserver.mycompany.com。
2、SSH免密登錄配置
ssh-keygen ssh-copy-id -i ~/.ssh/id_rsa.pub root@hdc-data1 ssh-copy-id -i ~/.ssh/id_rsa.pub root@hdc-data2 ssh-copy-id -i ~/.ssh/id_rsa.pub root@hdc-data3image.png
3、關閉及禁止防火墻
systemctl stop firewalld.service systemctl disable firewalld.service4、關閉SELinux
SELinux(Security-Enhanced Linux) 是美國國家安全局（NSA）對于強制訪問控制的實現，是 Linux歷史上最杰出的新安全子系統。NSA是在Linux社區的幫助下開發了一種訪問控制體系，在這種訪問控制體系的限制下，進程只能訪問那些在他的任務中所需要文件。
vi /etc/sysconfig/selinux #修改以下內容 SELINUX=disabledimage.png
5、開啟NTP服務
yum install -y ntp systemctl enable ntpd systemctl start ntpd6、安裝JDK
下載地址：https://www.oracle.com/technetwork/cn/java/javase/downloads/jdk8-downloads-2133151-zhs.html
通過文件傳輸上傳到服務器，這里不可用wget直接下載。
mkdir -p /opt/java tar -zxvf jdk-8u181-linux-x64.tar.gz -C /opt/java/ vi /etc/profileexport JAVA_HOME=/opt/java/jdk1.8.0_181 export JRE_HOME=$JAVA_HOME/jre export CLASSPATH=.:$JAVA_HOME/lib:$JAVA_HOME/jre/lib export PATH=$PATH:$JAVA_HOME/bin遠程分發到其他服務器
scp -r /opt/java/jdk1.8.0_181/ root@hdc-data2:/opt/java/ scp -r /opt/java/jdk1.8.0_181/ root@hdc-data3:/opt/java/scp /etc/profile root@hdc-data2:/etc/ scp /etc/profile root@hdc-data3:/etc/source /etc/profile二、Ambari安裝
1、制作Ambari本地yum源
選取一臺機器即可，在此選擇hdc-data1。
https://docs.hortonworks.com/HDPDocuments/Ambari-2.6.2.0/bk_ambari-installation/content/ch_obtaining-public-repos.html
image.png
?image.png
?image.png
需要下載資源安裝包如下：
wget http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.6.2.0/ambari-2.6.2.0-centos7.tar.gz wget http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.6.5.0/HDP-2.6.5.0-centos7-rpm.tar.gz wget http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.22/repos/centos7/HDP-UTILS-1.1.0.22-centos7.tar.gz wget http://public-repo-1.hortonworks.com/HDP-GPL/centos7/2.x/updates/2.6.5.0/HDP-GPL-2.6.5.0-centos7-gpl.tar.gz1.1 安裝 Apache HTTP 服務器
yum install httpd -y #開啟服務并設置開機自啟動 systemctl start httpd.service systemctl enable httpd.service1.2 安裝制作yum源工具
yum install yum-utils createrepo1.3 創建http服務器
http根目錄默認是即/var/www/html/
mkdir -p /var/www/html/ambari #cd /var/www/html/ambari #將下載好的安裝包上傳并解壓 tar xvf HDP-2.6.5.0-centos7-rpm.tar.gz -C /var/www/html/ambari tar xvf ambari-2.6.2.0-centos7.tar.gz -C /var/www/html/ambari tar xvf HDP-UTILS-1.1.0.22-centos7.tar.gz -C /var/www/html/ambari tar xvf HDP-GPL-2.6.5.0-centos7-gpl.tar.gz -C /var/www/html/ambari# 刪除壓縮包 rm -rf ambari-2.6.2.0-centos7.tar.gz rm -rf HDP-2.6.5.0-centos7-rpm.tar.gz rm -rf HDP-UTILS-1.1.0.22-centos7.tar.gz rm -rf HDP-GPL-2.6.5.0-centos7-gpl.tar.gz驗證
image.png
1.4 配置ambari、HDP、HDP-UTILS的本地源
#yum install wget -y cd /etc/yum.repos.d/ wget http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.6.2.0/ambari.repo wget http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.6.5.0/hdp.repo wget http://public-repo-1.hortonworks.com/HDP-GPL/centos7/2.x/updates/2.6.5.0/hdp.gpl.repo編輯ambari.repo，修改baseurl和gpgkey
[root@hdc-data1 yum.repos.d]# vi ambari.repo #VERSION_NUMBER=2.6.2.0-155 [ambari-2.6.2.0] name=ambari Version - ambari-2.6.2.0 #baseurl=http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.6.2.0 baseurl=http://192.168.163.51/ambari/ambari/centos7/2.6.2.0-155 gpgcheck=1 #gpgkey=http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.6.2.0/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins gpgkey=http://192.168.163.51/ambari/ambari/centos7/2.6.2.0-155/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins enabled=1 priority=1編輯hdp.repo，修改baseurl和gpgkey
[root@hdc-data1 yum.repos.d]# vi hdp.repo #VERSION_NUMBER=2.6.5.0-292 [HDP-2.6.5.0] name=HDP Version - HDP-2.6.5.0 #baseurl=http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.6.5.0 baseurl=http://192.168.163.51/ambari/HDP/centos7/2.6.5.0-292 gpgcheck=1 #gpgkey=http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.6.5.0/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins gpgkey=http://192.168.163.51/ambari/HDP/centos7/2.6.5.0-292/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins enabled=1 priority=1[HDP-UTILS-1.1.0.22] name=HDP-UTILS Version - HDP-UTILS-1.1.0.22 #baseurl=http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.22/repos/centos7 baseurl=http://192.168.163.51/ambari/HDP-UTILS/centos7/1.1.0.22 gpgcheck=1 #gpgkey=http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.6.5.0/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins gpgkey=http://192.168.163.51/ambari/HDP-UTILS/centos7/1.1.0.22/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins enabled=1 priority=1編輯hdp.gpl.repo，修改baseurl和gpgkey
[root@hdc-data1 yum.repos.d]# vi hdp.gpl.repo #VERSION_NUMBER=2.6.5.0-292 [HDP-GPL-2.6.5.0] name=HDP-GPL Version - HDP-GPL-2.6.5.0 #baseurl=http://public-repo-1.hortonworks.com/HDP-GPL/centos7/2.x/updates/2.6.5.0 baseurl=http://192.168.163.51/ambari/HDP-GPL/centos7/2.6.5.0-292 gpgcheck=1 #gpgkey=http://public-repo-1.hortonworks.com/HDP-GPL/centos7/2.x/updates/2.6.5.0/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins gpgkey=http://192.168.163.51/ambari/HDP-GPL/centos7/2.6.5.0-292/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins enabled=1 priority=1分發到其他機器
scp /etc/yum.repos.d/ambari.repo root@hdc-data2:/etc/yum.repos.d/ scp /etc/yum.repos.d/ambari.repo root@hdc-data3:/etc/yum.repos.d/ scp /etc/yum.repos.d/hdp.repo root@hdc-data2:/etc/yum.repos.d/ scp /etc/yum.repos.d/hdp.repo root@hdc-data3:/etc/yum.repos.d/ scp /etc/yum.repos.d/hdp.gpl.repo root@hdc-data2:/etc/yum.repos.d/ scp /etc/yum.repos.d/hdp.gpl.repo root@hdc-data3:/etc/yum.repos.d/每臺機器yum配置
yum clean all yum makecache yum list2、安裝Mysql數據庫
Ambari安裝會將安裝等信息寫入數據庫，建議使用MariaDB數據庫，也可以不安裝而使用默認數據庫PostgreSQL
2.1 安裝及初始化設置
[root@hdc-data1 ~]# yum install mariadb-server [root@hdc-data1 ~]# systemctl start mariadb [root@hdc-data1 ~]# systemctl enable mariadb [root@hdc-data1 ~]# mysql_secure_installation#首先是設置密碼，會提示先輸入密碼 Enter current password for root (enter for none):<–初次運行直接回車 #設置密碼 Set root password? [Y/n] <– 是否設置root用戶密碼，輸入y并回車或直接回車 New password: <– 設置root用戶的密碼 Re-enter new password: <– 再輸入一次你設置的密碼 #其他配置 Remove anonymous users? [Y/n] <– 是否刪除匿名用戶，回車 Disallow root login remotely? [Y/n] <–是否禁止root遠程登錄,回車, Remove test database and access to it? [Y/n] <– 是否刪除test數據庫，回車 Reload privilege tables now? [Y/n] <– 是否重新加載權限表，回車【可選】修改mysql端口（生產環境安全考慮）
查看端口
image.png
?image.png
2.2 安裝完成后創建ambari數據庫及用戶
#進入mysql shell mysql -uroot -p create database ambari character set utf8 ; CREATE USER 'ambari'@'%'IDENTIFIED BY 'ambari123'; GRANT ALL PRIVILEGES ON *.* TO 'ambari'@'%'; FLUSH PRIVILEGES;如果要安裝Hive，創建Hive數據庫和用戶
create database hive character set utf8 ; CREATE USER 'hive'@'%'IDENTIFIED BY 'hive123'; GRANT ALL PRIVILEGES ON *.* TO 'hive'@'%'; FLUSH PRIVILEGES;如果要安裝Oozie，創建Oozie數據庫和用戶
create database oozie character set utf8 ; CREATE USER 'oozie'@'%'IDENTIFIED BY 'oozie123'; GRANT ALL PRIVILEGES ON *.* TO 'oozie'@'%'; FLUSH PRIVILEGES;3、安裝 ambari
yum install ambari-server4、下載mysql驅動
http://central.maven.org/maven2/mysql/mysql-connector-java/5.1.40/
將mysql-connector-Java.jar復制到/usr/share/java目錄下
將mysql-connector-java.jar復制到/var/lib/ambari-server/resources目錄下
cp mysql-connector-java-5.1.40.jar /var/lib/ambari-server/resources/mysql-jdbc-driver.jar5、編輯/etc/ambari-server/conf/ambari.properties，添加如下內容
server.jdbc.driver.path=/usr/share/java/mysql-connector-java.jar #【可選】修改默認8080端口 #client.api.port=180806、ambaria初始化
#設置mysql驅動 ambari-server setup --jdbc-db=mysql --jdbc-driver=/usr/share/java/mysql-connector-java.jar ambari-server setup完整初始化如下：
[root@hdc-data1 ~]# ambari-server setup Using python /usr/bin/python Setup ambari-server Checking SELinux... SELinux status is 'disabled' Customize user account for ambari-server daemon [y/n] (n)? y Enter user account for ambari-server daemon (root): Adjusting ambari-server permissions and ownership... Checking firewall status... Checking JDK... [1] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8 [2] Oracle JDK 1.7 + Java Cryptography Extension (JCE) Policy Files 7 [3] Custom JDK ============================================================================== Enter choice (1): 3 WARNING: JDK must be installed on all hosts and JAVA_HOME must be valid on all hosts. WARNING: JCE Policy files are required for configuring Kerberos security. If you plan to use Kerberos,please make sure JCE Unlimited Strength Jurisdiction Policy Files are valid on all hosts. Path to JAVA_HOME: /opt/java/jdk1.8.0_181 Validating JDK on Ambari Server...done. Checking GPL software agreement... GPL License for LZO: https://www.gnu.org/licenses/old-licenses/gpl-2.0.en.html Enable Ambari Server to download and install GPL Licensed LZO packages [y/n] (n)? y Completing setup... Configuring database... Enter advanced database configuration [y/n] (n)? y Configuring database... ============================================================================== Choose one of the following options: [1] - PostgreSQL (Embedded) [2] - Oracle [3] - MySQL / MariaDB [4] - PostgreSQL [5] - Microsoft SQL Server (Tech Preview) [6] - SQL Anywhere [7] - BDB ============================================================================== Enter choice (1): 3 Hostname (localhost): Port (3306): Database name (ambari): Username (ambari): Enter Database Password (bigdata): Re-enter password: Configuring ambari database... Configuring remote database connection properties... WARNING: Before starting Ambari Server, you must run the following DDL against the database to create the schema: /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql Proceed with configuring remote database connection properties [y/n] (y)? y Extracting system views... .....ambari-admin-2.6.2.0.155.jar ...... Adjusting ambari-server permissions and ownership... Ambari Server 'setup' completed successfully.說明：
1：提示是否自定義設置。輸入：y 或 按回車繼續
2：ambari-server 賬號。使用root（推薦）用戶直接回車 若使用ambari用戶輸入：ambari
3：設置JDK,輸入數字3，選客戶端安裝jdk，如果服務器可以訪問外網，則可以選1，自動下載jdk1.8，默認下載的安裝目錄是/usr/java/default
4：如果上面選擇3自定義JDK,則需要設置JAVA_HOME
5：輸入：y 或 按回車繼續
6：輸入“y“，進行數據庫的高級配置
7：輸入“3“，選擇mysql/mariadb數據庫
8：設置數據庫的配置參數：host,port,database,username,password。根據實際情況輸入，如果和括號內相同，則可以直接回車。若端口改6033則輸入port:6033。
9：輸入：y 或 按回車繼續
（若設置更改，可以再次執行ambari-server setup進行設置）
7、將Ambari數據庫腳本導入到數據庫
#用Ambari用戶（上面設置的用戶）登錄mysql mysql -u ambari -p use ambari; source /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sqlimage.png
8、啟動Ambari
ambari-server startimage.png
瀏覽器訪問：http://hdc-data1:8080/?默認登錄用戶：admin，密碼：admin
9、WEBUI配置
image.png
? 創建集群名稱image.png
? 選擇HDP版本和選擇本地倉庫image.png
配置HDP的Repository
image.png
輸入集群節點host(FQDN)和Ambari節點SSH的私鑰
image.png
? 等待ambari-agents注冊image.png
若報錯
image.png
?image.png
【遇到問題 Confirm Hosts】
注冊ambari-agents時failed
NetUtil.py:96 - EOF occurred in violation of protocol (_ssl.c:579)
SSLError: Failed to connect. Please check openssl library versions.
解決：如下第三個方案，/etc/ambari-agent/conf/ambari-agent.ini文件只有執行該步驟進行注冊才生成。修改后點擊“Retry Failed”按鈕，等待注冊成功進入下一步。
image.png
選擇安裝組件，（最小化安裝HDFS+YARN+MAPREDUCE2+Ambari Metrics+SmartSense+ZooKeeper+Hbase）不必要的組件可以暫時不安裝節省安裝時間，后續可以再添加安裝組件image.png
?image.png
? 實際部署時需按規劃設計進行分配（注意：Hbase HA 的話在Hbase Master 點擊后面的小綠色+號，添加standby master）：image.png
工作節點角色分配(生產環境全勾了)
image.png
組件配置信息修改
通常數據目錄修改到/data/目錄下，有紅色信息提示的組件表示要輸入相應的賬號密碼。以下示例，可根據自己的實際情況配置。 HDFS
image.png
?image.png
HIVE,若數據庫端口修改按實際填寫
image.png
Oozie
image.png
Zookeeper
image.png
?image.png
?image.png
?image.png
暫時修改了以下屬性，具體配置信息可根據實際情況后續相應修改
| dfs.namenode.http-address | server1.hadoop:5070 （def:50070） |
| yarn.resourcemanager.webapp.address | server2.hadoop:8888 （def:8088） |
| yarn.resourcemanager.webapp.https.address | server2.hadoop:8890(def:8090) |
| mapreduce.jobhistory.webapp.address | server2.hadoop:18888 (def:19888) |
| HDFS | |
| NameNode | /data/hadoop/hdfs/namenode |
| DataNode | /data/hadoop/hdfs/data |
| SecondaryNameNode Checkpoint directories | /data/hadoop/hdfs/namesecondary |
| Hadoop PID Dir Prefix | /data/var/run/hadoop |
| Hadoop Log Dir Prefix | /data/var/log/hadoop |
| dfs.journalnode.edits.dir | /data/hadoop/hdfs/journalnode |
| Yarn | |
| yarn.nodemanager.local-dirs | /data/hadoop/yarn/local |
| yarn.nodemanager.log-dirs | /data/hadoop/yarn/log |
| yarn.timeline-service.leveldb-state-store.path | /data/hadoop/yarn/timeline |
| yarn.timeline-service.leveldb-timeline-store.path | /data/hadoop/yarn/timeline |
| YARN Log Dir Prefix | /data/var/log/hadoop-yarn |
| YARN PID Dir Prefix | /data/var/run/hadoop-yarn |
| Mapreduce | |
| Mapreduce Log Dir Prefix | /data/var/log/hadoop-mapreduce |
| Mapreduce PID Dir Prefix | /data/var/run/hadoop-mapreduce |
| mapreduce.jobhistory.recovery.store.leveldb.path | /data/hadoop/mapreduce/jhs |
| Hive | |
| Hive Log Dir | /data/var/log/hive |
| Hive PID Dir | /data/var/run/hive |
| HBase | |
| HBase Log Dir Prefix | /data/var/log/hbase |
| HBase PID Dir | /data/var/run/hbase |
| Oozie | |
| Oozie Data Dir | /data/hadoop/oozie/data |
| Oozie Log Dir | /data/var/log/oozie |
| Oozie PID Dir | /data/var/run/oozie |
| zookeeper | |
| ZooKeeper directory | /data/hadoop/zookeeper |
| ZooKeeper Log Dir | /data/var/log/zookeeper |
| ZooKeeper PID Dir | /data/var/run/zookeeper |
| ambari-infra | |
| Metrics Collector log dir | /data/var/log/ambari-metrics-collector |
| Metrics Collector pid dir | /data/var/run/ambari-metrics-collector |
| Metrics Monitor log dir | /data/var/log/ambari-metrics-monitor |
| Metrics Monitor pid dir | /data/var/run/ambari-metrics-monitor |
| Aggregator checkpoint directory | /data/var/lib/ambari-metrics-collector/checkpoint |
| Metrics Grafana data dir | /data/var/lib/ambari-metrics-grafana |
| Metrics Grafana log dir | /data/var/log/ambari-metrics-grafana |
| Metrics Grafana pid dir | /data/var/run/ambari-metrics-grafana |
| hbase_log_dir | /data/var/log/ambari-metrics-collector |
| hbase_pid_dir | /data/var/run/ambari-metrics-collector/ |
| hbase.tmp.dir | /data/var/lib/ambari-metrics-collector/hbase-tmp |
| ambari-infra | |
| Infra Solr Client log dir | /data/var/log/ambari-infra-solr-client |
| Infra Solr log dir | /data/var/log/ambari-infra-solr |
| Infra Solr pid dir | /data/var/run/ambari-infra-solr |
| spark | |
| livy2_log_dir | /data/var/log/livy2 |
| livy2_pid_dir | /data/var/run/livy2 |
| spark_log_dir | /data/var/log/spark2 |
| spark_pid_dir | /data/var/run/spark2 |
image.png
? 直到所有節點成功安裝完成才能進行下一步操作image.png
最終結果界面類似如下：
image.png
安裝完成！
三、后續操作
操作前最好先將集群打個快照備份
1、HDFS啟用HA(高可用)
參考http://www.louisvv.com/archives/1490.html
step1：關閉hbase、hive相關服務
image.png
step2:在HDFS選擇Enable NameNode HA
image.png
step3:輸入namenode ha serverid
image.png
step4:默認，進入下一步
image.png
step5:預覽，直接進入下一步
image.png
step6:創建checkpoint，根據提示執行下列命令
image.png
根據提示在server1執行下列兩條命令
image.png
命令執行完成方可進入下一步
image.png
step7:等待配置進程完成
image.png
step8:Journalnode初始化
image.png
需要在原NameNode節點執行下列命令 sudo su hdfs -l -c 'hdfs namenode -initializeSharedEdits'
image.png
step9:?啟動組件，進入下一步
image.png
step10:初始化元數據， sudo su hdfs -l -c 'hdfs zkfc -formatZK'
image.png
注意在新添加的NameNode節點上操作 sudo su hdfs -l -c 'hdfs namenode -bootstrapStandby'
image.png
?image.png
step11:等待所有服務重啟，完成
image.png
?image.png
總結
以上是生活随笔為你收集整理的CentOS7搭建部署Ambari 2.6.2.0最新版(HDP-UTILS、HDP-GPL)大数据平台的全部內容，希望文章能夠幫你解決所遇到的問題。
- 上一篇： Django代码编写规范
- 下一篇： Tensorflow Object de