當前位置：首頁 > 编程资源 > 编程问答 >内容正文

编程问答

在阿里云ECS 上安装 CDH6.2 standalong

發布時間：2023/12/8 编程问答 43 豆豆

生活随笔收集整理的這篇文章主要介紹了在阿里云ECS 上安装 CDH6.2 standalong 小編覺得挺不錯的,現在分享給大家,幫大家做個參考.

在阿里云ECS 上安裝 CDH6.2

0. pre Install

0.1 ECS 配置
基本配置至少 4核 16GB，比較燒內存

8核 + 64GB 100GB SSD 5M 寬帶

操作系統看 CDH 官網支持，推薦 RHEL6, RHEL7, CentOs7，總之不推薦 Ali Linux 或者 Amazon Linux

centos7.7

安全組

保證本地可以連接 7180, 22 端口

密鑰，win系統需要再轉換為 pub

xxx.pem密鑰
xxx.pem.pub 密鑰

CDH 6.2

下載路徑
公眾號【大數據1024】輸入 CDH
文件在 2.1 里面列出來了

配置的軟件

SecureCRT
FileZilla

1. 系統環境準備

1.1 設置 ssh 生存時間
避免遠程連接時一會不操作就斷開連接

vi /etc/ssh/sshd_configClientAliveInterval 30ClientAliveCountMax 1800

1.2 設置時區

timedatectl set-timezone Asia/Shanghai

1.3 設置主機名

vi /etc/hostnameStock-DMP source /etc/profile

1.4 安裝基礎包

yum -y install wget yum -y install lrzsz # SecureCRT 上傳下載小文件利器 yum -y install zip unzip yum -y install vim

1.5 關閉防火墻
阿里云防火墻默認是關閉的

systemctl status firewalld systemctl stop firewalld systemctl disable firewalld

1.6 關閉 SELINUX
阿里云 SELINUX 默認是關閉的

vim /etc/selinux/configSELINUX=disabled

1.7 修改 Linux swapping 和虛擬內存
我的機器默認 vm.swappiness = 0
swappiness = 0 表示最大限度使用物理內存，然后才是 swap 空間。有興趣自行搜索調整到合適參數。

vi /etc/sysctl.confvm.swappiness = 0vm.max_map_count=655360 sysctl -p

1.8 禁用 THP (Transparent HugePages)

對于數據庫應用，不推薦使用 THP。主要的原因是這類數據庫大部分訪問內存的方式是分散的，并不是訪問連續的頁面，而這樣的訪問模式,就會造成內存的碎片化。

# 查看 THP 狀態, Always 表示啟用 [root@Stock-DMP soft]# cat /sys/kernel/mm/transparent_hugepage/defrag [always] madvise never # 禁用 THP 并添加到配置文件 [root@Stock-DMP soft]# echo never > /sys/kernel/mm/transparent_hugepage/defrag [root@Stock-DMP soft]# echo never > /sys/kernel/mm/transparent_hugepage/enabled [root@Stock-DMP soft]# echo "echo never > /sys/kernel/mm/transparent_hugepage/defrag" >> /etc/rc.local [root@Stock-DMP soft]# echo "echo never > /sys/kernel/mm/transparent_hugepage/enabled" >> /etc/rc.local [root@Stock-DMP soft]# chmod +x /etc/rc.d/rc.local # 查看 THP 狀態, Never 表示禁用 [root@Stock-DMP soft]# cat /etc/rc.d/rc.local always madvise [never]

1.9 重啟服務器

reboot -h

2. 安裝環境

2.1 通過 FileZilla 上傳需要的文件
文件大致如下

mkdir -p /soft [root@Stock-DMP CDH6.2]# ll total 2754532 -rw-r--r-- 1 root root 2093332003 Jan 7 16:48 CDH-6.2.1-1.cdh6.2.1.p0.1425774-el7.parcel drwxr-xr-x 2 root root 4096 Jan 7 16:47 cloudera-repos -rw-r--r-- 1 root root 95840 Jan 7 16:33 createrepo-0.9.9-28.el7.noarch.rpm -rw-r--r-- 1 root root 541163520 Jan 7 16:37 mysql-5.7.31-1.el7.x86_64.rpm-bundle.tar -rw-r--r-- 1 root root 1007502 Jan 7 16:37 mysql-connector-java-5.1.47.jar -rw-r--r-- 1 root root 184988341 Jan 7 16:38 oracle-j2sdk1.8-1.8.0+update181-1.x86_64.rpm -rw-r--r-- 1 root root 32084 Jan 7 16:38 python-deltarpm-3.6-3.el7.x86_64.rpm [root@Stock-DMP CDH6.2]# ll cloudera-repos/ total 1181628 -rw-r--r-- 1 root root 14041 Jan 7 16:38 allkeys.asc -rw-r--r-- 1 root root 10215608 Jan 7 16:38 cloudera-manager-agent-6.2.1-1426065.el7.x86_64.rpm -rw-r--r-- 1 root root 1199718256 Jan 7 16:47 cloudera-manager-daemons-6.2.1-1426065.el7.x86_64.rpm -rw-r--r-- 1 root root 9988 Jan 7 16:47 cloudera-manager-server-6.2.1-1426065.el7.x86_64.rpm -rw-r--r-- 1 root root 10996 Jan 7 16:47 cloudera-manager-server-db-2-6.2.1-1426065.el7.x86_64.rpm -rw-r--r-- 1 root root 1718 Jan 7 16:47 RPM-GPG-KEY-cloudera [root@Stock-DMP CDH6.2]# pwd /soft/CDH6.2

2.2 安裝 JDK
CDH 6.2 支持 jdk1.8
其他版本注意版本對應關系
關于rpm, 可以參考 Linux RPM 命令參數解釋

[root@Stock-DMP CDH6.2]# rpm -ivh oracle-j2sdk1.8-1.8.0+update181-1.x86_64.rpm warning: oracle-j2sdk1.8-1.8.0+update181-1.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID b0b19c9f: NOKEY Preparing... ################################# [100%] Updating / installing...1:oracle-j2sdk1.8-1.8.0+update181-1################################# [100%] [root@Stock-DMP CDH6.2]# vi /etc/profile export JAVA_HOME=/usr/java/jdk1.8.0_181-cloudera export PATH=.:$PATH:$JAVA_HOME/bin [root@Stock-DMP CDH6.2]# source /etc/profile [root@Stock-DMP CDH6.2]# java -version java version "1.8.0_181" Java(TM) SE Runtime Environment (build 1.8.0_181-b13) Java HotSpot(TM) 64-Bit Server VM (build 25.181-b13, mixed mode) [root@Stock-DMP CDH6.2]# which java /usr/java/jdk1.8.0_181-cloudera/bin/java

2.3 安裝 Mysql
卸載 centos7 原生的 mariadb

[root@Stock-DMP CDH6.2]# rpm -qa|grep mariadb mariadb-libs-5.5.64-1.el7.x86_64 [root@Stock-DMP CDH6.2]# rpm -e --nodeps mariadb-libs-5.5.64-1.el7.x86_64

執行以下命令

[root@Stock-DMP CDH6.2]# tar -xvf mysql-5.7.31-1.el7.x86_64.rpm-bundle.tar [root@Stock-DMP CDH6.2]# rpm -ivh mysql-community-common-5.7.31-1.el7.x86_64.rpm --force --nodeps [root@Stock-DMP CDH6.2]# rpm -ivh mysql-community-libs-5.7.31-1.el7.x86_64.rpm --force --nodeps [root@Stock-DMP CDH6.2]# rpm -ivh mysql-community-client-5.7.31-1.el7.x86_64.rpm --force --nodeps [root@Stock-DMP CDH6.2]# rpm -ivh mysql-community-server-5.7.31-1.el7.x86_64.rpm --force --nodeps [root@Stock-DMP CDH6.2]# rpm -ivh mysql-community-libs-compat-5.7.31-1.el7.x86_64.rpm --force --nodeps # 驗證 [root@Stock-DMP CDH6.2]# rpm -qa|grep mysql mysql-community-libs-5.7.31-1.el7.x86_64 mysql-community-common-5.7.31-1.el7.x86_64 mysql-community-client-5.7.31-1.el7.x86_64 mysql-community-libs-compat-5.7.31-1.el7.x86_64 mysql-community-server-5.7.31-1.el7.x86_64

如果遇到以下 warning 是由于 yum 安裝了舊版本的 GPG keys 所造成，從 rpm 版本 4.1 后，在安裝或升級軟件包時會自動檢查軟件包的簽名參考資料

warning: mysql-community-common-5.7.31-1.el7.x86_64.rpm: Header V3 DSA/SHA1 Signature, key ID 5072e1f5: NOKEY

初始化 Mysql 并獲取初始密碼 farBa5Iys&PL

[root@Stock-DMP CDH6.2]# yum install -y libaio [root@Stock-DMP CDH6.2]# mysqld --initialize --user=mysql [root@Stock-DMP CDH6.2]# cat /var/log/mysqld.log | tail -1 2022-01-07T09:47:06.334066Z 1 [Note] A temporary password is generated for root@localhost: farBa5Iys&PL

登錄 mysql

# 啟動 mysql 服務 [root@Stock-DMP CDH6.2]# systemctl start mysqld.service # 設置 mysql 服務開機啟動 [root@Stock-DMP CDH6.2]# systemctl enable mysqld.service # 重啟 mysql 服務 [root@Stock-DMP CDH6.2]# systemctl restart mysqld.service # 登錄 mysql 并將 root 用戶密碼設置為 admin [root@Stock-DMP CDH6.2]# mysql -uroot -p Enter password: Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 2 Server version: 5.7.31Copyright (c) 2000, 2020, Oracle and/or its affiliates. All rights reserved.Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners.Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.mysql> alter user user() identified by "admin"; Query OK, 0 rows affected (0.00 sec)mysql>

2.3.1 創建 CM 需要的數據庫
在 3.6 配置數據庫需要使用

CREATE DATABASE scm DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci; CREATE DATABASE amon DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci; CREATE DATABASE rman DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci; CREATE DATABASE hue DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci; CREATE DATABASE metastore DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci; CREATE DATABASE sentry DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci; CREATE DATABASE nav DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci; CREATE DATABASE navms DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci; CREATE DATABASE oozie DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci; CREATE DATABASE hive DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;CREATE USER 'hive'@'%' IDENTIFIED BY 'hive@2021'; GRANT ALL ON *.* TO 'hive'@'%'; CREATE DATABASE hive; flush privileges; CREATE USER 'scm'@'%' IDENTIFIED BY 'scm@2021'; GRANT ALL ON *.* TO 'scm'@'%'; CREATE DATABASE scm; flush privileges; CREATE USER 'oozie'@'%' IDENTIFIED BY 'oozie@2021'; GRANT ALL ON *.* TO 'oozie'@'%'; CREATE DATABASE oozie; flush privileges; CREATE USER 'hue'@'%' IDENTIFIED BY 'hue@2021'; GRANT ALL ON *.* TO 'hue'@'%'; CREATE DATABASE hue; flush privileges;

更改 /etc/my.cnf 配置

vim /etc/my.cnf [mysqld] basedir = /usr/local/mysql datadir = /var/lib/mysql port = 3306 socket=/var/lib/mysql/mysql.sock character-set-server=utf8transaction-isolation = READ-COMMITTED # Disabling symbolic-links is recommended to prevent assorted security risks; # to do so, uncomment this line: symbolic-links = 0server_id=1 max-binlog-size = 500M log_bin=/var/lib/mysql/mysql_binary_log #binlog_format = mixed binlog_format = Row expire-logs-days = 14max_connections = 550 read_buffer_size = 2M read_rnd_buffer_size = 16M sort_buffer_size = 8M join_buffer_size = 8M# InnoDB settings innodb_file_per_table = 1 innodb_flush_log_at_trx_commit = 2 innodb_log_buffer_size = 64M innodb_buffer_pool_size = 4G innodb_thread_concurrency = 8 innodb_flush_method = O_DIRECT innodb_log_file_size = 512M[client] default-character-set=utf8 socket=/var/lib/mysql/mysql.sock[mysql] default-character-set=utf8 socket=/var/lib/mysql/mysql.sock[mysqld_safe] log-error=/var/log/mysqld.log pid-file=/var/run/mysqld/mysqld.pidsql_mode=STRICT_ALL_TABLES skip-ssl

MySQL 連接器配置
移動到新建目錄下并改為 mysql-connector-java.jar

[root@Stock-DMP CDH6.2]# mkdir -p /usr/share/java [root@Stock-DMP CDH6.2]# mv mysql-connector-java-5.1.47.jar /usr/share/java/mysql-connector-java.jar [root@Stock-DMP CDH6.2]# ll /usr/share/java total 984 -rw-r--r-- 1 root root 1007502 Jan 7 16:37 mysql-connector-java.jar

2.4 安裝第三方依賴

yum -y install chkconfig python bind-utils psmisc libxslt zlib sqlite cyrus-sasl-plain cyrus-sasl-gssapi fuse fuse-libs redhat-lsb httpd mod_ssl portmap /lib/lsb/init-functions openssl-devel python-psycopg2 MySQL-python yum -y install nfs-utils rpcbind

2.5 Cloudera Manager 相關文件
httpd 服務

[root@Stock-DMP CDH6.2]# yum install -y httpd [root@Stock-DMP CDH6.2]# systemctl start httpd.service [root@Stock-DMP CDH6.2]# systemctl enable httpd.service

移動文件到指定路徑

[root@Stock-DMP soft]# mkdir -p /var/www/html [root@Stock-DMP soft]# mv /soft/cloudera-repos /var/www/html [root@Stock-DMP soft]# ll /var/www/html/cloudera-repos/ total 1181628 -rw-r--r-- 1 root root 14041 Jan 7 16:38 allkeys.asc -rw-r--r-- 1 root root 10215608 Jan 7 16:38 cloudera-manager-agent-6.2.1-1426065.el7.x86_64.rpm -rw-r--r-- 1 root root 1199718256 Jan 7 16:47 cloudera-manager-daemons-6.2.1-1426065.el7.x86_64.rpm -rw-r--r-- 1 root root 9988 Jan 7 16:47 cloudera-manager-server-6.2.1-1426065.el7.x86_64.rpm -rw-r--r-- 1 root root 10996 Jan 7 16:47 cloudera-manager-server-db-2-6.2.1-1426065.el7.x86_64.rpm -rw-r--r-- 1 root root 1718 Jan 7 16:47 RPM-GPG-KEY-cloudera

瀏覽器可以訪問（IP 地址為阿里云公有 IP）

2.6 CDH Parcel 相關文件

mkdir -p /opt/cloudera/parcel-repo mv /soft/CDH6.2/CDH-6.2.1-1.cdh6.2.1.p0.1425774-el7.parcel /opt/cloudera/parcel-repo/ sha1sum CDH-6.2.1-1.cdh6.2.1.p0.1425774-el7.parcel | awk '{print $1}' > CDH-6.2.1-1.cdh6.2.1.p0.1425774-el7.parcel.sha useradd --system --home=/opt/cloudera-manager/cm-6.2.1/run/cloudera-scm-server --no-create-home --shell=/bin/false --comment "Cloudera SCM User" cloudera-scm chown -R cloudera-scm:cloudera-scm /opt/cloudera/parcel-repo/

2.7 Cloudera Manager 安裝
按順序安裝

[root@Stock-DMP cloudera-repos]# yum -y localinstall http://localhost/cloudera-repos/cloudera-manager-daemons-6.2.1-1426065.el7.x86_64.rpm [root@Stock-DMP cloudera-repos]# yum -y localinstall http://localhost/cloudera-repos/cloudera-manager-server-6.2.1-1426065.el7.x86_64.rpm [root@Stock-DMP cloudera-repos]# yum -y localinstall http://localhost/cloudera-repos/cloudera-manager-agent-6.2.1-1426065.el7.x86_64.rpm

初始化 Cloudera Manager Database

[root@Stock-DMP cloudera-repos]# /opt/cloudera/cm/schema/scm_prepare_database.sh mysql scm root admin JAVA_HOME=/usr/java/jdk1.8.0_181-cloudera Verifying that we can write to /etc/cloudera-scm-server Creating SCM configuration file in /etc/cloudera-scm-server Executing: /usr/java/jdk1.8.0_181-cloudera/bin/java -cp /usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/java/postgresql-connector-java.jar:/opt/cloudera/cm/schema/../lib/* com.cloudera.enterprise.dbutil.DbCommandExecutor /etc/cloudera-scm-server/db.properties com.cloudera.cmf.db. log4j:ERROR Could not find value for key log4j.appender.A log4j:ERROR Could not instantiate appender named "A". Fri Jan 07 18:58:59 CST 2022 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification. [2022-01-07 18:58:59,939] INFO 0[main] - com.cloudera.enterprise.dbutil.DbCommandExecutor.testDbConnection(DbCommandExecutor.java) - Successfully connected to database. All done, your SCM database is configured correctly!

啟動 Cloudera Manager

[root@Stock-DMP cloudera-repos]# systemctl start cloudera-scm-server [root@Stock-DMP cloudera-repos]# systemctl enable cloudera-scm-server [root@Stock-DMP cloudera-repos]# systemctl start cloudera-scm-agent [root@Stock-DMP cloudera-repos]# systemctl enable cloudera-scm-agent

通過瀏覽器訪問，初始用戶名 admin，密碼 admin

3. 配置Cloudera Manager

3.1 歡迎頁
3.2 版本選擇
3.3 集群名稱
3.4 集群名稱
選擇當前管理的主機，不然可能無法成功安裝
3.5 依照需求配置服務
后面可以加所以一開始不必要配置很多
3.6 配置數據庫
在 2.3.1 創建 CM 需要的數據庫操作過
3.7 啟動集群
這里可能會遇到一些問題，可能就是之前配置出了一些問題
3.8 完成
集群仍然有警告信息，由于 HDFS 副本不足和機器數量不足決定的

總結

以上是生活随笔為你收集整理的在阿里云ECS 上安装 CDH6.2 standalong的全部內容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網站內容還不錯，歡迎將生活随笔推薦給好友。