在安裝CDH版本的hadoop的時候, 我們下載下來的CDH的hadoop安裝包中, 并沒有native包,不支持各個任務(wù)階段的壓縮功能，但在生產(chǎn)任務(wù)中是必須要支持壓縮功能的，所以要對CDH進(jìn)行編譯以支持壓縮功能；

本次編譯使用的各軟件版本：

cdh：hadoop-2.6.0-cdh5.7.0-src.tar.gz jdk：1.7（經(jīng)多位同學(xué)踩坑，不要使用1.8版本的） maven：apache-maven-3.3.9-bin.tar.gz? protobuf：protobuf-2.5.0.tar.gz 系統(tǒng)：Centos?7.5cdh官方下載地址：http://archive.cloudera.com/cdh5/cdh/5

依賴的軟件百度網(wǎng)盤地址

repo.tar.gz	鏈接：https://pan.baidu.com/s/1wGCgV_3R3VUm2ka_aVA8GQ	提取碼：lrej
Hadoop Hadoop-2.6.0-cdh5.7.0-src.tar.gz?	鏈接：https://pan.baidu.com/s/1uRMGIhLSL9QHT-Ee4F16jw	提取碼：jb1d
jdk jdk-7u80-linux-x64.tar.gz?	鏈接：https://pan.baidu.com/s/1xSCQ8rjABVI-zDFQS5nCPA	提取碼：lfze
maven apache-maven-3.3.9-bin.tar.gz	鏈接：https://pan.baidu.com/s/1ddkdkLW7r7ahFZmgACGkVw	?提取碼：fdfz
protobuf protobuf-2.5.0.tar.gz	鏈接：https://pan.baidu.com/s/1RSNZGd_ThwknMB3vDkEfhQ?	提取碼：hvc2

首先安裝各種依賴：

yum?install?-y?svn?ncurses-devel yum?install?-y?gcc?gcc-c++?make?cmake yum?install?-y?openssl?openssl-devel?svn?ncurses-devel?zlib-devel?libtool yum?install?-y?snappy?snappy-devel?bzip2?bzip2-devel?lzo?lzo-devel?lzop?autoconf?automake?cmake

配置jdk的環(huán)境變量：

[root@hadoop004?soft]#?cat?/etc/profile.d/java.sh? export?JAVA_HOME=/usr/java/jdk export?CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar export?PATH=$JAVA_HOME/bin:$PATH[root@hadoop004?soft]#?java?-version java?version?"1.7.0_79" Java(TM)?SE?Runtime?Environment?(build?1.7.0_79-b15) Java?HotSpot(TM)?64-Bit?Server?VM?(build?24.79-b02,?mixed?mode)

配置maven的環(huán)境變量：

#配置環(huán)境變量 [root@hadoop004?soft]#?cat?/etc/profile.d/maven.sh? MAVEN_HOME=/usr/local/maven export?PATH=$MAVEN_HOME/bin:$PATH[root@hadoop004?local]#?mvn?--version Apache?Maven?3.3.9?(bb52d8502b132ec0a5a3f4c09453c07478323dc5;?2015-11-11T00:41:47+08:00) Maven?home:?/usr/local/maven Java?version:?1.7.0_79,?vendor:?Oracle?Corporation Java?home:?/usr/java/jdk1.7.0_79/jre Default?locale:?en_US,?platform?encoding:?UTF-8 OS?name:?"linux",?version:?"3.10.0-862.3.2.el7.x86_64",?arch:?"amd64",?family:?"unix"#配置mvn的本地存放地址 <localRepository>/usr/local/maven/repo</localRepository> 將上面下載repo.tar.gz解壓后的內(nèi)容放在此本地倉庫目錄下，以便于cdh編譯時直接使用，否則下載將浪費(fèi)大量的時間#配置mvn下載源為阿里云的maven倉庫 <mirror><id>alimaven</id><name>aliyun?maven</name><url>http://maven.aliyun.com/nexus/content/groups/public/</url><mirrorOf>central</mirrorOf> </mirror>

編譯安裝protobuf并配置環(huán)境變量：

#編譯安裝命令 tar?-zxvf?protobuf-2.5.0.tar.gz cd?protobuf-2.5.0 ./configure??--prefix=/root/protobuf/ make make?install#配置環(huán)境變量 [root@hadoop004?soft]#?cat?/etc/profile.d/protobuf.sh? PROTOC_HOME=/usr/local/protobuf export?PATH=$PROTOC_HOME/bin:$PATH#查看是否安裝成功 source?/etc/profile.d/protobuf.sh [root@hadoop004?soft]#?protoc?--version libprotoc?2.5.0

編譯cdh的hadoop源碼：

tar?-xzvf?hadoop-2.6.0-cdh5.7.0-src.tar.gz cd?hadoop-2.6.0-cdh5.7.0 mvn?clean?package?-Pdist,native?-DskipTests?-Dtar

遇到的錯誤：

[ERROR]?Failed?to?execute?goal?org.apache.maven.plugins:maven-antrun-plugin:1.7:run?(make)?on?project?hadoop-pipes:?An?Ant?BuildException?has?occured:?exec?returned:?1 [ERROR]?around?Ant?part?...<exec?dir="/data/soft/hadoop-2.6.0-cdh5.7.0/hadoop-tools/hadoop-pipes/target/native"?executable="cmake"?failοnerrοr="true">...?@?5:126?in?/data/soft/hadoop-2.6.0-cdh5.7.0/hadoop-tools/hadoop-pipes/target/antrun/build-main.xml [ERROR]?->?[Help?1] [ERROR]? [ERROR]?To?see?the?full?stack?trace?of?the?errors,?re-run?Maven?with?the?-e?switch. [ERROR]?Re-run?Maven?using?the?-X?switch?to?enable?full?debug?logging. [ERROR]? [ERROR]?For?more?information?about?the?errors?and?possible?solutions,?please?read?the?following?articles: [ERROR]?[Help?1]?http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException [ERROR]? [ERROR]?After?correcting?the?problems,?you?can?resume?the?build?with?the?command [ERROR]???mvn?<goals>?-rf?:hadoop-pipes

問題原因：在安裝依賴的過程，漏裝了openssl-devel；yum install -y?openssl-devel

再次執(zhí)行上面的安裝命令：

tar?-xzvf?hadoop-2.6.0-cdh5.7.0-src.tar.gz cd?hadoop-2.6.0-cdh5.7.0 mvn?clean?package?-Pdist,native?-DskipTests?-Dtar注：由于已經(jīng)提前將cdh需要下載的依賴，放入了maven的本地倉庫中了，所以節(jié)省了大量的下載時間

編譯后的文件：

????位置：./hadoop-dist/target目錄下

????文件：hadoop-2.6.0-cdh5.7.0

? ? 壓縮文件：hadoop-2.6.0-cdh5.7.0.tar.gz

????

測試查看是否支持各種壓縮：

[root@hadoop004?hadoop-2.6.0-cdh5.7.0]#?./bin/hadoop?checknative 19/04/18?15:09:34?INFO?bzip2.Bzip2Factory:?Successfully?loaded?&?initialized?native-bzip2?library?system-native 19/04/18?15:09:34?INFO?zlib.ZlibFactory:?Successfully?loaded?&?initialized?native-zlib?library Native?library?checking: hadoop:??true?/data/soft/hadoop-2.6.0-cdh5.7.0/hadoop-dist/target/hadoop-2.6.0-cdh5.7.0/lib/native/libhadoop.so.1.0.0 zlib:????true?/lib64/libz.so.1 snappy:??true?/lib64/libsnappy.so.1 lz4:?????true?revision:99 bzip2:???true?/lib64/libbz2.so.1 openssl:?true?/lib64/libcrypto.so

?????

移動到其它機(jī)器時的報錯及解決方案：

????注意如果拷貝tar包到新的機(jī)器安裝檢查checknative報錯

????openssl: false Cannot load libcrypto.so (libcrypto.so: 無法打開共享對象文件: 沒有那個文件或目錄)!

? ? ??

????centos中運(yùn)行yum install openssl-devel -y

? ? ??

轉(zhuǎn)載于:https://blog.51cto.com/14284607/2380601

創(chuàng)作挑戰(zhàn)賽新人創(chuàng)作獎勵來咯，堅(jiān)持創(chuàng)作打卡瓜分現(xiàn)金大獎

總結(jié)

以上是生活随笔為你收集整理的cdh编译安装支持各种压缩格式的全部內(nèi)容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網(wǎng)站內(nèi)容還不錯，歡迎將生活随笔推薦給好友。