腾讯云大数据套件Hermes-MR索引插件使用总结
2019獨角獸企業重金招聘Python工程師標準>>>
版權聲明：本文由王亮原創文章，轉載請注明出處:?
文章原文鏈接：https://www.qcloud.com/community/article/121
來源：騰云閣?https://www.qcloud.com/community
?
Hermes是多維分析利器，使用步驟分為索引創建和數據分發兩個步驟。
Hermes目前尚未集成到TBDS套件（3.0版本）中且外部有客戶需要在自己部署的集群上使用Hermes組件，這里就遇到了Hermes與外部Hadoop集群的適配問題。
Hermes與某客戶外部集群集成后，一次壓測時（2T數據量，445604010行，477字段全索引）使用單機版的Hermes索引創建插件由于數據量過大，出現Out of Memory等異常現象導致索引插件程序崩潰，實際產生的數據索引量和實際數據量差距很大。基于以上考慮，數平提供了基于MR的索引創建插件，提升索引創建效率。
以下記錄了基于hadoop2.2版本的MR索引插件和外部集群的適配過程。
一.集群相關組件版本
Hermes版本：hermes-2.1.0-1.x86_64
Hadoop集群版本：Hadoop 2.7.1.2.3.0.0-2557
Hermes-index-MR插件使用的Hadoop-common：hadoop-common-2.2.0.jar
二.Hermes-MR插件使用方法
1.需修改配置：（以$HERMES_INDEX_MR_HOME表示插件主目錄）
-
$HERMES_INDEX_MR_HOME/conf/hermes.properties
修改內容：hermes.zkConnectionString更改為本集群的zookeeper地址；hermes.hadoop.conf.dir修改為本集群的hadoop配置目錄；hermes.hadoop.home修改為本集群的hadoop安裝主目錄。 -
$HERMES_INDEX_MR_HOME/conf/hermes_index.properties
修改內容：hermes.hadoop.conf更改為本集群的hadoop配置目錄；hermes.index.user.conf更改為hermes-MR-index插件的用戶配置文件絕對地址。 - $HERMES_INDEX_MR_HOME/conf/user_conf.xml
修改內容：該配置即hermes-MR-index插件的用戶配置文件，一般默認配置項即可。需要注意的是插件支持指定被索引文件的字段分隔符。配置項為higo.input.record.split和higo.input.record.ascii.split。其中higo.input.record.ascii.split的優先級高于前者，指定higo.input.record.ascii.split后第一個配置將無效。其中higo.input.record.split的value項直接指定分隔符內容（如|，\，；等）；higo.input.record.ascii.split指定分隔符對應的ascii碼數字。
2.運行插件
-
執行命令：在插件主目錄下(其中labcluster為HDFS的nn通過做HA的名稱)：
sh bin/submit_index_job.sh \ clk_tag_info_test_500 \ 20160722 \ hdfs://labcluster/apps/hive/market_mid/clk_tag_info_test/ \ hdfs://labcluster/user/hermes/demo_dir/clk_tag_info_test_500/ \ hdfs://labcluster/user/hermes/demo_dir/schema/clk_tag_info_test_500_hermes.schema \ key_id \ 3 -
參數介紹：
sh bin/submit_index_job.sh表名 數據時間（時間分區） 源數據在HDFS上地址（單文件或目錄） 索引輸出的HDFS目錄 schema文件在HDFS的地址（需手動創建上傳） 主鍵 索引分片數
3.日志觀察：
創建索引插件在運行后會在$HERMES_INDEX_MR_HOME/logs輸出hermes.log和index.log。前者為hermes相關的記錄，后者為索引創建過程記錄（包括MR任務相關信息）。正常情況下index.log會記錄提交MR任務成功與否以及相關jobid，可通過HADOOP的RM管理頁面看到狀態，index.log也會記錄Map/Reduce的進度，完成后會輸出Job ${job.id} completed successfully以及MR任務相關信息（如圖）。如果出現錯誤日志，需具體分析，下文會總結本次集群適配遇到的一系列問題，目前已在TBDS3.0（Hadoop2.7）集群里測試通過。
4.適配基本過程
前面已提到Hermes-MR-index插件使用的Hadoop-common.jar版本為2.2，但集群本身為Hadoop2.7。在直接執行插件創建索引時出現以下“奇怪”異常。
Diagnostics: Exception from container-launch. Container id: container_e07_1469110119300_0022_02_000001 Exit code: 255 Stack trace: ExitCodeException exitCode=255: at org.apache.hadoop.util.Shell.runCommand(Shell.java:545) at org.apache.hadoop.util.Shell.run(Shell.java:456) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:722) at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:211) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745)查詢了所有異常日志后一無所獲，和數平Hadoop大神請教后，建議替換Hermes-MR-index插件里用到Hadoop*.jar包為集群內版本。這樣開始還是遇到了一系列問題，最終在hadoop2.7環境下Hermes-MR-index插件運行正常。
整理了以下思路進行適配：1.將Hermes-MR-index插件用到的hadoop-*.jar全部替換為集群內使用的版本；2.執行插件看日志錯誤一般會因為新版（2.7）有新的jar包依賴關系，提示錯誤，根據錯誤提示缺少的類找到對應jar包，添加到$HERMES_INDEX_MR_HOME/lib目錄，重復此操作，直到不再提示缺少類錯誤。3.執行以上操作時同時需要注意缺少的類關聯的jar包的版本必須和實際集群用到的版本一致（重復步驟2時發現的問題）。
5.問題匯總
插件和集群的適配過程中遇到的問題總結如下：
-
配置項mapreduce.framework.name異常
2016-07-21 15:39:51,522 (ERROR org.apache.hadoop.security.UserGroupInformation 1600): PriviledgedActionException as:root (auth:SIMPLE) cause:java.io.IOException: Cannot initialize Cluster. Please check your configuration for mapreduce.framework.name and the correspond server addresses. Exception in thread "main" java.io.IOException: Cannot initialize Cluster. Please check your configuration for mapreduce.framework.name and the correspond server addresses.解決方法：查看集群的hadoop相關配置（即hermes.properties里指定的hadoop配置路徑里配置目錄，也可以復制集群的出來，自己做單獨修改）mapred-site.xml里的mapreduce.framework.name配置項內容為yarn-tez，但目前插件只支持到yarn，故單獨修改此項配置為yarn后保存，異常解決。
-
插件無法向集群提交任務
2016-07-21 20:14:49,355 (ERROR org.apache.hadoop.security.UserGroupInformation 1600): PriviledgedActionException as:hermes (auth:SIMPLE) cause:java.io.IOException: Failed to run job : org.apache.hadoop.security.AccessControlException: User hermes cannot submit applications to queue root.default解決方法：使用hermes用戶向yarn提交任務時無權限提示。修改yarn集群的權限允許hermes即可。TBDS3.0有很方便的訪問控制頁面進行操作。
-
提交任務時變量替換異常
Exception message: /hadoop/data1/hadoop/yarn/local/usercache/hermes/appcache/applicati on_1469110119300_0004/container_e07_1469110119300_0004_02_000001/lau nch_container.sh: line 9: $PWD:$HADOOP_CONF_DIR:/usr/hdp/current/hadoop- client/*:/usr/hdp/current/hadoop- client/lib/*:/usr/hdp/current/hadoop-hdfs- client/*:/usr/hdp/current/hadoop-hdfs- client/lib/*:/usr/hdp/current/hadoop-yarn- client/*:/usr/hdp/current/hadoop-yarn-client/lib/*:$PWD/mr- framework/hadoop/share/hadoop/mapreduce/*:$PWD/mr- framework/hadoop/share/hadoop/mapreduce/lib/*:$PWD/mr- framework/hadoop/share/hadoop/common/*:$PWD/mr- framework/hadoop/share/hadoop/common/lib/*:$PWD/mr- framework/hadoop/share/hadoop/yarn/*:$PWD/mr- framework/hadoop/share/hadoop/yarn/lib/*:$PWD/mr- framework/hadoop/share/hadoop/hdfs/*:$PWD/mr- framework/hadoop/share/hadoop/hdfs/lib/*:$PWD/mr- framework/hadoop/share/hadoop/tools/lib/*:/usr/hdp/${hdp.version}/ha doop/lib/hadoop-lzo- 0.6.0.${hdp.version}.jar:/etc/hadoop/conf/secure:job.jar/job.jar:job .jar/classes/:job.jar/lib/*:$PWD/*: bad substitution /hadoop/data1/hadoop/yarn/local/usercache/hermes/appcache/applicatio n_1469110119300_0004/container_e07_1469110119300_0004_02_000001/laun ch_container.sh: line 67: $JAVA_HOME/bin/java - Dlog4j.configuration=container-log4j.properties - Dyarn.app.container.log.dir=/hadoop/data1/yarn/container- logs/application_1469110119300_0004/container_e07_1469110119300_0004 _02_000001 -Dyarn.app.container.log.filesize=0 - Dhadoop.root.logger=INFO,CLA -Dhdp.version=${hdp.version} -Xmx5120m org.apache.hadoop.mapreduce.v2.app.MRAppMaster 1>/hadoop/data1/yarn/container- logs/application_1469110119300_0004/container_e07_1469110119300_0004 _02_000001/stdout 2>/hadoop/data1/yarn/container- logs/application_1469110119300_0004/container_e07_1469110119300_0004 _02_000001/stderr : bad substitution Stack trace: ExitCodeException exitCode=1: /hadoop/data1/hadoop/yarn/local/usercache/hermes/appcache/applicatio n_1469110119300_0004/container_e07_1469110119300_0004_02_000001/laun ch_container.sh: line 9: $PWD:$HADOOP_CONF_DIR:/usr/hdp/current/hadoop- client/*:/usr/hdp/current/hadoop- client/lib/*:/usr/hdp/current/hadoop-hdfs- client/*:/usr/hdp/current/hadoop-hdfs- client/lib/*:/usr/hdp/current/hadoop-yarn- client/*:/usr/hdp/current/hadoop-yarn-client/lib/*:$PWD/mr- framework/hadoop/share/hadoop/mapreduce/*:$PWD/mr- framework/hadoop/share/hadoop/mapreduce/lib/*:$PWD/mr- framework/hadoop/share/hadoop/common/*:$PWD/mr- framework/hadoop/share/hadoop/common/lib/*:$PWD/mr- framework/hadoop/share/hadoop/yarn/*:$PWD/mr- framework/hadoop/share/hadoop/yarn/lib/*:$PWD/mr- framework/hadoop/share/hadoop/hdfs/*:$PWD/mr- framework/hadoop/share/hadoop/hdfs/lib/*:$PWD/mr- framework/hadoop/share/hadoop/tools/lib/*:/usr/hdp/${hdp.version}/ha doop/lib/hadoop-lzo- 0.6.0.${hdp.version}.jar:/etc/hadoop/conf/secure:job.jar/job.jar:job .jar/classes/:job.jar/lib/*:$PWD/*: bad substitution /hadoop/data1/hadoop/yarn/local/usercache/hermes/appcache/applicatio n_1469110119300_0004/container_e07_1469110119300_0004_02_000001/laun ch_container.sh: line 67: $JAVA_HOME/bin/java - Dlog4j.configuration=container-log4j.properties - Dyarn.app.container.log.dir=/hadoop/data1/yarn/container- logs/application_1469110119300_0004/container_e07_1469110119300_0004 _02_000001 -Dyarn.app.container.log.filesize=0 - Dhadoop.root.logger=INFO,CLA -Dhdp.version=${hdp.version} -Xmx5120m org.apache.hadoop.mapreduce.v2.app.MRAppMaster 1>/hadoop/data1/yarn/container- logs/application_1469110119300_0004/container_e07_1469110119300_0004 _02_000001/stdout 2>/hadoop/data1/yarn/container- logs/application_1469110119300_0004/container_e07_1469110119300_0004 _02_000001/stderr : bad substitution解決方法：從bad substitution可以判定為是某些配置的參數沒有正常替換造成。查看具體異常里面用到的變量有$PWD，$JAVA_HOME，${hdp.version}和$HADOOP_CONF_DIR以上變量在hadoop的配置文件里找到逐個替換為實際值而不用變量直到錯誤提示不再出現。實踐中發現是因為hdp.version這個變量沒有值造成的，可以在hadoop配置里增加一項此配置或者將用到該變量的地方替換為實際值即可。
-
一個“奇怪的”錯誤
2016-07-22 15:25:40,657 (INFO org.apache.hadoop.mapreduce.Job 1374): Job job_1469110119300_0022 failed with state FAILED due to: Application application_1469110119300_0022 failed 2 times due to AM Container for appattempt_1469110119300_0022_000002 exited with exitCode: 255 For more detailed output, check application tracking page:http://bdlabnn2:8088/cluster/app/application_1469110119300_0022 Then, click on links to logs of each attempt. Diagnostics: Exception from container-launch. Container id: container_e07_1469110119300_0022_02_000001 Exit code: 255 Stack trace: ExitCodeException exitCode=255: at org.apache.hadoop.util.Shell.runCommand(Shell.java:545) at org.apache.hadoop.util.Shell.run(Shell.java:456) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java :722) at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.l aunchContainer(DefaultContainerExecutor.java:211) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher. ContainerLaunch.call(ContainerLaunch.java:302)解決方法：這個錯誤是最難解決的錯誤，最終是用本文提到的插件和集群版本適配的辦法解決，解決方法及思路見“適配基本過程”。替換或者增加了的jar包列表如下：
jackson-core-2.2.3.jar jersey-json-1.9.jar jersey-client-1.9.jar jersey-core-1.9.jar jackson-xc-1.9.13.jar jersey-guice-1.9.jar jersey-server-1.9.jar jackson-jaxrs-1.9.13.jar commons-io-2.5.jar htrace-core-3.1.0-incubating.jar hermes-index-2.1.2.jar hadoop-cdh3-hdfs-2.2.0.jar hadoop-cdh3-core-2.2.0.jar hadoop-yarn-common-2.7.2.jar hadoop-yarn-client-2.7.2.jar hadoop-yarn-api-2.7.2.jar hadoop-mapreduce-client-jobclient-2.7.2.jar hadoop-mapreduce-client-core-2.7.2.jar hadoop-mapreduce-client-common-2.7.2.jar hadoop-hdfs-2.7.2.jar hadoop-common-2.7.2.jar hadoop-auth-2.7.2.jar -
無法連接yarn的RM任務提交端口
在TBDS3.0的環境下提交任務后日志提示重連RMserver失敗，一直提示該錯誤
解決方法：查看啟動進程發現內部集群接收mr請求的端口為8032，修改項里的RMserveraddress配置的端口后任務通過 -
適配完成替換/新增所有jar包后出現的異常
Exception in thread "main" java.lang.VerifyError: class org.codehaus.jackson.xc.JaxbAnnotationIntrospector overrides final method findDeserializer.(Lorg/codehaus/jackso n/map/introspect/Annotated;)Ljava/lang/Object; at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:800) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) at java.net.URLClassLoader.defineClass(URLClassLoader.java:449) at java.net.URLClassLoader.access$100(URLClassLoader.java:71) at java.net.URLClassLoader$1.run(URLClassLoader.java:361) at java.net.URLClassLoader$1.run(URLClassLoader.java:355) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:354) at java.lang.ClassLoader.loadClass(ClassLoader.java:425) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) at java.lang.ClassLoader.loadClass(ClassLoader.java:358) at java.lang.Class.getDeclaredMethods0(Native Method) at java.lang.Class.privateGetDeclaredMethods(Class.java:2615) at java.lang.Class.getDeclaredMethods(Class.java:1860) at com.sun.jersey.core.reflection.MethodList.getAllDeclaredMethods(Meth odList.java:70) at com.sun.jersey.core.reflection.MethodList.<init>(MethodList.java:64) at com.sun.jersey.core.spi.component.ComponentConstructor.getPostConstr uctMethods(ComponentConstructor.java:131) at com.sun.jersey.core.spi.component.ComponentConstructor.<init>(ComponentConstructor.java:123) at com.sun.jersey.core.spi.component.ProviderFactory.__getComponentProv ider(ProviderFactory.java:165) at com.sun.jersey.core.spi.component.ProviderFactory._getComponentProvider(ProviderFactory.java:159) at com.sun.jersey.core.spi.component.ProviderFactory.getComponentProvider(ProviderFactory.java:153) at com.sun.jersey.core.spi.component.ProviderServices.getComponent(ProviderServices.java:251)解決方法：查詢這個異常類屬于jackson*.jar，那問題就出在這一系列的包身上，檢查發現Hermes-MR-index插件的lib目錄下有
jackson-core-asl-1.7.3.jar jackson-mapper-asl-1.7.3.jar jackson-core-asl-1.9.13.jar jackson-mapper-asl-1.9.13.jar這兩個包的版本有2個，檢查Hadoop集群用的版本為1.9.13，將插件lib目錄下的1.7.3版本的兩個包刪除后，插件正常運行。原因歸結為jar包版本沖突。
-
提示無法找到MR框架路徑
Exception in thread "main" java.lang.IllegalArgumentException: Could not locate MapReduce framework name 'mr-framework' in mapreduce.application.classpath at org.apache.hadoop.mapreduce.v2.util.MRApps.setMRFrameworkClasspath(M RApps.java:231) at org.apache.hadoop.mapreduce.v2.util.MRApps.setClasspath(MRApps.java:258) at org.apache.hadoop.mapred.YARNRunner.createApplicationSubmissionContext(YARNRunner.java:458) at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:285) at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:240) at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1290) at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1287) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) at org.apache.hadoop.mapreduce.Job.submit(Job.java:1287) at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1308) at com.tencent.hermes.hadoop.job.HermesIndexJob.subRun(HermesIndexJob.java:262) at com.tencent.hermes.hadoop.job.HermesIndexJob.run(HermesIndexJob.java:122) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at com.tencent.hermes.hadoop.job.SubmitIndexJob.call(SubmitIndexJob.java:194) at com.tencent.hermes.hadoop.job.SubmitIndexJob.main(SubmitIndexJob.java:101)解決方法：提示mapreduce.application.framework.path配置里沒找到mr框架的路徑，檢查mapred-site.xml的該配置項確實配置有異常，在該配置項里增加mr框架路徑后通過（以下紅色為新增配置）。
轉載于:https://my.oschina.net/u/2987407/blog/781293
總結
以上是生活随笔為你收集整理的腾讯云大数据套件Hermes-MR索引插件使用总结的全部內容，希望文章能夠幫你解決所遇到的問題。
- 上一篇： 读《图解HTTP》总结--第九章
- 下一篇： WebMagic学习-解析json