?

場景：現(xiàn)有一批之前導(dǎo)出的數(shù)據(jù)，發(fā)現(xiàn)2.0版本hbck工具更新，無法直接導(dǎo)入，跨機(jī)房使用export/import方式需要重新外網(wǎng)傳輸數(shù)據(jù)比較耗時，現(xiàn)搭建臨時hbase版本1.2，在同機(jī)房進(jìn)行export/import方式

數(shù)據(jù)導(dǎo)入hbase1.2版本

? ? ? ? ? ?CDH默認(rèn)hbase用戶是不可登錄，修改權(quán)限，使其可登錄(完成操作后記得改回來)

##這里沒有全部顯示 [root@test ~]# vim /etc/passwd cloudera-scm:x:997:995:Cloudera Manager:/var/lib/cloudera-scm-server:/sbin/nologin mysql:x:27:27:MariaDB Server:/var/lib/mysql:/sbin/nologin flume:x:996:993:Flume:/var/lib/flume-ng:/bin/false hdfs:x:995:992:Hadoop HDFS:/var/lib/hadoop-hdfs:/bin/bash solr:x:994:991:Solr:/var/lib/solr:/sbin/nologin zookeeper:x:993:990:ZooKeeper:/var/lib/zookeeper:/bin/false llama:x:992:989:Llama:/var/lib/llama:/bin/bash httpfs:x:991:988:Hadoop HTTPFS:/var/lib/hadoop-httpfs:/bin/bash mapred:x:990:987:Hadoop MapReduce:/var/lib/hadoop-mapreduce:/bin/bash sqoop:x:989:986:Sqoop:/var/lib/sqoop:/bin/false yarn:x:988:985:Hadoop Yarn:/var/lib/hadoop-yarn:/bin/bash kms:x:987:984:Hadoop KMS:/var/lib/hadoop-kms:/bin/bash hive:x:986:983:Hive:/var/lib/hive:/bin/false sqoop2:x:985:982:Sqoop 2 User:/var/lib/sqoop2:/sbin/nologin oozie:x:984:981:Oozie User:/var/lib/oozie:/bin/false kudu:x:983:980:Kudu:/var/lib/kudu:/sbin/nologin hbase:x:982:979:HBase:/var/lib/hbase:/bin/false sentry:x:981:978:Sentry:/var/lib/sentry:/sbin/nologin impala:x:980:977:Impala:/var/lib/impala:/bin/bash spark:x:979:976:Spark:/var/lib/spark:/sbin/nologin hue:x:978:975:Hue:/usr/lib/hue:/bin/false ntp:x:38:38::/etc/ntp:/sbin/nologin

修改

hbase:x:982:979:HBase:/var/lib/hbase:/bin/false

為

hbase:x:982:979:HBase:/var/lib/hbase:/bin/bash

切換至Hbase用戶，將數(shù)據(jù)導(dǎo)入hbase1.2，使用一下命令將linux 上的數(shù)據(jù)遷移至hdfs相對應(yīng)目錄中

hdfs dfs -copyFromLocal /linux/dic/table1 /hdfs/hbase/table1

導(dǎo)入成功后修復(fù)元數(shù)據(jù)表

hbase hbck -fixMeta -fixAssignments

2.將Hbase1.2數(shù)據(jù)Export 至 Hbase2.0數(shù)據(jù)庫中

? ??

hbase org.apache.hadoop.hbase.mapreduce.Export 需要導(dǎo)出的Hbase表名 需要輸出到的位置(可以使其他集群的hdfs路徑，也可以是本地linux系統(tǒng)的路徑)

?

異常：權(quán)限不足

[hbase@test ~]$ hbase org.apache.hadoop.hbase.mapreduce.Export HbasetableName hdfs://xxxxx:8020/hbase/HbasetableName Java HotSpot(TM) 64-Bit Server VM warning: Using incremental CMS is deprecated and will likely be removed in a future release 18/09/10 21:39:53 INFO mapreduce.Export: versions=1, starttime=0, endtime=9223372036854775807, keepDeletedCells=false 18/09/10 21:39:54 INFO client.RMProxy: Connecting to ResourceManager at fwqml006.zh/10.248.161.16:8032 18/09/10 21:39:54 WARN security.UserGroupInformation: PriviledgedActionException as:hbase (auth:SIMPLE) cause:org.apache.hadoop.security.AccessControlException: Permission denied: user=hbase, access=WRITE, inode="/user":hdfs:supergroup:drwxr-xr-xat org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkFsPermission(DefaultAuthorizationProvider.java:279)at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.check(DefaultAuthorizationProvider.java:260)at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.check(DefaultAuthorizationProvider.java:240)at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkPermission(DefaultAuthorizationProvider.java:162)at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:152)at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:3770)at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:3753)at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkAncestorAccess(FSDirectory.java:3735)at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkAncestorAccess(FSNamesystem.java:6723)at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInternal(FSNamesystem.java:4493)at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInt(FSNamesystem.java:4463)at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:4436)at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:876)at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.mkdirs(AuthorizationProviderProxyClientProtocol.java:326)at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:640)at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2226)at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2222)at java.security.AccessController.doPrivileged(Native Method)at javax.security.auth.Subject.doAs(Subject.java:422)at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1917)at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2220)Exception in thread "main" org.apache.hadoop.security.AccessControlException: Permission denied: user=hbase, access=WRITE, inode="/user":hdfs:supergroup:drwxr-xr-xat org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkFsPermission(DefaultAuthorizationProvider.java:279)at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.check(DefaultAuthorizationProvider.java:260)at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.check(DefaultAuthorizationProvider.java:240)at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkPermission(DefaultAuthorizationProvider.java:162)at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:152)at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:3770)at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:3753)at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkAncestorAccess(FSDirectory.java:3735)at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkAncestorAccess(FSNamesystem.java:6723)at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInternal(FSNamesystem.java:4493)at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInt(FSNamesystem.java:4463)at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:4436)at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:876)at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.mkdirs(AuthorizationProviderProxyClientProtocol.java:326)at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:640)at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2226)at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2222)at java.security.AccessController.doPrivileged(Native Method)at javax.security.auth.Subject.doAs(Subject.java:422)at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1917)at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2220)at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)at java.lang.reflect.Constructor.newInstance(Constructor.java:423)at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73)at org.apache.hadoop.hdfs.DFSClient.primitiveMkdir(DFSClient.java:3120)at org.apache.hadoop.hdfs.DFSClient.mkdirs(DFSClient.java:3085)at org.apache.hadoop.hdfs.DistributedFileSystem$19.doCall(DistributedFileSystem.java:1004)at org.apache.hadoop.hdfs.DistributedFileSystem$19.doCall(DistributedFileSystem.java:1000)at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)at org.apache.hadoop.hdfs.DistributedFileSystem.mkdirsInternal(DistributedFileSystem.java:1000)at org.apache.hadoop.hdfs.DistributedFileSystem.mkdirs(DistributedFileSystem.java:992)at org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:133)at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:148)at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1307)at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1304)at java.security.AccessController.doPrivileged(Native Method)at javax.security.auth.Subject.doAs(Subject.java:422)at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1917)at org.apache.hadoop.mapreduce.Job.submit(Job.java:1304)at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1325)at org.apache.hadoop.hbase.mapreduce.Export.main(Export.java:188) Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException): Permission denied: user=hbase, access=WRITE, inode="/user":hdfs:supergroup:drwxr-xr-xat org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkFsPermission(DefaultAuthorizationProvider.java:279)at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.check(DefaultAuthorizationProvider.java:260)at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.check(DefaultAuthorizationProvider.java:240)at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkPermission(DefaultAuthorizationProvider.java:162)at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:152)at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:3770)at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:3753)at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkAncestorAccess(FSDirectory.java:3735)at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkAncestorAccess(FSNamesystem.java:6723)at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInternal(FSNamesystem.java:4493)at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInt(FSNamesystem.java:4463)at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:4436)at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:876)at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.mkdirs(AuthorizationProviderProxyClientProtocol.java:326)at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:640)at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2226)at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2222)at java.security.AccessController.doPrivileged(Native Method)at javax.security.auth.Subject.doAs(Subject.java:422)at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1917)at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2220)at org.apache.hadoop.ipc.Client.call(Client.java:1504)at org.apache.hadoop.ipc.Client.call(Client.java:1441)at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230)at com.sun.proxy.$Proxy11.mkdirs(Unknown Source)at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.mkdirs(ClientNamenodeProtocolTranslatorPB.java:573)at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)at java.lang.reflect.Method.invoke(Method.java:498)at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:260)at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104)at com.sun.proxy.$Proxy12.mkdirs(Unknown Source)at org.apache.hadoop.hdfs.DFSClient.primitiveMkdir(DFSClient.java:3118)... 16 more

提示hdfs中/user目錄hbase用戶權(quán)限不足，修改/user目錄訪問權(quán)限，這里為了方便使用

-chmod -R 777，記得數(shù)據(jù)導(dǎo)出完成后修改為之前權(quán)限。

又報錯jdk環(huán)境：

[hdfs@test ~]$ hdfs dfs -chmod -R 777 /user ERROR: JAVA_HOME /home/xxx/tools/jdk1.8.0_101 does not exist. 在~/bashrc修改環(huán)境變量

執(zhí)行導(dǎo)出成功。

hbase org.apache.hadoop.hbase.mapreduce.Export HbasetableName hdfs://xxxxx:8020/hbase/HbasetableName

注：導(dǎo)入到本地磁盤使用? file://

hbase org.apache.hadoop.hbase.mapreduce.Export HbasetableName file:///home/zsh/HbasetableName

在目標(biāo)集群中發(fā)現(xiàn)數(shù)據(jù)已傳輸過去

[hbase@fwqzx002 ~]$ hdfs dfs -ls /hbase/xxx/xxx Found 12 items -rw-r--r-- 1 hbase hbase 0 2018-09-10 23:04 /hbase/tt_user/offline_user/_SUCCESS -rw-r--r-- 1 hbase hbase 47065003510 2018-09-10 23:04 /hbase/xxx/xxx/part-m-00003 -rw-r--r-- 1 hbase hbase 23541633987 2018-09-10 22:24 /hbase/xxx/xxx/part-m-00004 -rw-r--r-- 1 hbase hbase 23532345447 2018-09-10 22:40 /hbase/xxx/xxx/part-m-00005 -rw-r--r-- 1 hbase hbase 23551359671 2018-09-10 22:39 /hbase/xxx/xxx/part-m-00006 -rw-r--r-- 1 hbase hbase 23522350569 2018-09-10 22:25 /hbase/xxx/xxx/part-m-00007 -rw-r--r-- 1 hbase hbase 23544202929 2018-09-10 22:48 /hbase/xxx/xxx/part-m-00008 -rw-r--r-- 1 hbase hbase 23529537743 2018-09-10 22:40 /hbase/xxx/xxx/part-m-00009 -rw-r--r-- 1 hbase hbase 11749139280 2018-09-10 22:36 /hbase/xxx/xxx/part-m-00010 -rw-r--r-- 1 hbase hbase 11754855832 2018-09-10 22:36 /hbase/xxx/xxx/part-m-00011 -rw-r--r-- 1 hbase hbase 11775381448 2018-09-10 22:27 /hbase/xxx/xxx/part-m-00012 -rw-r--r-- 1 hbase hbase 11767607324 2018-09-10 22:35 /hbase/xxx/xxx/part-m-00013

3.數(shù)據(jù)導(dǎo)入Hbase2.0版本

hbase org.apache.hadoop.hbase.mapreduce.Import -Dmapred.job.queue.name=etl tableName /hbase/test4

少量數(shù)據(jù)正常導(dǎo)入，大量數(shù)據(jù)時Hbase寫入過快處理不過來，主要原因是region分裂時導(dǎo)致memstore數(shù)據(jù)量過大，提示異常RetriesExhaustedWithDetailsException

解決方法：https://blog.csdn.net/zhangshenghang/article/details/82621101

再次執(zhí)行導(dǎo)入成功。

?

創(chuàng)作挑戰(zhàn)賽新人創(chuàng)作獎勵來咯，堅持創(chuàng)作打卡瓜分現(xiàn)金大獎

總結(jié)

以上是生活随笔為你收集整理的Hbase1.2数据导入2.0的全部內(nèi)容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網(wǎng)站內(nèi)容還不錯，歡迎將生活随笔推薦給好友。