當前位置：首頁 > 编程资源 > 编程问答 >内容正文

编程问答

hbase映射为hive表(转载＋整理＋自己验证)

發布時間：2023/12/31 编程问答 34 豆豆

生活随笔收集整理的這篇文章主要介紹了 hbase映射为hive表(转载＋整理＋自己验证) 小編覺得挺不錯的,現在分享給大家,幫大家做個參考.

本文是對[1][2]的整理以及在新版本上的驗證

-----------------------------------------------------------------------------------------

環境版本

大數據組件	版本
Hadoop	3.1.2
Zookeeper	3.6.0
Hbase	2.2.4
Hive	3.0.0

-------------------------------------------------

實驗前提:

啟動HDFS集群

啟動Zookeeper集群

啟動Hbase集群

啟動Hive的metastore和hiveserver2

-------------------------------------------------

Hive建表

hbase中插入數據

查詢hive

hbase->hive內表

CREATE TABLE member(
m_id string ,
address_contry string ,
address_province string ,
address_city string ,
info_age string ,
info_birthday string ,
info_company string
)STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' ???
WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key,address:contry,address:province,address:city,info:age,info:birthday,info:company") ???
TBLPROPERTIES("hbase.table.name" = "member");

put'member','scutshuxue','info:age','24'
put'member','scutshuxue','info:birthday','1987-06-17'
put'member','scutshuxue','info:company','alibaba'

put'member','scutshuxue','address:contry','china'
put'member','scutshuxue','address:province','zhejiang'?
put'member','scutshuxue','address:city','hangzhou'?
put'member','zhangsan','info:age','25'
put'member','zhangsan','info:birthday','1997-06-17'
put'member','zhangsan','info:company','alibaba'
put'member','zhangsan','address:contry','china'
put'member','zhangsan','address:province','sichuan'?
put'member','zhangsan','address:city','chendu'

select * from member;

hbase->hive外表

CREATE EXTERNAL TABLE student(
m_id string ,
address_contry string ,
address_province string ,
address_city string ,
info_age string ,
info_birthday string ,
info_company string
)STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' ? ?
WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key,address:contry,address:province,address:city,info:age,info:birthday,info:company") ? ?
TBLPROPERTIES("hbase.table.name" = "student");

create 'student','m_id','address','info'

put'student','scutshuxue','info:age','24'
put'student','scutshuxue','info:birthday','1987-06-17'
put'student','scutshuxue','info:company','alibaba'
put'student','scutshuxue','address:contry','china'
put'student','scutshuxue','address:province','zhejiang'
put'student','scutshuxue','address:city','hangzhou'
put'student','zhangsan','info:age','25'
put'student','zhangsan','info:birthday','1997-06-17'
put'student','zhangsan','info:company','alibaba'
put'student','zhangsan','address:contry','china'
put'student','zhangsan','address:province','sichuan'
put'student','zhangsan','address:city','chendu'

select * from student;

調試tips:

Hive用beeline建表的時候可能會卡住，這個時候可以退出beeline>引擎，采用老的hive>引擎

注意:

上述表格中的任何一條命令，如果超過8秒(我的是5秒，普通的4核8線程電腦)，一般是由于故障卡住了，不要盲目等待。

如果hive的metastore清空，想要重新操作，那么上述過程中，必須先清除hbase中的數據，清除過程以[3]為準

＃－－－－－－－－－－－－－－－－－－列映射－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－

上半部分是全部數據從hbase映射到hive，

如果只想映射一部分數據呢？列映射來了！

列映射表格關系(根據不同映射需求，hbase->hive)

Hbase表Hive表映射方法

index1	hbase_table1	列映射（單列）
index1	hbase_table2	多列映射
index1	hbase_table3	列簇映射

index1表結構以及實驗數據

Hbase表名:index1
Row Key	列簇(column family):cf0
Row Key	cf0:indexId	cf0:muMac
11	11	?
13	11	?
9696	100	?
9901	?	uzzu
990111	11	?
990_11aa_aza	11	?

hbase操作

建立index1

create "index1","cf0"

傳入數據操作

create "index1","cf0"
put 'index1',11,'cf0:indexId',11
put 'index1',13,'cf0:indexId',11? ? ? ? ? ? ? ? ? ? ? ? ? ?

scan 'index1'? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?

put 'index1',9696,'cf0:indexId',100? ? ? ? ? ? ? ? ? ??

put 'index1',9901,'cf0:muMac','uzzu'
put 'index1',990111,'cf0:indexId',11
put 'index1','990_11aa_aza','cf0:indexId',11

?	Hive操作	查詢語句
單列映射	CREATE EXTERNAL TABLE hbase_table_1(key string, value string)? STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' WITH SERDEPROPERTIES("hbase.columns.mapping" = "cf0:indexId")? TBLPROPERTIES("hbase.table.name" = "index1", "hbase.mapred.output.outputtable" = "index1"); ?	select * from hbase_table_1;
多列映射	CREATE EXTERNAL TABLE hbase_table_2(key int, indexId string, muMac string)? STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'? WITH SERDEPROPERTIES ("hbase.columns.mapping" = "cf0:indexId,cf0:muMac")? TBLPROPERTIES("hbase.table.name" = "index1", "hbase.mapred.output.outputtable" = "index1");	select * from hbase_table_2;
列簇映射	CREATE EXTERNAL TABLE hbase_table_3(value map<string,string>,row_key string)? STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' WITH SERDEPROPERTIES ("hbase.columns.mapping" = "cf0:,:key")? TBLPROPERTIES("hbase.table.name" = "index1", "hbase.mapred.output.outputtable" = "index1");	select * from hbase_table_3;

hbase_table_1結果:

+--------------------+----------------------+ | hbase_table_1.key | hbase_table_1.value | +--------------------+----------------------+ | 11 | 11 | | 13 | 11 | | 9696 | 100 | | 990111 | 11 | | 990_11aa_aza | 11 | +--------------------+----------------------+

hbase_table_2結果:

+--------------------+------------------------+----------------------+ | hbase_table_2.key | hbase_table_2.indexid | hbase_table_2.mumac | +--------------------+------------------------+----------------------+ | 11 | 11 | NULL | | 13 | 11 | NULL | | 9696 | 100 | NULL | | 9901 | NULL | uzzu | | 990111 | 11 | NULL | | NULL | 11 | NULL | +--------------------+------------------------+----------------------+

hbase_table_3結果:

+----------------------+------------------------+ | hbase_table_3.value | hbase_table_3.row_key | +----------------------+------------------------+ | {"indexId":"11"} | 11 | | {"indexId":"11"} | 13 | | {"indexId":"100"} | 9696 | | {"muMac":"uzzu"} | 9901 | | {"indexId":"11"} | 990111 | | {"indexId":"11"} | 990_11aa_aza | +----------------------+------------------------+

＃－－－－－－－－－－－－－－－－－－列映射－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－

Reference:

[1]hive：hbase表映射到hive中

[2]Hbase表映射Hive表三種方法

[3]HBase表創建、刪除、清空

創作挑戰賽新人創作獎勵來咯，堅持創作打卡瓜分現金大獎