概述

桶表是對(duì)數(shù)據(jù)進(jìn)行哈希取值，然后放到不同文件中存儲(chǔ)。

數(shù)據(jù)加載到桶表時(shí)，會(huì)對(duì)字段取hash值，然后與桶的數(shù)量取模。把數(shù)據(jù)放到對(duì)應(yīng)的文件中。
物理上，每個(gè)桶就是表(或分區(qū)）目錄里的一個(gè)文件，一個(gè)作業(yè)產(chǎn)生的桶(輸出文件)和reduce任務(wù)個(gè)數(shù)相同。

作用

桶表專門用于抽樣查詢，是很專業(yè)性的，不是日常用來(lái)存儲(chǔ)數(shù)據(jù)的表，需要抽樣查詢時(shí)，才創(chuàng)建和使用桶表。

實(shí)驗(yàn)

創(chuàng)建

[22:39:03]hive (zmgdb)> create table bucket_t1(id string)
[22:39:26] ? ? ? ? ? ?> clustered by(id) into 6 buckets;
[22:39:27]OK
[22:39:27]Time taken: 0.546 seconds
clustered by：以哪個(gè)字段分桶。對(duì)id進(jìn)行哈希取值，隨機(jī) ?地放到4個(gè)桶里。

-----------------------------

準(zhǔn)備數(shù)據(jù)

[root@hello110 data]# vi bucket_test
1
2
3
4
5
6

.............

.........

導(dǎo)入數(shù)據(jù)

正確的導(dǎo)入方式：從日常保存數(shù)據(jù)的表insert

[21:27:45]hive (zmgdb)> create table t2(id string);
[21:27:45]OK
[21:27:45]Time taken: 0.073 seconds
[21:28:24]hive (zmgdb)> load data local inpath '/data/bucket_test' into table t2;
[21:28:24]Loading data to table zmgdb.t2
[21:28:25]OK

從日常表導(dǎo)入

[22:39:47]hive (zmgdb)> insert overwrite table bucket_t1 select id from t2;

hive會(huì)啟動(dòng)mapreduce
[22:39:48]WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
[22:39:48]Query ID = hadoop_20160922063946_34bf30c4-3f23-43e9-ad8f-edd5ee214948
[22:39:48]Total jobs = 1
[22:39:48]Launching Job 1 out of 1
[22:39:48]Number of reduce tasks determined at compile time: 6
[22:39:48]In order to change the average load for a reducer (in bytes):
[22:39:48] ?set hive.exec.reducers.bytes.per.reducer=<number>
[22:39:48]In order to limit the maximum number of reducers:
[22:39:48] ?set hive.exec.reducers.max=<number>
[22:39:48]In order to set a constant number of reducers:
[22:39:48] ?set mapreduce.job.reduces=<number>
[22:39:51]Starting Job = job_1474497386931_0001, Tracking URL = http://hello110:8088/proxy/application_1474497386931_0001/
[22:39:51]Kill Command = /home/hadoop/app/hadoop-2.7.2/bin/hadoop job ?-kill job_1474497386931_0001
[22:39:59]Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 6
[22:39:59]2016-09-22 06:39:59,419 Stage-1 map = 0%, ?reduce = 0%
[22:40:06]2016-09-22 06:40:05,828 Stage-1 map = 100%, ?reduce = 0%, Cumulative CPU 1.63 sec
[22:40:12]2016-09-22 06:40:12,347 Stage-1 map = 100%, ?reduce = 17%, Cumulative CPU 3.48 sec
[22:40:16]2016-09-22 06:40:15,739 Stage-1 map = 100%, ?reduce = 33%, Cumulative CPU 5.4 sec
[22:40:17]2016-09-22 06:40:16,807 Stage-1 map = 100%, ?reduce = 50%, Cumulative CPU 7.52 sec
[22:40:19]2016-09-22 06:40:18,929 Stage-1 map = 100%, ?reduce = 83%, Cumulative CPU 11.35 sec
[22:40:20]2016-09-22 06:40:19,991 Stage-1 map = 100%, ?reduce = 100%, Cumulative CPU 13.19 sec
[22:40:21]MapReduce Total cumulative CPU time: 13 seconds 190 msec
[22:40:21]Ended Job = job_1474497386931_0001
[22:40:21]Loading data to table zmgdb.bucket_t1
[22:40:22]MapReduce Jobs Launched:?
[22:40:22]Stage-Stage-1: Map: 1 ?Reduce: 6 ? Cumulative CPU: 13.19 sec ? HDFS Read: 25355 HDFS Write: 1434 SUCCESS
[22:40:22]Total MapReduce CPU Time Spent: 13 seconds 190 msec
[22:40:22]OK
[22:40:22]id
[22:40:22]Time taken: 34.91 seconds

錯(cuò)誤的導(dǎo)入方式：從文件load data?

hive (zmgdb)> create table bucket_t2 like bucket_t1;
OK
Time taken: 0.707 seconds

hive (zmgdb)> load data local inpath '/data/bucket_test' into table bucket_t2;
Loading data to table zmgdb.bucket_t2
OK
Time taken: 1.485 seconds

沒(méi)有啟動(dòng)mapreduce對(duì)數(shù)據(jù)進(jìn)行哈希取值，只是簡(jiǎn)單的原樣導(dǎo)入，沒(méi)有起到抽樣查詢的目的。通過(guò)select * from 比較會(huì)發(fā)現(xiàn)bucket_t1的數(shù)據(jù)和bucket_t2的數(shù)據(jù)順序是不同的，bucket_t2的表順序與原數(shù)據(jù)文件順序一致，沒(méi)有做過(guò)哈希取值。

查詢

select * from bucket_table tablesample(bucket x out of y on column);
tablesample是抽樣語(yǔ)句
語(yǔ)法解析：TABLESAMPLE(BUCKET x OUT OF y on 字段)
y必須是table總bucket數(shù)的倍數(shù)或者因子。
hive根據(jù)y的大小，決定抽樣的比例。
例如，table總共分了64份，當(dāng)y=32時(shí)，抽取(64/32=)2個(gè)bucket的數(shù)據(jù)，當(dāng)y=128時(shí)，抽取(64/128=)1/2個(gè)bucket的數(shù)據(jù)。x表示從哪個(gè)bucket開(kāi)始抽取。
例如，table總bucket數(shù)為32，tablesample(bucket 3 out of 16)，表示總共抽取（32/16=）2個(gè)bucket的數(shù)據(jù)，分別為第3個(gè)bucket和第（3+16=）19個(gè)bucket的數(shù)據(jù)。如果是y=64，則抽取半個(gè)第3個(gè)桶的值。

[22:44:31]hive (zmgdb)> select * from bucket_t1 tablesample (bucket 1 out of 6 on id);
[22:44:31]OK
[22:44:31]bucket_t1.id
[22:44:31]6
[22:44:31]iu
[22:44:31]0
[22:44:31]6
[22:44:31]hj
[22:44:31]6
[22:44:31]6
[22:44:31]51
[22:44:31]
[22:44:31]
[22:44:31]r
[22:44:31]99
[22:44:31]0
[22:44:31]57
[22:44:31]loo
[22:44:31]r
[22:44:31]r
[22:44:31]r
[22:44:31]60
[22:44:31]66
[22:44:31]75
[22:44:31]6
[22:44:31]84
[22:44:31]x
[22:44:31]24
[22:44:31]93
[22:44:31]99
[22:44:31]105
[22:44:31]f
[22:44:31]r
[22:44:31]114
[22:44:31]0
[22:44:31]123
[22:44:31]129
[22:44:31]132
[22:44:31]x
[22:44:31]138
[22:44:31]141
[22:44:31]147
[22:44:31]33
[22:44:31]150
[22:44:31]156
[22:44:31]r
[22:44:31]f
[22:44:31]39
[22:44:31]15
[22:44:31]r
[22:44:31]ddd
[22:44:31]
[22:44:31]06
[22:44:31]hj
[22:44:31]f
[22:44:31]l
[22:44:31]f
[22:44:31]f
[22:44:31]f
[22:44:31]f
[22:44:31]42
[22:44:31]f
[22:44:31]r
[22:44:31]r
[22:44:31]f
[22:44:31]f
[22:44:31]r
[22:44:31]48
[22:44:31]6
[22:44:31]Time taken: 0.142 seconds, Fetched:66 row(s)

[22:44:43]hive (zmgdb)> select * from bucket_t1 tablesample (bucket 1 out of 60 on id);
[22:44:43]OK
[22:44:43]bucket_t1.id
[22:44:43]
[22:44:43]
[22:44:43]loo
[22:44:43]x
[22:44:43]114
[22:44:43]132
[22:44:43]x
[22:44:43]150
[22:44:43]ddd
[22:44:43]
[22:44:43]Time taken: 0.064 seconds, Fetched: 10 row(s)

創(chuàng)作挑戰(zhàn)賽新人創(chuàng)作獎(jiǎng)勵(lì)來(lái)咯，堅(jiān)持創(chuàng)作打卡瓜分現(xiàn)金大獎(jiǎng)

總結(jié)

以上是生活随笔為你收集整理的Hive的数据模型—桶表的全部?jī)?nèi)容，希望文章能夠幫你解決所遇到的問(wèn)題。

如果覺(jué)得生活随笔網(wǎng)站內(nèi)容還不錯(cuò)，歡迎將生活随笔推薦給好友。