Hive入门教程
Hive 安裝
相比起很多教程先介紹概念，我喜歡先動手裝上，然后用例子來介紹概念。我們先來安裝一下Hive
先確認是否已經安裝了對應的yum源，如果沒有照這個教程里面寫的安裝cdh的yum源http://blog.csdn.net/nsrainbow/article/details/36629339
?
Hive是什么
Hive 提供了一個讓大家可以使用sql去查詢數據的途徑。但是最好不要拿Hive進行實時的查詢。因為Hive的實現原理是把sql語句轉化為多個Map Reduce任務所以Hive非常慢，官方文檔說Hive 適用于高延時性的場景而且很費資源。
舉個簡單的例子，可以像這樣去查詢
hive> select * from h_employee; OK 1 1 peter 2 2 paul Time taken: 9.289 seconds, Fetched: 2 row(s)?
?這個h_employee不一定是一個數據庫表
?
metastore
Hive 中建立的表都叫metastore表。這些表并不真實的存儲數據，而是定義真實數據跟hive之間的映射，就像傳統數據庫中表的meta信息，所以叫做metastore。實際存儲的時候可以定義的存儲模式有四種：
?
內部表（默認）分區表桶表外部表 舉個例子，這是一個簡歷內部表的語句
CREATE TABLE worker(id INT, name STRING) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\054';?
?
?這個語句的意思是建立一個worker的內部表，內部表是默認的類型，所以不用寫存儲的模式。并且使用逗號作為分隔符存儲?
建表語句支持的類型
基本數據類型
tinyint / smalint / int /bigint
float / double
boolean
string
復雜數據類型
Array/Map/Struct
沒有date /datetime
建完的表存在哪里呢？
在 /user/hive/warehouse 里面，可以通過hdfs來查看建完的表位置
$ hdfs dfs -ls /user/hive/warehouse Found 11 items drwxrwxrwt - root supergroup 0 2014-12-02 14:42 /user/hive/warehouse/h_employee drwxrwxrwt - root supergroup 0 2014-12-02 14:42 /user/hive/warehouse/h_employee2 drwxrwxrwt - wlsuser supergroup 0 2014-12-04 17:21 /user/hive/warehouse/h_employee_export drwxrwxrwt - root supergroup 0 2014-08-18 09:20 /user/hive/warehouse/h_http_access_logs drwxrwxrwt - root supergroup 0 2014-06-30 10:15 /user/hive/warehouse/hbase_apache_access_log drwxrwxrwt - username supergroup 0 2014-06-27 17:48 /user/hive/warehouse/hbase_table_1 drwxrwxrwt - username supergroup 0 2014-06-30 09:21 /user/hive/warehouse/hbase_table_2 drwxrwxrwt - username supergroup 0 2014-06-30 09:43 /user/hive/warehouse/hive_apache_accesslog drwxrwxrwt - root supergroup 0 2014-12-02 15:12 /user/hive/warehouse/hive_employee?
?
?一個文件夾對應一個metastore表
Hive 各種類型表使用
CREATE TABLE workers( id INT, name STRING) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\054';?
?通過這樣的語句就建立了一個內部表叫 workers，并且分隔符是逗號， \054 是ASCII 碼?
我們可以通過 show tables; 來看看有多少表，其實hive的很多語句是模仿mysql的，當你們不知道語句的時候，把mysql的語句拿來基本可以用。除了limit比較怪，這個后面會說?
建立完后，我們試著插入幾條數據。這邊要告訴大家Hive不支持單句插入的語句，必須批量，所以不要指望能用insert into workers values (1,'jack') 這樣的語句插入數據。hive支持的插入數據的方式有兩種： 從文件讀取數據從別的表讀出數據插入(insert from select) 這里我采用從文件讀數據進來。先建立一個叫 worker.csv的文件
$ cat workers.csv 1,jack 2,terry 3,michael用LOAD DATA 導入到Hive的表中
hive> LOAD DATA LOCAL INPATH '/home/alex/workers.csv' INTO TABLE workers; Copying data from file:/home/alex/workers.csv Copying file: file:/home/alex/workers.csv Loading data to table default.workers Table default.workers stats: [num_partitions: 0, num_files: 1, num_rows: 0, total_size: 25, raw_data_size: 0] OK Time taken: 0.655 seconds注意 不要少了那個 LOCAL ， LOAD DATA LOCAL INPATH 跟 LOAD DATA INPATH 的區別是一個是從你本地磁盤上找源文件，一個是從hdfs上找文件如果加上OVERWRITE可以再導入之前先清空表，比如 LOAD DATA LOCAL INPATH '/home/alex/workers.csv' OVERWRITE INTO TABLE workers; 查詢一下數據
hive> select * from workers; OK 1 jack 2 terry 3 michael Time taken: 0.177 seconds, Fetched: 3 row(s)我們去看下導入后在hive內部表是怎么存的
# hdfs dfs -ls /user/hive/warehouse/workers/ Found 1 items -rwxrwxrwt 2 root supergroup 25 2014-12-08 15:23 /user/hive/warehouse/workers/workers.csv原來就是原封不動的把文件拷貝進去啊！就是這么土！ 我們可以試驗再放一個文件 workers2.txt （我故意把擴展名換一個，其實hive是不看擴展名的）
# cat workers2.txt 4,peter 5,kate 6,ted導入
hive> LOAD DATA LOCAL INPATH '/home/alex/workers2.txt' INTO TABLE workers; Copying data from file:/home/alex/workers2.txt Copying file: file:/home/alex/workers2.txt Loading data to table default.workers Table default.workers stats: [num_partitions: 0, num_files: 2, num_rows: 0, total_size: 46, raw_data_size: 0] OK Time taken: 0.79 seconds去看下文件的存儲結構
# hdfs dfs -ls /user/hive/warehouse/workers/ Found 2 items -rwxrwxrwt 2 root supergroup 25 2014-12-08 15:23 /user/hive/warehouse/workers/workers.csv -rwxrwxrwt 2 root supergroup 21 2014-12-08 15:29 /user/hive/warehouse/workers/workers2.txt多出來一個workers2.txt 再用sql查詢下
hive> select * from workers; OK 1 jack 2 terry 3 michael 4 peter 5 kate 6 ted Time taken: 0.144 seconds, Fetched: 6 row(s)分區表
分區表是用來加速查詢的，比如你的數據非常多，但是你的應用場景是基于這些數據做日報表，那你就可以根據日進行分區，當你要做2014-05-05的報表的時候只需要加載2014-05-05這一天的數據就行了。我們來創建一個分區表來看下?
create table partition_employee(id int, name string) partitioned by(daytime string) row format delimited fields TERMINATED BY '\054';可以看到分區的屬性，并不是任何一個列 我們先建立2個測試數據文件，分別對應兩天的數據
# cat 2014-05-05 22,kitty 33,lily # cat 2014-05-06 14,sami 45,micky導入到分區表里面
hive> LOAD DATA LOCAL INPATH '/home/alex/2014-05-05' INTO TABLE partition_employee partition(daytime='2014-05-05'); Copying data from file:/home/alex/2014-05-05 Copying file: file:/home/alex/2014-05-05 Loading data to table default.partition_employee partition (daytime=2014-05-05) Partition default.partition_employee{daytime=2014-05-05} stats: [num_files: 1, num_rows: 0, total_size: 21, raw_data_size: 0] Table default.partition_employee stats: [num_partitions: 1, num_files: 1, num_rows: 0, total_size: 21, raw_data_size: 0] OK Time taken: 1.154 seconds hive> LOAD DATA LOCAL INPATH '/home/alex/2014-05-06' INTO TABLE partition_employee partition(daytime='2014-05-06'); Copying data from file:/home/alex/2014-05-06 Copying file: file:/home/alex/2014-05-06 Loading data to table default.partition_employee partition (daytime=2014-05-06) Partition default.partition_employee{daytime=2014-05-06} stats: [num_files: 1, num_rows: 0, total_size: 21, raw_data_size: 0] Table default.partition_employee stats: [num_partitions: 2, num_files: 2, num_rows: 0, total_size: 42, raw_data_size: 0] OK Time taken: 0.763 seconds導入的時候通過 partition 來指定分區。?
查詢的時候通過指定分區來查詢
我的查詢語句并沒有什么特別的語法，hive 會自動判斷你的where語句中是否包含分區的字段。而且可以使用大于小于等運算符
hive> select * from partition_employee where daytime>='2014-05-05'; OK 22 kitty 2014-05-05 33 lily 2014-05-05 14 sami 2014-05-06 45 mick' 2014-05-06 Time taken: 0.273 seconds, Fetched: 4 row(s)我們去看看存儲的結構
# hdfs dfs -ls /user/hive/warehouse/partition_employee Found 2 items drwxrwxrwt - root supergroup 0 2014-12-08 15:57 /user/hive/warehouse/partition_employee/daytime=2014-05-05 drwxrwxrwt - root supergroup 0 2014-12-08 15:57 /user/hive/warehouse/partition_employee/daytime=2014-05-06我們試試二維的分區表
create table p_student(id int, name string) partitioned by(daytime string,country string) row format delimited fields TERMINATED BY '\054';查入一些數據
# cat 2014-09-09-CN 1,tammy 2,eric # cat 2014-09-10-CN 3,paul 4,jolly # cat 2014-09-10-EN 44,ivan 66,billy導入hive
hive> LOAD DATA LOCAL INPATH '/home/alex/2014-09-09-CN' INTO TABLE p_student partition(daytime='2014-09-09',country='CN'); Copying data from file:/home/alex/2014-09-09-CN Copying file: file:/home/alex/2014-09-09-CN Loading data to table default.p_student partition (daytime=2014-09-09, country=CN) Partition default.p_student{daytime=2014-09-09, country=CN} stats: [num_files: 1, num_rows: 0, total_size: 19, raw_data_size: 0] Table default.p_student stats: [num_partitions: 1, num_files: 1, num_rows: 0, total_size: 19, raw_data_size: 0] OK Time taken: 0.736 seconds hive> LOAD DATA LOCAL INPATH '/home/alex/2014-09-10-CN' INTO TABLE p_student partition(daytime='2014-09-10',country='CN'); Copying data from file:/home/alex/2014-09-10-CN Copying file: file:/home/alex/2014-09-10-CN Loading data to table default.p_student partition (daytime=2014-09-10, country=CN) Partition default.p_student{daytime=2014-09-10, country=CN} stats: [num_files: 1, num_rows: 0, total_size: 19, raw_data_size: 0] Table default.p_student stats: [num_partitions: 2, num_files: 2, num_rows: 0, total_size: 38, raw_data_size: 0] OK Time taken: 0.691 seconds hive> LOAD DATA LOCAL INPATH '/home/alex/2014-09-10-EN' INTO TABLE p_student partition(daytime='2014-09-10',country='EN'); Copying data from file:/home/alex/2014-09-10-EN Copying file: file:/home/alex/2014-09-10-EN Loading data to table default.p_student partition (daytime=2014-09-10, country=EN) Partition default.p_student{daytime=2014-09-10, country=EN} stats: [num_files: 1, num_rows: 0, total_size: 21, raw_data_size: 0] Table default.p_student stats: [num_partitions: 3, num_files: 3, num_rows: 0, total_size: 59, raw_data_size: 0] OK Time taken: 0.622 seconds看看存儲結構
# hdfs dfs -ls /user/hive/warehouse/p_student Found 2 items drwxr-xr-x - root supergroup 0 2014-12-08 16:10 /user/hive/warehouse/p_student/daytime=2014-09-09 drwxr-xr-x - root supergroup 0 2014-12-08 16:10 /user/hive/warehouse/p_student/daytime=2014-09-10 # hdfs dfs -ls /user/hive/warehouse/p_student/daytime=2014-09-09 Found 1 items drwxr-xr-x - root supergroup 0 2014-12-08 16:10 /user/hive/warehouse/p_student/daytime=2014-09-09/country=CN查詢一下數據
hive> select * from p_student; OK 1 tammy 2014-09-09 CN 2 eric 2014-09-09 CN 3 paul 2014-09-10 CN 4 jolly 2014-09-10 CN 44 ivan 2014-09-10 EN 66 billy 2014-09-10 EN Time taken: 0.228 seconds, Fetched: 6 row(s) hive> select * from p_student where daytime='2014-09-10' and country='EN'; OK 44 ivan 2014-09-10 EN 66 billy 2014-09-10 EN Time taken: 0.224 seconds, Fetched: 2 row(s)?
?
?
桶表
?桶表是根據某個字段的hash值，來將數據扔到不同的“桶”里面。外國人有個習慣，就是分類東西的時候擺幾個桶，上面貼不同的標簽，所以他們取名的時候把這種表形象的取名為桶表。桶表表專門用于采樣分析?
下面這個例子是官網教程直接拷貝下來的，因為分區表跟桶表是可以同時使用的，所以這個例子中同時使用了分區跟桶兩種特性
?
?意思是根據userid來進行計算hash值，用viewTIme來排序存儲 做數據跟導入的過程我就不在贅述了，這是導入后的數據
hive> select * from b_student; OK 1 tammy 2014-09-09 CN 2 eric 2014-09-09 CN 3 paul 2014-09-10 CN 4 jolly 2014-09-10 CN 34 allen 2014-09-11 EN Time taken: 0.727 seconds, Fetched: 5 row(s)?
?從4個桶中采樣抽取一個桶的數據
hive> select * from b_student tablesample(bucket 1 out of 4 on id); Total MapReduce jobs = 1 Launching Job 1 out of 1 Number of reduce tasks is set to 0 since there's no reduce operator Starting Job = job_1406097234796_0041, Tracking URL = http://hadoop01:8088/proxy/application_1406097234796_0041/ Kill Command = /usr/lib/hadoop/bin/hadoop job -kill job_1406097234796_0041 Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 0 2014-12-08 17:35:56,995 Stage-1 map = 0%, reduce = 0% 2014-12-08 17:36:06,783 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 2.9 sec 2014-12-08 17:36:07,845 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 2.9 sec MapReduce Total cumulative CPU time: 2 seconds 900 msec Ended Job = job_1406097234796_0041 MapReduce Jobs Launched: Job 0: Map: 1 Cumulative CPU: 2.9 sec HDFS Read: 482 HDFS Write: 22 SUCCESS Total MapReduce CPU Time Spent: 2 seconds 900 msec OK 4 jolly 2014-09-10 CN?
?
外部表
外部表就是存儲不是由hive來存儲的，比如可以依賴Hbase來存儲，hive只是做一個映射而已。我用Hbase來舉例?
先建立一張Hbase表叫 employee
建立外部表進行映射
hive> CREATE EXTERNAL TABLE h_employee(key int, id int, name string) > STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' > WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key, info:id,info:name") > TBLPROPERTIES ("hbase.table.name" = "employee"); OK Time taken: 0.324 seconds hive> select * from h_employee; OK 1 1 peter 2 2 paul Time taken: 1.129 seconds, Fetched: 2 row(s)?
?
查詢語法
具體語法可以參考官方手冊https://cwiki.apache.org/confluence/display/Hive/Tutorial 我只說幾個比較奇怪的點
顯示條數
展示x條數據，用的還是limit，比如
hive> select * from h_employee limit 1> ; OK 1 1 peter Time taken: 0.284 seconds, Fetched: 1 row(s)但是不支持起點，比如offset?
?
?
（轉自：http://www.2cto.com/database/201412/359250.html?）
?
轉載于:https://www.cnblogs.com/rxbook/p/6294133.html
總結
- 上一篇： [OSG]OSG的相关扩展
- 下一篇： 我写的第一本书《TypeScript 入