背景：

數(shù)據(jù)表字段有IP，IP省份，IP城市，最新訪問時(shí)間，假如目前一共有100W的IP我第一次初始化的時(shí)候完成初始化表插入，然后每天都有5W左右的IP活躍，然后我要更新這5W個(gè)IP的記錄到初始化的表，他的IP省份如果有變更也要更新保留最新的數(shù)據(jù)，更新最新的訪問時(shí)間等。

所有操作均在hive表上

1.需要2個(gè)表 數(shù)據(jù)拉鏈表，每日更新數(shù)據(jù)表

DDL語句如下

每日更新表

create table dm.day_tbl(

ip string comment'ip',

country string comment'國(guó)家',

province string comment'省份',

city string comment'城市',

day string comment'day',

)

comment '每日更新表'

row format delimited

fields terminated by '\u0001'

stored as parquet;

數(shù)據(jù)拉鏈表

create table dm.zipper_tbl(

ip string comment'ip',

country string comment'國(guó)家',

province string comment'省份',

city string comment'城市',

t_start_date string comment'start_date',

t_end_date string comment'end_date'

)

comment '數(shù)據(jù)拉鏈表'

row format delimited

fields terminated by '\u0001'

stored as parquet;

ETL語句如下

INSERT OVERWRITE TABLE dm.zipper_tbl

SELECT * FROM

(

SELECT zipper_tbl.ip,

zipper_tbl.country,

zipper_tbl.province,

zipper_tbl.city,

zipper_tbl.t_start_date,

CASE

WHEN zipper_tbl.t_end_date = '9999-12-31' AND day_tbl.ip IS NOT NULL THEN date_add(day_tbl.day,-1)

ELSE zipper_tbl.t_end_date

END AS t_end_date

FROM dm.zipper_tbl AS zipper_tbl

LEFT JOIN dm.day_tbl AS day_tbl

ON zipper_tbl.ip = day_tbl.ip

UNION ALL

SELECT union_day_tbl.ip,

union_day_tbl.country,

union_day_tbl.province,

union_day_tbl.city,

union_day_tbl.day AS t_start_date,

'9999-12-31' AS t_end_date

FROM dm.day_tbl AS union_day_tbl

) AS main_tbl WHERE main_tbl.t_end_date = "9999-12-31"

上面ETL是參考常見的拉鏈表，最后過濾一下時(shí)間，就可以得到全量最新的數(shù)據(jù)

總結(jié)

以上是生活随笔為你收集整理的hive增量表和全量表_hive 拉链表 实现全量数据 增量更新的全部?jī)?nèi)容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網(wǎng)站內(nèi)容還不錯(cuò)，歡迎將生活随笔推薦給好友。