生活随笔
收集整理的這篇文章主要介紹了
MaxCompute 实现增量数据推送(全量比对增量逻辑)
小編覺得挺不錯的,現在分享給大家,幫大家做個參考.
ODPS 2.0 支持了很多新的集合命令(專有云升級到3版本后陸續支持)，簡化了日常工作中求集合操作的繁瑣程度。增加的SQL語法包括：UNOIN ALL、UNION DISTINCT并集，INTERSECT ALL、INTERSECT
select_statement UNION ALL select_statement;
select_statement UNION [DISTINCT] select_statement;
select_statement INTERSECT ALL select_statement;
select_statement INTERSECT [DISTINCT] select_statement;
select_statement EXCEPT ALL select_statement;
select_statement EXCEPT [DISTINCT] select_statement;
select_statement MINUS ALL select_statement;
select_statement MINUS [DISTINCT] select_statement;
用途：分別求兩個數據集的并集、交集以及求第二個數據集在第一個數據集中的補集。
具體語法參考：
實際項目中有一個利用兩日全量數據，比對出增量的需求（推送全量數據速度很慢，ADB／DRDS等產品數據量超過1億，建議試用增量同步）。我按照舊的JOIN方法和新的集合方法做了下比對驗證，試用了下新的集合命令EXCEPT ALL。測試
-- 方法一：JOIN
-- other_columns 代表很多列
create table tmp_opcode1 as
select * from(
select uuid,other_columns,opcode2
from(
-- 今日新增+今日變化
selectt1.uuid
,t1.other_columns
,case when t2.uuid is null then 'I' else 'U' end AS opcode2from prject1.table1 t1left outer join prject1.table1 t2on t1.uuid=t2.uuidand t2.dt='20200730'where t1.dt='20200731'and(t2.uuid is nullor coalesce(t1.other_columns,'')<>coalesce(t2.other_columns,''))union all
-- 今日刪除
selectt2.uuid
,t2.other_columns
,'D' as opcode2from prject1.table1 t2left outer join prject1.table1 t1on t1.uuid=t2.uuidand t1.dt='20200731'where t2.dt='20200730'and t1.uuid is null)t3)t4
;
Summary:
resource cost: cpu 13.37 Core * Min, memory 30.48 GB * Min
inputs:
prject1.table1/dt=20200730: 32530802 (946172216 bytes)
prject1.table1/dt=20200731: 32533538 (947161664 bytes)
outputs:
prject1.tmp_opcode1: 4506 (271632 bytes)
Job run time: 26.000
-- 方法二：集合
-- other_columns 代表很多列
create table tmp_opcode2 as
select * from(
select t3.*
from(
-- 今日新增+今日變化
select uuid,other_columns,'I' as opcode2
from(
select uuid,other_columns
from prject1.table1
where dt = '20200731'except all
select uuid,other_columns
from prject1.table1
where dt = '20200730')tunion all
-- 今日刪除
selectt2.uuid
,t2.other_columns
,'D' as opcode2from prject1.table1 t2left outer join prject1.table1 t1on t1.uuid=t2.uuidand t1.dt='20200731'where t2.dt='20200730'and t1.uuid is null)t3)t4
;
Summary:
resource cost: cpu 35.92 Core * Min, memory 74.26 GB * Min
inputs:
prject1.table1/rfq=20200730: 32530802 (946172216 bytes)
prject1.table1/rfq=20200731: 32533538 (947161664 bytes)
outputs:
prject1.tmp_opcode2: 4506 (259416 bytes)
Job run time: 66.000
性能 結果
?
原文鏈接
總結
以上是生活随笔 為你收集整理的MaxCompute 实现增量数据推送(全量比对增量逻辑) 的全部內容，希望文章能夠幫你解決所遇到的問題。
如果覺得生活随笔 網站內容還不錯，歡迎將生活随笔 推薦給好友。
