從一個(gè)表里group by 之后 計(jì)算累加值、去重值:

為了效率設(shè)置并行：set hive.exec.parallel=true（可選：set hive.exec.parallel.thread.number=16）、set hive.groupby.skewindata=true、set hive.map.aggr=true

select plat, pagetype, count(*) pv, count(distinct userkey) uv from client_pv_form where dt = '2015-08-19' group by plat, pagetype union all select plat, 'all' pagetype, count(*) pv, count(distinct userkey) uv from client_pv_form where dt = '2015-08-19' group by plat union all select 'all' plat, pagetype, count(*) pv, count(distinct userkey) uv from client_pv_form where dt = '2015-08-19' group by pagetype union all select 'all' plat, 'all' pagetype, count(*) pv, count(distinct userkey) uv from client_pv_form where dt = '2015-08-19'

壞就壞在：set hive.map.aggr=true，map端聚合的設(shè)置；

出來(lái)的pv數(shù)跟真實(shí)值對(duì)不上；

改成下邊代碼運(yùn)行正確；

select plat, pagetype, sum(1) pv, count(distinct userkey) uv from client_pv_form where dt = '2015-08-19' group by plat, pagetype union all select plat, 'all' pagetype, sum(1) pv, count(distinct userkey) uv from client_pv_form where dt = '2015-08-19' group by plat union all select 'all' plat, pagetype, sum(1) pv, count(distinct userkey) uv from client_pv_form where dt = '2015-08-19' group by pagetype union all select 'all' plat, 'all' pagetype, sum(1) pv, count(distinct userkey) uv from client_pv_form where dt = '2015-08-19'

?

轉(zhuǎn)載于:https://www.cnblogs.com/sudz/p/4745985.html

與50位技術(shù)專(zhuān)家面對(duì)面20年技術(shù)見(jiàn)證，附贈(zèng)技術(shù)全景圖

總結(jié)

以上是生活随笔為你收集整理的set hive.map.aggr=true 时统计PV数据错误的全部?jī)?nèi)容，希望文章能夠幫你解決所遇到的問(wèn)題。

如果覺(jué)得生活随笔網(wǎng)站內(nèi)容還不錯(cuò)，歡迎將生活随笔推薦給好友。