?

創(chuàng)建兩張表，通過一種是parquet ， 一種使用parquet snappy壓縮

創(chuàng)建表

使用snappy CREATE EXTERNAL TABLE IF NOT EXISTS tableName(xxx string) partitioned by (pt_xvc string) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\001' STORED AS PARQUET TBLPROPERTIES('parquet.compression'='SNAPPY');使用gzip CREATE EXTERNAL TABLE IF NOT EXISTS tableName(xxx string) partitioned by (pt_xvc string) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\001' STORED AS PARQUET TBLPROPERTIES('parquet.compression'='GZIP');使用uncompressed CREATE EXTERNAL TABLE IF NOT EXISTS tableName(xxx string) partitioned by (pt_xvc string) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\001' STORED AS PARQUET TBLPROPERTIES('parquet.compression'='UNCOMPRESSED');使用默認(rèn) CREATE EXTERNAL TABLE IF NOT EXISTS tableName(xxx string) partitioned by (pt_xvc string) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\001' STORED AS PARQUET;也可以在執(zhí)行語(yǔ)句前執(zhí)行 set parquet.compression=SNAPPY; 會(huì)對(duì)之后跑的數(shù)據(jù)進(jìn)行壓縮，之前已經(jīng)存在的不會(huì)進(jìn)行snappy壓縮 通過 desc formatted tableName 查看表結(jié)構(gòu)

使用parquet snappy

Table Type: EXTERNAL_TABLE Table Parameters: EXTERNAL TRUE numFiles 25 numPartitions 1 numRows 0 parquet.compression SNAPPY rawDataSize 0 totalSize 4570350557 transient_lastDdlTime 1552269085 # Storage Information SerDe Library: org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe InputFormat: org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat OutputFormat: org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat Compressed: No Num Buckets: -1 Bucket Columns: [] Sort Columns: [] Storage Desc Params: field.delim \u0001 serialization.format \u0001

使用parquet默認(rèn)

Table Type: EXTERNAL_TABLE Table Parameters: EXTERNAL TRUE numFiles 25 numPartitions 1 numRows 0 rawDataSize 0 totalSize 4570650197 transient_lastDdlTime 1552269039 # Storage Information SerDe Library: org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe InputFormat: org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat OutputFormat: org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat Compressed: No Num Buckets: -1 Bucket Columns: [] Sort Columns: [] Storage Desc Params: field.delim \u0001 serialization.format \u0001

測(cè)試數(shù)據(jù)量：20208432?

UNCOMPRESSED :4570325699 PARQUET 默認(rèn) :4570650197 parquet gzip :4570314033 parquet snappy :4570350557 textfile :10356207038

?

通過對(duì)比發(fā)現(xiàn)，當(dāng)數(shù)據(jù)量較少時(shí)parquet各壓縮方式差別不大，但相比TEXTFILE壓縮減少了1倍以上，后續(xù)再做一下性能對(duì)比測(cè)試一下。

總結(jié)

以上是生活随笔為你收集整理的Hive表 Parquet压缩 ， Gzip，Snappy，uncompressed 效果对比的全部?jī)?nèi)容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網(wǎng)站內(nèi)容還不錯(cuò)，歡迎將生活随笔推薦給好友。