?

創建兩張表，通過一種是parquet ， 一種使用parquet snappy壓縮

創建表

使用snappy CREATE EXTERNAL TABLE IF NOT EXISTS tableName(xxx string) partitioned by (pt_xvc string) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\001' STORED AS PARQUET TBLPROPERTIES('parquet.compression'='SNAPPY');使用gzip CREATE EXTERNAL TABLE IF NOT EXISTS tableName(xxx string) partitioned by (pt_xvc string) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\001' STORED AS PARQUET TBLPROPERTIES('parquet.compression'='GZIP');使用uncompressed CREATE EXTERNAL TABLE IF NOT EXISTS tableName(xxx string) partitioned by (pt_xvc string) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\001' STORED AS PARQUET TBLPROPERTIES('parquet.compression'='UNCOMPRESSED');使用默認 CREATE EXTERNAL TABLE IF NOT EXISTS tableName(xxx string) partitioned by (pt_xvc string) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\001' STORED AS PARQUET;也可以在執行語句前執行 set parquet.compression=SNAPPY; 會對之后跑的數據進行壓縮，之前已經存在的不會進行snappy壓縮 通過 desc formatted tableName 查看表結構

使用parquet snappy

Table Type: EXTERNAL_TABLE Table Parameters: EXTERNAL TRUE numFiles 25 numPartitions 1 numRows 0 parquet.compression SNAPPY rawDataSize 0 totalSize 4570350557 transient_lastDdlTime 1552269085 # Storage Information SerDe Library: org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe InputFormat: org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat OutputFormat: org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat Compressed: No Num Buckets: -1 Bucket Columns: [] Sort Columns: [] Storage Desc Params: field.delim \u0001 serialization.format \u0001

使用parquet默認

Table Type: EXTERNAL_TABLE Table Parameters: EXTERNAL TRUE numFiles 25 numPartitions 1 numRows 0 rawDataSize 0 totalSize 4570650197 transient_lastDdlTime 1552269039 # Storage Information SerDe Library: org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe InputFormat: org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat OutputFormat: org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat Compressed: No Num Buckets: -1 Bucket Columns: [] Sort Columns: [] Storage Desc Params: field.delim \u0001 serialization.format \u0001

測試數據量：20208432?

UNCOMPRESSED :4570325699 PARQUET 默認 :4570650197 parquet gzip :4570314033 parquet snappy :4570350557 textfile :10356207038

?

通過對比發現，當數據量較少時parquet各壓縮方式差別不大，但相比TEXTFILE壓縮減少了1倍以上，后續再做一下性能對比測試一下。

總結

以上是生活随笔為你收集整理的Hive表 Parquet压缩 ， Gzip，Snappy，uncompressed 效果对比的全部內容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網站內容還不錯，歡迎將生活随笔推薦給好友。