hive有三種默認(rèn)的存儲(chǔ)格式，TEXT、ORC、PARQUET。TEXT是默認(rèn)的格式，ORC、PARQUET是列存儲(chǔ)格式，占用空間和查詢效率是不同的，專門測(cè)試過后記錄一下。

一：建表語句差別

create table if not exists text(

a bigint

) partitioned by (dt string)

row format delimited fields terminated by '\001'

location '/hdfs/text/';

create table if not exists orc(

a bigint)

partitioned by (dt string)

row format delimited fields terminated by '\001'

stored as orc

location '/hdfs/orc/';

create table if not exists parquet(

a bigint)

partitioned by (dt string)

row format delimited fields terminated by '\001'

stored as parquet

location '/hdfs/parquet/';

其實(shí)就是stored as 后面跟的不一樣

二：HDFS存儲(chǔ)對(duì)比

parquet

orc

text

709M

275M

1G

687M

249M

1G

647M

265M

1G

三：查詢時(shí)間對(duì)比

parquet

orc

text

36.451

26.133

42.574

38.425

29.353

41.673

36.647

27.825

43.938

四：文件如何生成

val sparkSession = SparkSession.builder().master("local").appName("pushFunnelV3").getOrCreate()

val javasc = new JavaSparkContext(sparkSession.sparkContext)

val nameRDD = javasc.parallelize(util.Arrays.asList("{'name':'zhangsan','age':'18'}", "{'name':'lisi','age':'19'}")).rdd;

sparkSession.read.json(nameRDD).write.mode(SaveMode.Overwrite).csv("/data/aa")

sparkSession.read.json(nameRDD).write.mode(SaveMode.Overwrite).orc("/data/bb")

sparkSession.read.json(nameRDD).write.mode(SaveMode.Overwrite).parquet("/data/cc")

總結(jié)

以上是生活随笔為你收集整理的hdfs orc格式_HIVE存储格式ORC、PARQUET对比的全部?jī)?nèi)容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網(wǎng)站內(nèi)容還不錯(cuò)，歡迎將生活随笔推薦給好友。