一、hive配置文件?

在spak/conf目錄添加hive-site.xml配置，設(shè)置mysql作為元數(shù)據(jù)存儲的數(shù)據(jù)庫

<?xml version="1.0" encoding="UTF-8" standalone="no"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <configuration><property><name>javax.jdo.option.ConnectionURL</name><value>jdbc:mysql://192.168.150.1:3306/spark_metadata_db?createDatabaseIfNotExist=true&amp;characterEncoding=UTF-8</value><description>JDBC connect string for a JDBC metastore</description></property><property><name>javax.jdo.option.ConnectionDriverName</name><value>com.mysql.jdbc.Driver</value><description>Driver class name for a JDBC metastore</description></property><property><name>javax.jdo.option.ConnectionUserName</name><value>root</value><description>Username to use against metastore database</description></property><property><name>javax.jdo.option.ConnectionPassword</name><value>admin</value><description>password to use against metastore database</description></property><!-- hive查詢時輸出列名 --><property><name>hive.cli.print.header</name><value>true</value></property><!-- 顯示當(dāng)前數(shù)據(jù)庫名 --><property><name>hive.cli.print.current.db</name><value>true</value></property> </configuration>

?

二、啟動spark-sql shell

--driver-class-path 是spark元數(shù)據(jù)存儲的驅(qū)動類路徑，這里使用mysql作為metastore，故使用mysql-connector-java-5.1.26-bin.jar?

--jars 是executer執(zhí)行器的額外添加類的路徑，這里使用mysql的test表進(jìn)行操作，故使用mysql-connector-java-5.1.26-bin.jar?

--total-executor-cores? 啟動的核數(shù)，默認(rèn)是所有核數(shù)

--executor-memory? 每個work分配的內(nèi)存，默認(rèn)是work的所有內(nèi)存

cd ~/software/spark-2.4.4-bin-hadoop2.6 bin/spark-sql --master spark://hadoop01:7077,hadoop02:7077,hadoop03:7077 --driver-class-path /home/mk/mysql-connector-java-5.1.26-bin.jar --jars /home/mk/mysql-connector-java-5.1.26-bin.jar --total-executor-cores 2 --executor-memory 1g

?

啟動shell前：

啟動shell后：

??mysql數(shù)據(jù)庫里面創(chuàng)建了spark_metadata_db

?

三、執(zhí)行sql

show tables;create table test(id int, name string) USING org.apache.spark.sql.jdbc options(url 'jdbc:mysql://192.168.150.1:3306/spark-mysql?user=root&password=admin', dbtable 'test_a');show tables;select * from test;insert into test values(4, 'd'), (5, 'e'), (6, 'f');select * from test;

?

?

?

?

?

總結(jié)

以上是生活随笔為你收集整理的Spark入门（六）Spark SQL shell启动方式(元数据存储在mysql)的全部內(nèi)容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網(wǎng)站內(nèi)容還不錯，歡迎將生活随笔推薦給好友。