架構(gòu)使用spark streaming 消費kafka的數(shù)據(jù)，并通過impala來插入到kudu中，但是通過對比發(fā)現(xiàn)落地到kudu表中的數(shù)據(jù)比kafka消息數(shù)要少，通過后臺日志發(fā)現(xiàn)，偶發(fā)性的出現(xiàn)java.sql.SQLException: [Simba][ImpalaJDBCDriver](500051) ERROR processing query/statement. Error Code: 0, SQL state: TStatus(statusCode:ERROR_STATUS, sqlState:HY000, errorMessage:AnalysisException: Syntax error in line 1 原因是調(diào)用過程中使用了數(shù)據(jù)庫連接池,會合并多行sql執(zhí)行，實現(xiàn)如下：

import com.alibaba.druid.pool.DruidDataSource; import com.alibaba.druid.pool.DruidDataSourceFactory; import com.alibaba.druid.pool.DruidPooledConnection; import org.apache.commons.logging.Log; import org.apache.commons.logging.LogFactory;import java.sql.SQLException; import java.util.Properties; public class ImapalConnPool {private static Log logger = LogFactory.getLog(ImapalConnPool.class);private static ImapalConnPool imapalConnPool = null;private static DruidDataSource druidDataSource = null;static {Properties properties = new Properties();properties.setProperty("driverClassName","com.cloudera.impala.jdbc41.Driver");properties.setProperty("url","jdbc:impala://127.0.0.1:21050"); properties.setProperty("username","");properties.setProperty("password","");properties.setProperty("initialSize","50");properties.setProperty("maxActive","100");properties.setProperty("maxWait","60000");properties.setProperty("timeBetweenEvictionRunsMillis","60000");properties.setProperty("minEvictableIdleTimeMillis","300000");properties.setProperty("validationQuery","SELECT 1");properties.setProperty("testWhileIdle","true");properties.setProperty("testOnBorrow","false");properties.setProperty("testOnReturn","false");properties.setProperty("poolPreparedStatements","false");//當該值大于0時，啟用pool，poolPreparedStatements為trueproperties.setProperty("maxPoolPreparedStatementPerConnectionSize","-1");try {druidDataSource = (DruidDataSource) DruidDataSourceFactory.createDataSource(properties); //DruidDataSrouce工廠模式} catch (Exception e) {logger.error(e);}}public static ImapalConnPool getInstance(){if (null == imapalConnPool){synchronized(ImapalConnPool.class) {if(null == imapalConnPool) {imapalConnPool = new ImapalConnPool();}}}return imapalConnPool;}public DruidPooledConnection getConnection() throws SQLException {return druidDataSource.getConnection();}}

?

調(diào)整上述參數(shù)poolPreparedStatements和maxPoolPreparedStatementPerConnectionSize任然不能解決同一個connection合并多行sql的問題 后去掉連接池，改為jdbc直連問題解決。有解決過上面數(shù)據(jù)庫連接池問題的麻煩告知我一下 Connection connectionn = null; Statement statement = null; try {Class.forName("com.cloudera.impala.jdbc41.Driver");connectionn = DriverManager.getConnection("jdbc:impala://127.0.0.1:21050");statement = connectionn.createStatement();for (String item : execSql) {statement.execute(item);}processResult = true; }catch (Exception ex){logger.error(this,ex); }finally {if(null != statement){try {statement.close();}catch (Exception ex){logger.error(this,ex);}}if(null != connectionn) {try {connectionn.close();}catch (Exception ex){logger.error(this,ex);}} }

?

?

轉(zhuǎn)載于:https://www.cnblogs.com/qizhelongdeyang/p/10339029.html

總結(jié)

以上是生活随笔為你收集整理的impala jdbc驱动执行impala sql的一个坑（不支持多行sql）的全部內(nèi)容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網(wǎng)站內(nèi)容還不錯，歡迎將生活随笔推薦給好友。