select报错 spark_spark-sql master on yarn 模式运行 select count(*) 报错日志
啟動hive --service metastore
啟動 dfs yarn
[root@bigdatastorm bin]# ./spark-sql --master yarn --deploy-mode client --driver-memory 512m --executor-memory 512m --total-executor-cores 1
spark-sql>select count(*)? ;
Log
=======================================================
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/hadoop-2.5.1/nm-local-dir/usercache/root/filecache/11/spark-assembly-1.6.0-hadoop2.6.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
16/09/05 21:59:45 INFO yarn.ApplicationMaster: Registered signal handlers for [TERM, HUP, INT]
16/09/05 21:59:50 INFO yarn.ApplicationMaster: ApplicationAttemptId: appattempt_1473082245027_0003_000001
16/09/05 21:59:53 INFO spark.SecurityManager: Changing view acls to: root
16/09/05 21:59:53 INFO spark.SecurityManager: Changing modify acls to: root
16/09/05 21:59:53 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); users with modify permissions: Set(root)
16/09/05 21:59:55 INFO yarn.ApplicationMaster: Waiting for Spark driver to be reachable.
16/09/05 21:59:55 INFO yarn.ApplicationMaster: Driver now available: 192.168.184.188:45475
16/09/05 21:59:57 INFO yarn.ApplicationMaster$AMEndpoint: Add WebUI Filter. AddWebUIFilter(org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter,Map(PROXY_HOSTS -> bigdatastorm, PROXY_URI_BASES -> http://bigdatastorm:8088/proxy/application_1473082245027_0003),/proxy/application_1473082245027_0003)
16/09/05 21:59:57 INFO yarn.YarnRMClient: Registering the ApplicationMaster
16/09/05 21:59:58 INFO yarn.YarnAllocator: Will request 1 executor containers, each with 1 cores and 896 MB memory including 384 MB overhead
16/09/05 21:59:58 INFO yarn.YarnAllocator: Container request (host: Any, capability: <896 vcores:1>)896>
16/09/05 21:59:58 INFO yarn.ApplicationMaster: Started progress reporter thread with (heartbeat : 3000, initial allocation : 200) intervals
16/09/05 21:59:58 INFO impl.AMRMClientImpl: Received new token for : bigdatastorm:59055
16/09/05 21:59:58 INFO yarn.YarnAllocator: Launching container container_1473082245027_0003_01_000002 for on host bigdatastorm
16/09/05 21:59:58 INFO yarn.YarnAllocator: Launching ExecutorRunnable. driverUrl: spark://CoarseGrainedScheduler@192.168.184.188:45475, executorHostname: bigdatastorm
16/09/05 21:59:58 INFO yarn.YarnAllocator: Received 1 containers from YARN, launching executors on 1 of them.
16/09/05 21:59:58 INFO yarn.ExecutorRunnable: Starting Executor Container
16/09/05 21:59:58 INFO impl.ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0
16/09/05 21:59:58 INFO yarn.ExecutorRunnable: Setting up ContainerLaunchContext
16/09/05 21:59:58 INFO yarn.ExecutorRunnable: Preparing Local resources
16/09/05 21:59:59 INFO yarn.ExecutorRunnable: Prepared Local resources Map(__spark__.jar -> resource { scheme: "hdfs" host: "mycluster" port: -1 file: "/user/root/.sparkStaging/application_1473082245027_0003/spark-assembly-1.6.0-hadoop2.6.0.jar" } size: 187548272 timestamp: 1473083954792 type: FILE visibility: PRIVATE)
16/09/05 21:59:59 INFO yarn.ExecutorRunnable:
===============================================================================
YARN executor launch context:
env:
CLASSPATH -> {{PWD}}{{PWD}}/__spark__.jar$HADOOP_CONF_DIR$HADOOP_COMMON_HOME/share/hadoop/common/*$HADOOP_COMMON_HOME/share/hadoop/common/lib/*$HADOOP_HDFS_HOME/share/hadoop/hdfs/*$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*$HADOOP_YARN_HOME/share/hadoop/yarn/*$HADOOP_YARN_HOME/share/hadoop/yarn/lib/*$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*
SPARK_LOG_URL_STDERR -> http://bigdatastorm:8042/node/containerlogs/container_1473082245027_0003_01_000002/root/stderr?start=-4096
SPARK_YARN_STAGING_DIR -> .sparkStaging/application_1473082245027_0003
SPARK_YARN_CACHE_FILES_FILE_SIZES -> 187548272
SPARK_USER -> root
SPARK_YARN_CACHE_FILES_VISIBILITIES -> PRIVATE
SPARK_YARN_MODE -> true
SPARK_YARN_CACHE_FILES_TIME_STAMPS -> 1473083954792
SPARK_LOG_URL_STDOUT -> http://bigdatastorm:8042/node/containerlogs/container_1473082245027_0003_01_000002/root/stdout?start=-4096
SPARK_YARN_CACHE_FILES -> hdfs://mycluster/user/root/.sparkStaging/application_1473082245027_0003/spark-assembly-1.6.0-hadoop2.6.0.jar#__spark__.jar
command:
{{JAVA_HOME}}/bin/java -server -XX:OnOutOfMemoryError='kill %p' -Xms512m -Xmx512m -Djava.io.tmpdir={{PWD}}/tmp '-Dspark.driver.port=45475' -Dspark.yarn.app.container.log.dir= -XX:MaxPermSize=256m org.apache.spark.executor.CoarseGrainedExecutorBackend --driver-url spark://CoarseGrainedScheduler@192.168.184.188:45475 --executor-id 1 --hostname bigdatastorm --cores 1 --app-id application_1473082245027_0003 --user-class-path file:$PWD/__app__.jar 1> /stdout 2> /stderr
===============================================================================
16/09/05 21:59:59 INFO impl.ContainerManagementProtocolProxy: Opening proxy : bigdatastorm:59055
16/09/05 22:09:45 INFO yarn.YarnAllocator: Completed container container_1473082245027_0003_01_000002 on host: bigdatastorm (state: COMPLETE, exit status: 50)
16/09/05 22:09:45 WARN yarn.YarnAllocator: Container marked as failed: container_1473082245027_0003_01_000002 on host: bigdatastorm. Exit status: 50. Diagnostics: Exception from container-launch: ExitCodeException exitCode=50:
ExitCodeException exitCode=50:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
at org.apache.hadoop.util.Shell.run(Shell.java:455)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:300)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Container exited with a non-zero exit code 50
16/09/05 22:09:48 INFO yarn.YarnAllocator: Will request 1 executor containers, each with 1 cores and 896 MB memory including 384 MB overhead
16/09/05 22:09:48 INFO yarn.YarnAllocator: Container request (host: Any, capability: <896 vcores:1>)896>
16/09/05 22:09:49 INFO yarn.YarnAllocator: Launching container container_1473082245027_0003_01_000003 for on host bigdatastorm
16/09/05 22:09:49 INFO yarn.YarnAllocator: Launching ExecutorRunnable. driverUrl: spark://CoarseGrainedScheduler@192.168.184.188:45475, executorHostname: bigdatastorm
16/09/05 22:09:49 INFO yarn.YarnAllocator: Received 1 containers from YARN, launching executors on 1 of them.
16/09/05 22:09:49 INFO yarn.ExecutorRunnable: Starting Executor Container
16/09/05 22:09:49 INFO impl.ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0
16/09/05 22:09:49 INFO yarn.ExecutorRunnable: Setting up ContainerLaunchContext
16/09/05 22:09:49 INFO yarn.ExecutorRunnable: Preparing Local resources
16/09/05 22:09:49 INFO yarn.ExecutorRunnable: Prepared Local resources Map(__spark__.jar -> resource { scheme: "hdfs" host: "mycluster" port: -1 file: "/user/root/.sparkStaging/application_1473082245027_0003/spark-assembly-1.6.0-hadoop2.6.0.jar" } size: 187548272 timestamp: 1473083954792 type: FILE visibility: PRIVATE)
16/09/05 22:09:49 INFO yarn.ExecutorRunnable:
===============================================================================
YARN executor launch context:
env:
CLASSPATH -> {{PWD}}{{PWD}}/__spark__.jar$HADOOP_CONF_DIR$HADOOP_COMMON_HOME/share/hadoop/common/*$HADOOP_COMMON_HOME/share/hadoop/common/lib/*$HADOOP_HDFS_HOME/share/hadoop/hdfs/*$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*$HADOOP_YARN_HOME/share/hadoop/yarn/*$HADOOP_YARN_HOME/share/hadoop/yarn/lib/*$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*
SPARK_LOG_URL_STDERR -> http://bigdatastorm:8042/node/containerlogs/container_1473082245027_0003_01_000003/root/stderr?start=-4096
SPARK_YARN_STAGING_DIR -> .sparkStaging/application_1473082245027_0003
SPARK_YARN_CACHE_FILES_FILE_SIZES -> 187548272
SPARK_USER -> root
SPARK_YARN_CACHE_FILES_VISIBILITIES -> PRIVATE
SPARK_YARN_MODE -> true
SPARK_YARN_CACHE_FILES_TIME_STAMPS -> 1473083954792
SPARK_LOG_URL_STDOUT -> http://bigdatastorm:8042/node/containerlogs/container_1473082245027_0003_01_000003/root/stdout?start=-4096
SPARK_YARN_CACHE_FILES -> hdfs://mycluster/user/root/.sparkStaging/application_1473082245027_0003/spark-assembly-1.6.0-hadoop2.6.0.jar#__spark__.jar
command:
{{JAVA_HOME}}/bin/java -server -XX:OnOutOfMemoryError='kill %p' -Xms512m -Xmx512m -Djava.io.tmpdir={{PWD}}/tmp '-Dspark.driver.port=45475' -Dspark.yarn.app.container.log.dir= -XX:MaxPermSize=256m org.apache.spark.executor.CoarseGrainedExecutorBackend --driver-url spark://CoarseGrainedScheduler@192.168.184.188:45475 --executor-id 2 --hostname bigdatastorm --cores 1 --app-id application_1473082245027_0003 --user-class-path file:$PWD/__app__.jar 1> /stdout 2> /stderr
===============================================================================
16/09/05 22:09:49 INFO impl.ContainerManagementProtocolProxy: Opening proxy : bigdatastorm:59055
16/09/05 22:12:14 INFO yarn.YarnAllocator: Completed container container_1473082245027_0003_01_000003 on host: bigdatastorm (state: COMPLETE, exit status: 1)
16/09/05 22:12:14 WARN yarn.YarnAllocator: Container marked as failed: container_1473082245027_0003_01_000003 on host: bigdatastorm. Exit status: 1. Diagnostics: Exception from container-launch: ExitCodeException exitCode=1:
ExitCodeException exitCode=1:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
at org.apache.hadoop.util.Shell.run(Shell.java:455)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:300)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Container exited with a non-zero exit code 1
16/09/05 22:12:17 INFO yarn.YarnAllocator: Will request 1 executor containers, each with 1 cores and 896 MB memory including 384 MB overhead
16/09/05 22:12:17 INFO yarn.YarnAllocator: Container request (host: Any, capability: <896 vcores:1>)896>
16/09/05 22:12:18 INFO impl.AMRMClientImpl: Received new token for : bigdatahadoop:39892
16/09/05 22:12:18 INFO yarn.YarnAllocator: Launching container container_1473082245027_0003_01_000004 for on host bigdatahadoop
16/09/05 22:12:18 INFO yarn.YarnAllocator: Launching ExecutorRunnable. driverUrl: spark://CoarseGrainedScheduler@192.168.184.188:45475, executorHostname: bigdatahadoop
16/09/05 22:12:18 INFO yarn.YarnAllocator: Received 1 containers from YARN, launching executors on 1 of them.
16/09/05 22:12:18 INFO yarn.ExecutorRunnable: Starting Executor Container
16/09/05 22:12:18 INFO impl.ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0
16/09/05 22:12:18 INFO yarn.ExecutorRunnable: Setting up ContainerLaunchContext
16/09/05 22:12:18 INFO yarn.ExecutorRunnable: Preparing Local resources
16/09/05 22:12:18 INFO yarn.ExecutorRunnable: Prepared Local resources Map(__spark__.jar -> resource { scheme: "hdfs" host: "mycluster" port: -1 file: "/user/root/.sparkStaging/application_1473082245027_0003/spark-assembly-1.6.0-hadoop2.6.0.jar" } size: 187548272 timestamp: 1473083954792 type: FILE visibility: PRIVATE)
16/09/05 22:12:18 INFO yarn.ExecutorRunnable:
===============================================================================
YARN executor launch context:
env:
CLASSPATH -> {{PWD}}{{PWD}}/__spark__.jar$HADOOP_CONF_DIR$HADOOP_COMMON_HOME/share/hadoop/common/*$HADOOP_COMMON_HOME/share/hadoop/common/lib/*$HADOOP_HDFS_HOME/share/hadoop/hdfs/*$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*$HADOOP_YARN_HOME/share/hadoop/yarn/*$HADOOP_YARN_HOME/share/hadoop/yarn/lib/*$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*
SPARK_LOG_URL_STDERR -> http://bigdatahadoop:8042/node/containerlogs/container_1473082245027_0003_01_000004/root/stderr?start=-4096
SPARK_YARN_STAGING_DIR -> .sparkStaging/application_1473082245027_0003
SPARK_YARN_CACHE_FILES_FILE_SIZES -> 187548272
SPARK_USER -> root
SPARK_YARN_CACHE_FILES_VISIBILITIES -> PRIVATE
SPARK_YARN_MODE -> true
SPARK_YARN_CACHE_FILES_TIME_STAMPS -> 1473083954792
SPARK_LOG_URL_STDOUT -> http://bigdatahadoop:8042/node/containerlogs/container_1473082245027_0003_01_000004/root/stdout?start=-4096
SPARK_YARN_CACHE_FILES -> hdfs://mycluster/user/root/.sparkStaging/application_1473082245027_0003/spark-assembly-1.6.0-hadoop2.6.0.jar#__spark__.jar
command:
{{JAVA_HOME}}/bin/java -server -XX:OnOutOfMemoryError='kill %p' -Xms512m -Xmx512m -Djava.io.tmpdir={{PWD}}/tmp '-Dspark.driver.port=45475' -Dspark.yarn.app.container.log.dir= -XX:MaxPermSize=256m org.apache.spark.executor.CoarseGrainedExecutorBackend --driver-url spark://CoarseGrainedScheduler@192.168.184.188:45475 --executor-id 3 --hostname bigdatahadoop --cores 1 --app-id application_1473082245027_0003 --user-class-path file:$PWD/__app__.jar 1> /stdout 2> /stderr
===============================================================================
16/09/05 22:12:18 INFO impl.ContainerManagementProtocolProxy: Opening proxy : bigdatahadoop:39892
16/09/05 22:14:36 INFO yarn.YarnAllocator: Completed container container_1473082245027_0003_01_000004 on host: bigdatahadoop (state: COMPLETE, exit status: 1)
16/09/05 22:14:36 WARN yarn.YarnAllocator: Container marked as failed: container_1473082245027_0003_01_000004 on host: bigdatahadoop. Exit status: 1. Diagnostics: Exception from container-launch: ExitCodeException exitCode=1:
ExitCodeException exitCode=1:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
at org.apache.hadoop.util.Shell.run(Shell.java:455)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:300)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Container exited with a non-zero exit code 1
16/09/05 22:14:39 INFO yarn.ApplicationMaster: Final app status: FAILED, exitCode: 11, (reason: Max number of executor failures (3) reached)
16/09/05 22:14:42 INFO util.ShutdownHookManager: Shutdown hook called
創作挑戰賽新人創作獎勵來咯，堅持創作打卡瓜分現金大獎總結
以上是生活随笔為你收集整理的select报错 spark_spark-sql master on yarn 模式运行 select count(*) 报错日志的全部內容，希望文章能夠幫你解決所遇到的問題。
- 上一篇： linux内存使用策略swap,Linu
- 下一篇： quill鼠标悬浮 出现提示_「超逸酷玩