Job執行可以分為本地執行或者集群執行。hadoop集群安裝部署在遠程centos系統中。使用經典的WordCount代碼為例。
1. 本地執行模式（本地為MacOS環境），無需啟動遠程的hadoop集群，本地job會提交給本地執行器LocalJobRunner去執行。
1）輸入輸出數據存放在本地路徑下：
首先，MapReduce代碼如下：
package com.nasuf.hadoop.mr;import java.io.IOException;import org.apache.commons.lang.StringUtils;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Mapper;public class WCMapper extends Mapper<LongWritable, Text, Text, LongWritable> {@Overrideprotected void map(LongWritable key, Text value, Context context)throws IOException, InterruptedException {String line = value.toString();String[] words = StringUtils.split(line, " ");for (String word: words) {context.write(new Text(word), new LongWritable(1));}}}
package com.nasuf.hadoop.mr;import java.io.IOException;import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Reducer;public class WCReducer extends Reducer<Text, LongWritable, Text, LongWritable>{@Overrideprotected void reduce(Text key, Iterable<LongWritable> values, Context context) throws IOException, InterruptedException {long count = 0;for (LongWritable value: values) {count += value.get();}context.write(key, new LongWritable(count));}}
package com.nasuf.hadoop.mr;import java.io.IOException;import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;public class WCRunner {public static void main(String[] args) throws IOException, ClassNotFoundException, InterruptedException {Configuration conf = new Configuration();Job job = Job.getInstance(conf);// 設置整個job所用的類在哪個jar包job.setJarByClass(WCRunner.class);// 本job實用的mapper和reducer的類job.setMapperClass(WCMapper.class);job.setReducerClass(WCReducer.class);// 指定reducer的輸出數據kv類型（若不指定下面mapper的輸出類型，此處可以同時表明mapper和reducer的輸出類型）job.setOutputKeyClass(Text.class);job.setOutputValueClass(LongWritable.class);// 指定mapper的輸出數據kv類型job.setMapOutputKeyClass(Text.class);job.setMapOutputValueClass(LongWritable.class);// 指定原始數據存放位置FileInputFormat.setInputPaths(job, new Path("/Users/nasuf/Desktop/wc/srcdata"));// 處理結果的輸出數據存放路徑FileOutputFormat.setOutputPath(job, new Path("/Users/nasuf/Desktop/wc/output"));// 將job提交給集群運行job.waitForCompletion(true);}}
在本地模式中，可以將測試數據存放在"/Users/nasuf/Desktop/wc/srcdata"路徑下，注意輸出路徑不能是已經存在的路徑，不然會拋出異常。
// 指定原始數據存放位置FileInputFormat.setInputPaths(job, new Path("hdfs://hdcluster01:9000/wc/srcdata"));// 處理結果的輸出數據存放路徑FileOutputFormat.setOutputPath(job, new Path("hdfs://hdcluster01:9000/wc/output1"));
如果出現如下錯誤：
org.apache.hadoop.security.AccessControlException: Permission denied: user=nasuf, access=WRITE, inode="/wc":parallels:supergroup:drwxr-xr-x
顯然是權限問題。hadoop的用戶目錄是parallels，權限是rwxr-xr-x，而本地操作使用的用戶是nasuf。解決方法如下：在vm啟動參數中加入如下參數：-DHADOOP_USER_NAME=parallels即可。
2. 集群執行模式（首先需要啟動yarn，job會提交到yarn框架中去執行。訪問http://hdcluster01:8088可以查看job執行狀態。）
1）使用命令直接執行jar
hadoop jar wc.jar com.nasuf.hadoop.mr.WCRunner
查看http://hdcluster01:8088中job執行狀態
2) 通過main方法直接在本地提交job到yarn集群中執行
conf.set("mapreduce.framework.name", "yarn");
conf.set("yarn.resourcemanager.hostname", "hdcluster01");
conf.set("yarn.nodemanager.aux-services", "mapreduce_shuffle");
如果出現如下錯誤信息：
2018-08-26 10:25:37,544 INFO [main] mapreduce.Job (Job.java:monitorAndPrintJob(1375)) - Job job_1535213323614_0010 failed with state FAILED due to: Application application_1535213323614_0010 failed 2 times due to AM Container for appattempt_1535213323614_0010_000002 exited with exitCode: -1000 due to: File file:/tmp/hadoop-yarn/staging/nasuf/.staging/job_1535213323614_0010/job.jar does not exist
.Failing this attempt.. Failing the application.
可以將core-site.xml配置文件同時拷貝到classpath中，或者同樣配置如下參數：
conf.set("hadoop.tmp.dir", "/home/parallels/app/hadoop-2.4.1/data/");
即可解決問題。
總結
以上是生活随笔 為你收集整理的Hadoop系列（三）MapReduce Job的几种提交运行模式 的全部內容，希望文章能夠幫你解決所遇到的問題。
如果覺得生活随笔 網站內容還不錯，歡迎將生活随笔 推薦給好友。
