我想運行一系列map reduce工作,所以最簡單的解決方案似乎是jobcontroller.說我有兩個工作,job1和job2.我想在job1之后運行job2.嗯,它遇到了一些問題.經過幾個小時的調試后,我將代碼縮小到以下幾行：

JobConf jobConf1 = new JobConf();

JobConf jobConf2 = new JobConf();

System.out.println("*** Point 1");

Job job1 = new Job(jobConf1);

System.out.println("*** Point 2");

Job job2 = new Job(jobConf2);

System.out.println("*** Point 3");

我在運行代碼時不斷獲得此輸出：

*** Point 1

10/12/06 17:19:30 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=

*** Point 2

10/12/06 17:19:30 INFO jvm.JvmMetrics: Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized

*** Point 3

我想我的問題是以某種方式與“無法初始化JMV ……”相關聯.那是什么？以及如何實例化多個作業,以便將它們傳遞給JobController.

當我在初始化第二個作業之前添加job1.waitForTheCompletion(true)時,它給了我這個錯誤：

10/12/07 11:28:21 INFO jvm.JvmMetrics: Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized

Exception in thread "main" org.apache.hadoop.mapreduce.lib.input.InvalidInputException: Input path does not exist: file:/home/workspace/WikipediaSearch/__TEMP1

at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(FileInputFormat.java:224)

at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:241)

at org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:885)

at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:779)

at org.apache.hadoop.mapreduce.Job.submit(Job.java:432)

at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:447)

at ch.ethz.nis.query.HadoopQuery.run(HadoopQuery.java:353)

at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)

at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)

at ch.ethz.nis.query.HadoopQuery.main(HadoopQuery.java:308)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

at java.lang.reflect.Method.invoke(Method.java:597)

at org.apache.hadoop.util.RunJar.main(RunJar.java:156)

__Temp1是第一個作業的輸出文件夾,我希望它是第二個作業的輸入.即使我的代碼中有這個waitForCompletion行,它仍然抱怨這條路徑不存在.

總結

以上是生活随笔為你收集整理的java多个mapreduce_java – 在hadoop中运行多个MapReduce作业的全部內容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網站內容還不錯，歡迎將生活随笔推薦給好友。