首先確保hadoop已經(jīng)正確安裝、配置以及運(yùn)行。

拷貝WordCount.java到我們的文件夾，下載的hadoop里帶有WordCount.java，路徑為：

hadoop-0.20.203.0/src/examples/org/apache/hadoop/examples/WordCount.java

進(jìn)行拷貝操作

[plain]?view plaincopy

[hadoop@localhost~]$?cp?hadoop-0.20.203.0/src/examples/org/apache/hadoop/examples/WordCount.java?~??

[plain]?view plaincopy

[hadoop@localhost~]$?ls??

Desktop? hadoop-0.20.203.0? WordCount.java

在當(dāng)前目錄下創(chuàng)建一個用來存放WordCount.class的文件夾

[plain]?view plaincopy

[hadoop@localhost~]$?mkdir?classes??

編譯WordCount.java

[plain]?view plaincopy

[hadoop@localhost~]$?javac?-classpath?hadoop-0.20.203.0/hadoop-core-0.20.203.0.jar?-d?classesWordCount.java??

上面的方法是按照hadoop自帶的doc文檔進(jìn)行編譯的，如果發(fā)現(xiàn)報錯，出現(xiàn)如下異常

WordCount.java:53:error: cannot access Options

??? String[] otherArgs = new GenericOptionsParser(conf, args).getRemainingArgs();

??????????????????????? ? ? ? ? ? ? ???????? ^

? class file for org.apache.commons.cli.Optionsnot found

1 error

則按如下進(jìn)行編譯

[plain]?view plaincopy

[hadoop@localhost~]$?javac?-classpath?hadoop-0.20.203.0/hadoop-core-0.20.203.0.jar:hadoop-0.20.203.0/lib/commons-cli-1.2.jar?-d?classes?WordCount.java??

編譯成功，classes下會出現(xiàn)一個org的文件夾

[plain]?view plaincopy

[hadoop@localhost~]$?ls?classes??

org

對編譯好的class進(jìn)行打包

[plain]?view plaincopy

[hadoop@localhost~]$?jar?-cvf?wordcount.jar?-C?classes/?.??

added manifest

adding: org/(in =0) (out= 0)(stored 0%)

adding:org/apache/(in = 0) (out= 0)(stored 0%)

adding:org/apache/hadoop/(in = 0) (out= 0)(stored 0%)

adding: org/apache/hadoop/examples/(in= 0) (out= 0)(stored 0%)

adding:org/apache/hadoop/examples/WordCount$TokenizerMapper.class(in = 1790) (out=765)(deflated 57%)

adding:org/apache/hadoop/examples/WordCount$IntSumReducer.class(in = 1793) (out=746)(deflated 58%)

adding:org/apache/hadoop/examples/WordCount.class(in = 1911) (out= 996)(deflated 47%)

[plain]?view plaincopy

[hadoop@localhost~]$?ls??

classes? Desktop?hadoop-0.20.203.0?wordcount.jar? WordCount.java

到此為止，java文件的編譯工作已經(jīng)完成

現(xiàn)在為測試工作準(zhǔn)備需要的測試文件，創(chuàng)建2個文件file01、file02，內(nèi)容如下

[plain]?view plaincopy

[hadoop@localhost~]$?ls??

classes? Desktop?file01? file02? hadoop-0.20.203.0? wordcount.jar?WordCount.java

[plain]?view plaincopy

[hadoop@localhost~]$?cat?file01??

Hello World Bye World

[plain]?view plaincopy

[hadoop@localhost~]$?cat?file02??

Hello Hadoop Goodbye Hadoop

啟動hadoop，在hadoop中創(chuàng)建input文件夾

[plain]?view plaincopy

[hadoop@localhost~]$?hadoop-0.20.203.0/bin/hadoop?dfs?-ls??

??

[hadoop@localhost~]$?hadoop-0.20.203.0/bin/hadoop?dfs?-mkdir?input??

??

[hadoop@localhost~]$?hadoop-0.20.203.0/bin/hadoop?dfs?-ls??

Found 1 items

drwxr-xr-x?? - hadoop supergroup????????? 0 2011-11-23 05:20 /user/hadoop/input

把file01、file02上傳input中

[plain]?view plaincopy

[hadoop@localhost~]$?hadoop-0.20.203.0/bin/hadoop?fs?-put?file01?input??

??

[hadoop@localhost~]$?hadoop-0.20.203.0/bin/hadoop?fs?-put?file02?input??

??

[hadoop@localhost~]$?hadoop-0.20.203.0/bin/hadoop?fs?-ls?input??

Found 2 items

-rw-r--r--?? 1 hadoop supergroup???????? 22 2011-11-23 05:22/user/hadoop/input/file01

-rw-r--r--?? 1 hadoop supergroup???????? 28 2011-11-23 05:22/user/hadoop/input/file02

運(yùn)行及輸出過程如下

[plain]?view plaincopy

[hadoop@localhost~]$?hadoop-0.20.203.0/bin/hadoop?jar?wordcount.jar?org.apache.hadoop.examples.WordCount?input?output??

11/11/23 05:25:11INFO input.FileInputFormat: Total input paths to process : 2

11/11/23 05:25:12INFO mapred.JobClient: Running job: job_201111230519_0001

11/11/23 05:25:13INFO mapred.JobClient:? map 0% reduce 0%

11/11/23 05:25:45INFO mapred.JobClient:? map 50% reduce 0%

11/11/23 05:25:48INFO mapred.JobClient:? map 100% reduce0%

11/11/23 05:26:03INFO mapred.JobClient:? map 100% reduce 100%

11/11/23 05:26:07INFO mapred.JobClient: Job complete: job_201111230519_0001

11/11/23 05:26:07INFO mapred.JobClient: Counters: 25

11/11/23 05:26:07INFO mapred.JobClient:?? Job Counters

11/11/23 05:26:07INFO mapred.JobClient:???? Launchedreduce tasks=1

11/11/23 05:26:07INFO mapred.JobClient:????SLOTS_MILLIS_MAPS=48562

11/11/23 05:26:07INFO mapred.JobClient:???? Total timespent by all reduces waiting after reserving slots (ms)=0

11/11/23 05:26:07INFO mapred.JobClient:???? Total timespent by all maps waiting after reserving slots (ms)=0

11/11/23 05:26:07INFO mapred.JobClient:???? Launched maptasks=2

11/11/23 05:26:07INFO mapred.JobClient:???? Data-local maptasks=2

11/11/23 05:26:07INFO mapred.JobClient:????SLOTS_MILLIS_REDUCES=16678

11/11/23 05:26:07INFO mapred.JobClient:?? File OutputFormat Counters

11/11/23 05:26:07INFO mapred.JobClient:???? BytesWritten=41

11/11/23 05:26:07INFO mapred.JobClient:??FileSystemCounters

11/11/23 05:26:07INFO mapred.JobClient:????FILE_BYTES_READ=79

11/11/23 05:26:07INFO mapred.JobClient:????HDFS_BYTES_READ=272

11/11/23 05:26:07INFO mapred.JobClient:????FILE_BYTES_WRITTEN=63583

11/11/23 05:26:07INFO mapred.JobClient:????HDFS_BYTES_WRITTEN=41

11/11/23 05:26:07INFO mapred.JobClient:?? File Input FormatCounters

11/11/23 05:26:07INFO mapred.JobClient:???? Bytes Read=50

11/11/23 05:26:07INFO mapred.JobClient:?? Map-ReduceFramework

11/11/23 05:26:07INFO mapred.JobClient:???? Reduce inputgroups=5

11/11/23 05:26:07INFO mapred.JobClient:???? Map outputmaterialized bytes=85

11/11/23 05:26:07INFO mapred.JobClient:???? Combine outputrecords=6

11/11/23 05:26:07INFO mapred.JobClient:???? Map inputrecords=2

11/11/23 05:26:07INFO mapred.JobClient:???? Reduce shufflebytes=85

11/11/23 05:26:07INFO mapred.JobClient:???? Reduce outputrecords=5

11/11/23 05:26:07INFO mapred.JobClient:???? SpilledRecords=12

11/11/23 05:26:07INFO mapred.JobClient:???? Map outputbytes=82

11/11/23 05:26:07INFO mapred.JobClient:???? Combine inputrecords=8

11/11/23 05:26:07INFO mapred.JobClient:???? Map outputrecords=8

11/11/23 05:26:07INFO mapred.JobClient:????SPLIT_RAW_BYTES=222

11/11/23 05:26:07INFO mapred.JobClient:???? Reduce inputrecords=6

進(jìn)行結(jié)果的查看

[plain]?view plaincopy

[hadoop@localhost~]$?hadoop-0.20.203.0/bin/hadoop?fs?-ls???

Found 2 items

drwxr-xr-x?? - hadoop supergroup????????? 0 2011-11-23 05:22 /user/hadoop/input

drwxr-xr-x?? - hadoop supergroup????????? 0 2011-11-23 05:26/user/hadoop/output

發(fā)現(xiàn)在hadoop中多了一個output文件夾，查看output中的文件信息

[plain]?view plaincopy

[hadoop@localhost~]$?hadoop-0.20.203.0/bin/hadoop?fs?-ls?output??

Found 3 items

-rw-r--r--?? 1 hadoop supergroup????????? 0 2011-11-23 05:26/user/hadoop/output/_SUCCESS

drwxr-xr-x?? - hadoop supergroup????????? 0 2011-11-23 05:25/user/hadoop/output/_logs

-rw-r--r--?? 1 hadoop supergroup???????? 41 2011-11-23 05:25/user/hadoop/output/part-r-00000

對WordCount的運(yùn)行結(jié)果進(jìn)行查看

[plain]?view plaincopy

[hadoop@localhost~]$?hadoop-0.20.203.0/bin/hadoop?fs?-cat?output/part-r-00000??

Bye???? 1

Goodbye 1

Hadoop? 2

Hello?? 2

World?? 2

至此，hadoop下的WordCount實(shí)例運(yùn)行結(jié)束，如果還想重新運(yùn)行一遍，這需把hadoop下的output文件夾刪除，因為hadoop為了保證結(jié)果的正確性，存在輸出的文件夾的話，就會報異常，異常如下

[plain]?view plaincopy

[hadoop@localhost~]$?hadoop-0.20.203.0/bin/hadoop?jar?wordcount.jar?org.apache.hadoop.examples.WordCount?input?output??

11/11/23 05:33:05INFO mapred.JobClient: Cleaning up the staging areahdfs://localhost:9000/tmp/hadoop-hadoop/mapred/staging/hadoop/.staging/job_201111230519_0002

Exception inthread "main" org.apache.hadoop.mapred.FileAlreadyExistsException:Output directory output already exists

??????? atorg.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:134)

???? ???atorg.apache.hadoop.mapred.JobClient$2.run(JobClient.java:830)

??????? atorg.apache.hadoop.mapred.JobClient$2.run(JobClient.java:791)

??????? atjava.security.AccessController.doPrivileged(Native Method)

??????? at javax.security.auth.Subject.doAs(Subject.java:415)

??????? atorg.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)

??????? atorg.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:791)

??????? atorg.apache.hadoop.mapreduce.Job.submit(Job.java:465)

??????? atorg.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:494)

??????? atorg.apache.hadoop.examples.WordCount.main(WordCount.java:67)

??????? atsun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

??????? at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

??????? atsun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

??????? atjava.lang.reflect.Method.invoke(Method.java:601)

??????? at org.apache.hadoop.util.RunJar.main(RunJar.java:156)

刪除output文件夾的操作為

[plain]?view plaincopy

[hadoop@localhost~]$?hadoop-0.20.203.0/bin/hadoop?dfs?-rmr?output??

Deletedhdfs://localhost:9000/user/hadoop/output

《新程序員》：云原生和全面數(shù)字化實(shí)踐50位技術(shù)專家共同創(chuàng)作，文字、視頻、音頻交互閱讀

總結(jié)

以上是生活随笔為你收集整理的Hadoop示例程序手动编译的全部內(nèi)容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網(wǎng)站內(nèi)容還不錯，歡迎將生活随笔推薦給好友。