使用版本是0.19.2,據(jù)說0.20以后,MultipleOutputFormat不好使,不知道真假

api可以參考

http://hadoop.apache.org/common/docs/r0.19.2/api/

但是說老實話,光看api有的時候有點混亂,每個函數(shù)到底影響些啥呢?

protected ?K	generateActualKey(K?key, V?value) ??????????Generate the actual key from the given key/value.
protected ?V	generateActualValue(K?key, V?value) ??????????Generate the actual value from the given key and value.
protected ?String	generateFileNameForKeyValue(K?key, V?value, String?name) ??????????Generate the file output file name based on the given key and the leaf file name.
protected ?String	generateLeafFileName(String?name) ??????????Generate the leaf name for the output file name.
protected abstract ?RecordWriter<K,V>	getBaseRecordWriter(FileSystem?fs, JobConf?job, String?name, Progressable?arg3) ???????????
protected ?String	getInputFileBasedOutputFileName(JobConf?job, String?name) ??????????Generate the outfile name based on a given anme and the input file name.
?RecordWriter<K,V>	getRecordWriter(FileSystem?fs, JobConf?job, String?name, Progressable?arg3) ??????????Create a composite record writer that can write key/value data to different output files

?

現(xiàn)在簡單介紹了下調(diào)用的過程

ReduceTask.java文件中

?1?public?void?run(JobConf?job,?final?TaskUmbilicalProtocol?umbilical)?throws?IOException
?2?{
?3?..........
?4?
?5?String?finalName?=?getOutputName(getPartition());//return?"part-"?+?NUMBER_FORMAT.format(partition);依據(jù)taskid產(chǎn)生諸如part-00000這樣的文件名
?6?
?7?FileSystem?fs?=?FileSystem.get(job);
?8?
?9?final?RecordWriter?out?=?job.getOutputFormat().getRecordWriter(fs,?job,?finalName,?reporter);//finalName=part-00000
10?
11?.............
12?}

?

?在MultipleOutputFormat.java里面,請注意這些個函數(shù)的調(diào)用順序

?

public?RecordWriter<K,?V>?getRecordWriter(FileSystem?fs,?JobConf?job,?String?name,?Progressable?arg3) throws?IOException
????{
????????final?FileSystem?myFS?=?fs;
????????final?String?myName?=?generateLeafFileName(name);//在這里可以硬性的指定文件名名稱
????????final?JobConf?myJob?=?job;
????????final?Progressable?myProgressable?=?arg3;

????????return?new?RecordWriter<K,?V>()?{
????????????//?a?cache?storing?the?record?writers?for?different?output?files.
????????????TreeMap<String,?RecordWriter<K,?V>>?recordWriters?=?new?TreeMap<String,?RecordWriter<K,?V>>();

????????????public?void?write(K?key,?V?value)?throws?IOException
????????????{
????????????????//?get?the?file?name?based?on?the?key
????????????????String?keyBasedPath?=?generateFileNameForKeyValue(key,?value,?myName);//一般依據(jù)key來決定文件名的時候 就在這個函數(shù)

????????????????//?get?the?file?name?based?on?the?input?file?name
????????????????String?finalPath?=?getInputFileBasedOutputFileName(myJob,?keyBasedPath);//如果想依據(jù)jobconf配置來確定名稱的話 就在這個函數(shù)里實現(xiàn)? finalPath?就是最終的文件名

????????????????//?get?the?actual?key
????????????????K?actualKey?=?generateActualKey(key,?value);
????????????????V?actualValue?=?generateActualValue(key,?value);

????????????????RecordWriter<K,?V>?rw?=?this.recordWriters.get(finalPath);
????????????????if?(rw?==?null)
????????????????{
????????????????????//?if?we?don't?have?the?record?writer?yet?for?the?final?path,?create one and?add?it?to?the?cache
????????????????????rw?=?getBaseRecordWriter(myFS,?myJob,?finalPath,?myProgressable);//必須自己實現(xiàn)的
????????????????????this.recordWriters.put(finalPath,?rw);
????????????????}
????????????????rw.write(actualKey,?actualValue);//
????????????};
....... };
????}

?

?上述函數(shù),除了getInputFileBasedOutputFileName,其他的紅色函數(shù)基本上都只是簡單的返回輸入值.

轉(zhuǎn)載于:https://www.cnblogs.com/xuxm2007/archive/2012/02/23/2365332.html

總結(jié)

以上是生活随笔為你收集整理的关于MultipleOutputFormat若干小记的全部內(nèi)容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網(wǎng)站內(nèi)容還不錯，歡迎將生活随笔推薦給好友。