2019獨角獸企業重金招聘Python工程師標準>>>

? ? ? 如何在eclipse或myeclipse調試mapreduce程序，這個可能是初學mr程序者碰到的一個難題

在hadoop1.2.1后，在下載的源代碼中找不到hadoop-eclipse-plugin相關的jar或源代碼。

其實hadoop目前使用maven進行源代碼的管理與調試，可以參考文獻：

http://blog.cloudera.com/blog/2012/08/developing-cdh-applications-with-maven-and-eclipse/

A sample POM for setting up a basic Maven project for CDH application development

https://gist.github.com/jnatkins/3517129

注意：CDH是hadoop的封裝版本，很穩定，并且更新也很快。

如果需要在eclipse下編寫MR程序并進行調試，需要以下前提條件：

1：安裝maven，建議安裝maven3.0.4或上以版本

2：使用eclipse較新的版本，如Kepler Service Release 1

3：在eclipse上安裝m2eclipse插件進行maven項目的管理

4：創建maven項目，并將pom.xml替換為https://gist.github.com/jnatkins/3517129項目中的pom.xml或下面的pom.xml（本人使用的，可以開發MR程序，操作HDFS等）

5：如果出現權限問題（如在windows下調試），可以將Administrator的用戶名修改為hdfs即可。

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"><modelVersion>4.0.0</modelVersion><groupId>com.cdh</groupId><artifactId>cdh-test</artifactId><version>SNAPSHOT-1.0.0</version><packaging>jar</packaging><name>cdh-test</name><url>http://maven.apache.org</url><properties><hadoop.version>2.0.0-mr1-cdh4.4.0</hadoop.version><hbase.version>0.94.6-cdh4.4.0</hbase.version><project.build.sourceEncoding>utf-8</project.build.sourceEncoding><maven.compiler.encoding>utf-8</maven.compiler.encoding></properties><build><pluginManagement><plugins><plugin><groupId>org.apache.maven.plugins</groupId><artifactId>maven-compiler-plugin</artifactId><version>3.1</version><configuration><encoding>utf-8</encoding><source>1.6</source><target>1.6</target></configuration></plugin></plugins></pluginManagement><plugins><plugin><groupId>org.apache.maven.plugins</groupId><artifactId>maven-shade-plugin</artifactId><version>2.1</version><executions><execution><phase>package</phase><goals><goal>shade</goal></goals></execution></executions></plugin><plugin><groupId>org.apache.maven.plugins</groupId><artifactId>maven-eclipse-plugin</artifactId><version>2.9</version><configuration><buildOutputDirectory>eclipse-classes</buildOutputDirectory><downloadSources>true</downloadSources><downloadJavadocs>false</downloadJavadocs></configuration></plugin></plugins></build><dependencies><dependency><groupId>junit</groupId><artifactId>junit</artifactId><version>4.8.2</version><scope>test</scope></dependency><dependency><groupId>org.apache.hadoop</groupId><artifactId>hadoop-client</artifactId><version>${hadoop.version}</version><scope>provided</scope></dependency><dependency><groupId>org.apache.hadoop</groupId><artifactId>hadoop-mapreduce-client-core</artifactId><version>2.0.0-cdh4.4.0</version><exclusions><exclusion><artifactId>jersey-test-framework-grizzly2</artifactId><groupId>com.sun.jersey.jersey-test-framework</groupId></exclusion><exclusion><artifactId>netty</artifactId><groupId>org.jboss.netty</groupId></exclusion></exclusions><scope>provided</scope></dependency><dependency><groupId>org.apache.hbase</groupId><artifactId>hbase</artifactId><version>${hbase.version}</version><scope>provided</scope></dependency><dependency><groupId>com.hadoop.gplcompression</groupId><artifactId>hadoop-lzo-cdh4</artifactId><version>0.4.15-gplextras</version></dependency><dependency><groupId>commons-httpclient</groupId><artifactId>commons-httpclient</artifactId><version>3.1</version></dependency><dependency><groupId>org.hsqldb</groupId><artifactId>hsqldb</artifactId><version>2.2.9</version></dependency></dependencies><repositories><repository><id>cloudera</id><url>https://repository.cloudera.com/artifactory/cloudera-repos</url><releases><enabled>true</enabled></releases><snapshots><enabled>false</enabled></snapshots></repository></repositories> </project>

轉載于:https://my.oschina.net/cloudcoder/blog/179078

總結

以上是生活随笔為你收集整理的如何在eclipse调试mapreduce程序的全部內容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網站內容還不錯，歡迎將生活随笔推薦給好友。