2019獨角獸企業重金招聘Python工程師標準>>>

Hadoop的自定制數據類型
一般有兩個辦法，一種較為簡單的是針對值，另外一種更為完整的是對于鍵和值都適應的方法：
1、實現Writable接口：

/* DataInput and DataOutput 類是java.io的類 */
public interface Writable {
??? void readFields(DataInput in);
??? void write(DataOutput out);
}

下面是一個小例子：
public class Point3D implement Writable {
?public float x, y, z;
?public Point3D(float fx, float fy, float fz) {
??this.x = fx;
??this.y = fy;
??this.z = fz;
?}
?public Point3D() {
??this(0.0f, 0.0f, 0.0f);
?}
?public void readFields(DataInput in) throws IOException {
??x = in.readFloat();
??y = in.readFloat();
??z = in.readFloat();
?}
?public void write(DataOutput out) throws IOException {
??out.writeFloat(x);
??out.writeFloat(y);
??out.writeFloat(z);
?}
?public String toString() {
??return Float.toString(x) + ", "
???+ Float.toString(y) + ", "
???+ Float.toString(z);
?}
}

2、對于鍵來說，需要指定排序規則，對此，Java版Hadoop的辦法是實現WritableComparable這個泛型接口，WritableComparable，顧名思義了，一半是Writable，一半是Comparable

public interface WritableComparable<T> {
?public void readFields(DataInput in);
?public void write(DataOutput out);
?public int compareTo(T other);
}

這里的compareTo方法是默認的key排序

先給出下面的簡單例子，再做說明和擴展。
public class Point3D inplements WritableComparable {
?public float x, y, z;
?public Point3D(float fx, float fy, float fz) {
??this.x = fx;
??this.y = fy;
??this.z = fz;
?}
?public Point3D() {
??this(0.0f, 0.0f, 0.0f);
?}
?public void readFields(DataInput in) throws IOException {
??x = in.readFloat();
??y = in.readFloat();
??z = in.readFloat();
?}
?public void write(DataOutput out) throws IOException {
??out.writeFloat(x);
??out.writeFloat(y);
??out.writeFloat(z);
?}
?public String toString() {
??return Float.toString(x) + ", "
???+ Float.toString(y) + ", "
???+ Float.toString(z);
?}
?public float distanceFromOrigin() {
??return (float) Math.sqrt( x*x + y*y +z*z);
?}
?public int compareTo(Point3D other) {
??return Float.compareTo(
???distanceFromOrigin(),
???other.distanceFromOrigin());
?}
?public boolean equals(Object o) {
??if( !(o instanceof Point3D)) {
???return false;
??}
??Point3D other = (Point3D) o;
??return this.x == o.x
???&& this.y == o.y
???&& this.z == o.z;
?}
?/* 實現 hashCode() 方法很重要
? * Hadoop的Partitioners會用到這個方法，后面再說
? */
?public int hashCode() {
??return Float.floatToIntBits(x)
???^ Float.floatToIntBits(y)
???^ Float.floatToIntBits(z);
?}
}
如果要將對象寫入數據庫則還要繼承DBWritable接口

public interface WritableComparable<T> {
?public void write(PreparedStatement statement) throwsSQLException;
public void readFields(ResultSet resultSet) throws SQLException;}

下面寫個例子

public class LocationBean implements Writable, DBWritable {?
??? private String mobilenetworkcode;?
?
??? private String mobilecountrycode;?
?
??? private Integer cellid;?
?
??? private Integer locationareacode;?
?
??? private Integer baiduareaid;?
?
??? private Double lat;?
?
??? private Double lng;?
?
??? private Integer areaid;?
??? @Override?
??? public void write(PreparedStatement statement) throws SQLException {?
??????? int index = 1;???
??????? statement.setString(index++, this.getMobilenetworkcode());???
??????? statement.setString(index++, this.getMobilecountrycode());???
??????? statement.setInt(index++, this.getCellid());???
??????? statement.setInt(index++, this.getLocationareacode());?
??????? statement.setInt(index++, this.getBaiduareaid());?
??????? statement.setDouble(index++, this.getLat());?
??????? statement.setDouble(index++, this.getLng());?
??????? statement.setInt(index, this.getAreaid());?
??? }?
?
??? @Override?
??? public void readFields(ResultSet resultSet) throws SQLException {?
???????? this.mobilenetworkcode = resultSet.getString(1);?
???????? this.mobilecountrycode = resultSet.getString(2);?
???????? this.cellid = resultSet.getInt(3);?
???????? this.locationareacode = resultSet.getInt(4);?
???????? this.baiduareaid = resultSet.getInt(5);?
???????? this.lat = resultSet.getDouble(6);?
???????? this.lng = resultSet.getDouble(7);?
???????? this.areaid = resultSet.getInt(8);?
??? }?
?
??? @Override?
??? public void write(DataOutput out) throws IOException {?
??????? // TODO Auto-generated method stub?
?????????
??? }?
?
??? @Override?
??? public void readFields(DataInput in) throws IOException {?
?????????
?????????
??? }?
?
??? public String getMobilenetworkcode() {?
??????? return mobilenetworkcode;?
??? }?
?
??? public void setMobilenetworkcode(String mobilenetworkcode) {?
??????? this.mobilenetworkcode = mobilenetworkcode;?
??? }?
?
??? public String getMobilecountrycode() {?
??????? return mobilecountrycode;?
??? }?
?
??? public void setMobilecountrycode(String mobilecountrycode) {?
??????? this.mobilecountrycode = mobilecountrycode;?
??? }?
?
??? public Integer getCellid() {?
??????? return cellid;?
??? }?
?
??? public void setCellid(Integer cellid) {?
??????? this.cellid = cellid;?
??? }?
?
??? public Integer getLocationareacode() {?
??????? return locationareacode;?
??? }?
?
??? public void setLocationareacode(Integer locationareacode) {?
??????? this.locationareacode = locationareacode;?
??? }?
?
??? public Integer getBaiduareaid() {?
??????? return baiduareaid;?
??? }?
?
??? public void setBaiduareaid(Integer baiduareaid) {?
??????? this.baiduareaid = baiduareaid;?
??? }?
?
??? public Double getLat() {?
??????? return lat;?
??? }?
?
??? public void setLat(Double lat) {?
??????? this.lat = lat;?
??? }?
?
??? public Double getLng() {?
??????? return lng;?
??? }?
?
??? public void setLng(Double lng) {?
??????? this.lng = lng;?
??? }?
?
??? public Integer getAreaid() {?
??????? return areaid;?
??? }?
?
??? public void setAreaid(Integer areaid) {?
??????? this.areaid = areaid;?
??? }?
?
}?

轉載于:https://my.oschina.net/u/2000675/blog/654627

總結

以上是生活随笔為你收集整理的hadoop自定义key，value的全部內容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網站內容還不錯，歡迎將生活随笔推薦給好友。