實驗 目的 要求	? 目的： 了解HDFS文件系統(tǒng)； 掌握HDFS的架構及核心組件的職能； 掌握HDFS數(shù)據(jù)的讀寫操作； HDFS常用操作（Shell。Java API） 了解Hadoop2.0中HDFS相關的新特性
? 實 驗 環(huán) 境 ?	Java jdk 1.8； apache-maven-3.6.0； Myeclipse C10； Hadoop集群；

?

練習內(nèi)容

任務一：HDFS API連接測試；

1、Test類測試連接；

關鍵代碼：

package org.gy.myhadoop.mr;import org.apache.hadoop.conf.Configuration;import org.apache.hadoop.fs.FileSystem;import org.apache.hadoop.fs.Path;import org.apache.hadoop.io.IOUtils;import org.junit.Before;import java.io.FileOutputStream;import java.io.IOException;import java.io.InputStream;import java.io.OutputStream;import java.net.URI;import java.net.URISyntaxException;/**** @author yanxukun* @date 2019年3月7日 上午11:46:23*/public class Test {@Beforepublic static void main(String[] args )throws Exception{System.out.println("Hello World!");Configuration conf = new Configuration();conf.set("fs.defaultFS","hdfs://192.168.10.111:9000");FileSystem fs = null;fs = FileSystem.get(conf);fs.mkdirs(new Path("hdfs://192.168.10.111:9000/user/input/test2"));}}

運行結果：

?

任務二：單元測試的setUp和tearDown方法

關鍵代碼：

package org.gy.myhadoop.mr;import org.apache.hadoop.conf.Configuration;import org.apache.hadoop.fs.FileSystem;import org.apache.hadoop.fs.Path;import org.apache.hadoop.io.IOUtils;import org.apache.hadoop.conf.Configuration;import org.apache.hadoop.fs.*;import org.junit.Before;import org.junit.After;import org.junit.Test;public class HDFSApp {public static final String HDFS_PATH = "hdfs://192.168.10.111:9000";public static Configuration configuration = null;public static FileSystem fileSystem = null;@Beforepublic void setUp() throws Exception {System.out.println("HDFSApp.setUp()");configuration = new Configuration();configuration.set("fs.defaultFS", "hdfs://192.168.10.111:9000");fileSystem = FileSystem.get(configuration);}@Afterpublic void tearDown() throws Exception{fileSystem = null;configuration = null;System.out.println("HDFSApp.tearDown()");}@Testpublic void rename() throws Exception{Path oldPath = new Path("hdfs://192.168.10.111:9000/user/test/a.txt");Path newPath = new Path("hdfs://192.168.10.111:9000/user/test/b.txt");System.out.println(fileSystem.rename(oldPath, newPath));}@Testpublic void copyFromLocalFile() throws Exception{Path src = new Path("C:/luke/hello.txt");Path dist = new Path("hdfs://192.168.10.111:9000/user/test/");fileSystem.copyFromLocalFile(src, dist);}@Testpublic void listFiles() throws Exception{FileStatus[] listStatus = fileSystem.listStatus(new Path("hdfs://192.168.2.100:9000/user/test"));for (FileStatus fileStatus : listStatus) {String isDir = fileStatus.isDirectory() ? "文件夾" : "文件";String permission = fileStatus.getPermission().toString();short replication = fileStatus.getReplication();long len = fileStatus.getLen();String path = fileStatus.getPath().toString();System.out.println(isDir+"\t"+permission+"\t"+replication+"\t"+len+"\t"+path);}}}

運行結果：

?

任務三：SequenceFile的基本操作

1、SequenceFile文件的寫操作；

關鍵代碼：

package org.gy.myhadoop.mr;import java.io.IOException;import java.text.SimpleDateFormat;import java.util.Date;import org.apache.hadoop.conf.Configuration;import org.apache.hadoop.fs.FileSystem;import org.apache.hadoop.fs.Path;import org.apache.hadoop.io.IOUtils;import org.apache.hadoop.io.IntWritable;import org.apache.hadoop.io.SequenceFile;import org.apache.hadoop.io.Text;public class SequenceFileWriter {private static Configuration configuration = new Configuration();private static String url = "hdfs://192.168.10.111:9000";private static String[] data = {"a,b,c,d,e,f,g","e,f,g,h,i,j,k","l,m,n,o,p,q,r,s","t,u,v,w,x,y,z"};public static void main(String[] args) throws Exception {FileSystem fs = FileSystem.get(configuration);Path outputPath = new Path("MySequenceFile.seq");IntWritable key = new IntWritable();Text value = new Text();SequenceFile.Writer writer = SequenceFile.createWriter(fs, configuration, outputPath, IntWritable.class, Text.class);for (int i=0;i<10;i++) {key.set(10-i);value.set(data[i%data.length]);writer.append(key, value);}IOUtils.closeStream(writer);Date day = new Date();SimpleDateFormat df = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss");System.out.println("計算機161燕旭坤在"+df.format(day)+"完成");}}

運行結果：

2、SequenceFile讀操作；

關鍵代碼：

package org.gy.myhadoop.mr;import java.io.IOException;import org.apache.hadoop.conf.Configuration;import org.apache.hadoop.fs.FileSystem;import org.apache.hadoop.fs.Path;import org.apache.hadoop.io.IOUtils;import org.apache.hadoop.io.IntWritable;import org.apache.hadoop.io.SequenceFile;import org.apache.hadoop.io.Text;import org.apache.hadoop.io.Writable;import org.apache.hadoop.util.ReflectionUtils;public class SequenceFileReader{private static Configuration configuration = new Configuration();private static String url = "hdfs://192.168.10.111:9000";public static void main(String[] args) throws Exception{FileSystem fs = FileSystem.get(configuration);Path inputPath = new Path("MySequenceFile.seq");SequenceFile.Reader reader = new SequenceFile.Reader(fs,inputPath, configuration);Writable keyClass = (Writable) ReflectionUtils.newInstance(reader.getKeyClass(), configuration);Writable valueClass = (Writable) ReflectionUtils.newInstance(reader.getValueClass(), configuration);while(reader.next(keyClass,valueClass)){System.out.println("key:"+keyClass);System.out.println("value:"+valueClass);System.out.println("position:"+reader.getPosition());}IOUtils.closeStream(reader);}}

?

運行結果：

任務四：MapFile的基本操作

1、MapFile的寫操作；

關鍵代碼：

package org.gy.myhadoop.mr;import java.io.IOException;import java.net.URI;import java.text.SimpleDateFormat;import java.util.Date;import org.apache.hadoop.conf.Configuration;import org.apache.hadoop.fs.FileSystem;import org.apache.hadoop.fs.Path;import org.apache.hadoop.io.IOUtils;import org.apache.hadoop.io.MapFile;import org.apache.hadoop.io.Text;public class MapFileWriter {static Configuration configuration = new Configuration();private static String url = "hdfs://192.168.10.111:9000";public static void main(String[] args) throws Exception {FileSystem fs = FileSystem.get(URI.create(url),configuration);Path outPath = new Path("MyMapFile.map");Text key = new Text();key.set("mymapkey");Text value = new Text();value.set("mymapvalue");MapFile.Writer writer = new MapFile.Writer(configuration, fs,outPath.toString(),Text.class,Text.class);writer.append(key, value);IOUtils.closeStream(writer);Date day = new Date();SimpleDateFormat df = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss");System.out.println("計算機161燕旭坤在"+df.format(day)+"完成");}}

運行結果：

2、MapFile的讀操作；

關鍵代碼：

package org.gy.myhadoop.mr;import java.io.IOException;import java.net.URI;import org.apache.hadoop.conf.Configuration;import org.apache.hadoop.fs.FileSystem;import org.apache.hadoop.fs.Path;import org.apache.hadoop.io.IOUtils;import org.apache.hadoop.io.MapFile;import org.apache.hadoop.io.Text;import org.apache.hadoop.io.Writable;import org.apache.hadoop.io.WritableComparable;import org.apache.hadoop.util.ReflectionUtils;public class MapFileReader {static Configuration configuration = new Configuration();private static String url = "hdfs://192.168.10.111:9000";public static void main(String[] args) throws Exception {FileSystem fs = FileSystem.get(URI.create(url),configuration);Path inPath = new Path("MyMapFile.map");MapFile.Reader reader? = new MapFile.Reader(fs,inPath.toString(),configuration);Writable keyclass = (Writable) ReflectionUtils.newInstance(reader.getKeyClass(), configuration);Writable valueclass = (Writable) ReflectionUtils.newInstance(reader.getValueClass(), configuration);while(reader.next((WritableComparable) keyclass,valueclass)) {System.out.println(keyclass);System.out.println(valueclass);}IOUtils.closeStream(reader);}}

運行結果：

?

出現(xiàn)的問題與解決方案

排錯一：JavaAPI操作HDFS文件系統(tǒng)報錯，Connection refused

錯誤：在WIN10系統(tǒng)下使用IDEA操作部HDFS系統(tǒng)時報錯java.net.ConnectException:Connection refused:no，我們要首先檢查以下設置是否正確；

排錯思路：

1.虛擬機的防火墻需要關閉

2.虛擬機和物理機需要互相ping通。

3.虛擬機HDFS的Hadoop目錄下的etc/hadoop/core-site.xml配置正確

4.物理機C:\Windows\System32\drivers\etc\hosts需要配置虛擬機ip和主機名的映射

在以上幾步均正確的情況下，依然報錯，那么檢查下虛擬機的主機名是不是localhost（我當時把localhost改成hadoop之后問題解決），如果是的話，建議換成別的

?

排錯二：

錯誤：ava.lang.IllegalArgumentException: Wrong FS: hdfs://master:9000/test.txt, expected: file:///? ?at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:648)

原因： 默認讀取file:///..., 其中的core-site.xml文件，我們沒有配置，所以得

解決：在工程下創(chuàng)建core-site.xml，將虛擬機master的core-site.xml復制到工程 下 的 core- site.xml.

?

排錯三：

錯誤：org.apache.hadoop.hdfs.protocol.DSQuotaExceededException:?

?The DiskSpace quota of /spaceQuota is exceeded: quota = 402653184 B = 384 MB?but diskspace consumed = 402653220 B = 384.00 MB

原因：文件的空間配額做出了限制，導致不能進行寫操作

解決：2個

? $> hdfs dfsadmin -setSpaceQuota? 384p /spaceQuota? //增大空間配額

? $>hadoop dfsadmin -clrSpaceQuota /user/hadoop? ? ? ? //清除空間配額的限制