1. hdfs定義

HDFS is the primary distributed storage used by Hadoop applications. A HDFS cluster primarily consists of a NameNode that manages the file system metadata and DataNodes that store the actual data.

2. hdfs架構

3. hdfs實例

? 作為文件系統，文件的讀寫才是核心：

/*** Licensed to the Apache Software Foundation (ASF) under one* or more contributor license agreements. See the NOTICE file* distributed with this work for additional information* regarding copyright ownership. The ASF licenses this file* to you under the Apache License, Version 2.0 (the* "License"); you may not use this file except in compliance* with the License. You may obtain a copy of the License at** http://www.apache.org/licenses/LICENSE-2.0** Unless required by applicable law or agreed to in writing, software* distributed under the License is distributed on an "AS IS" BASIS,* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.* See the License for the specific language governing permissions and* limitations under the License.*/import java.io.File; import java.io.IOException;import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.FileSystem; import org.apache.hadoop.fs.FSDataInputStream; import org.apache.hadoop.fs.FSDataOutputStream; import org.apache.hadoop.fs.Path;public class HadoopDFSFileReadWrite {static void usage () {System.out.println("Usage : HadoopDFSFileReadWrite <inputfile> <output file>");System.exit(1);}static void printAndExit(String str) {System.err.println(str);System.exit(1);}public static void main (String[] argv) throws IOException {Configuration conf = new Configuration();FileSystem fs = FileSystem.get(conf);if (argv.length != 2)usage();// Hadoop DFS deals with PathPath inFile = new Path(argv[0]);Path outFile = new Path(argv[1]);// Check if input/output are validif (!fs.exists(inFile))printAndExit("Input file not found");if (!fs.isFile(inFile))printAndExit("Input should be a file");if (fs.exists(outFile))printAndExit("Output already exists");// Read from and write to new fileFSDataInputStream in = fs.open(inFile);FSDataOutputStream out = fs.create(outFile);byte buffer[] = new byte[256];try {int bytesRead = 0;while ((bytesRead = in.read(buffer)) > 0) {out.write(buffer, 0, bytesRead);}} catch (IOException e) {System.out.println("Error while copying file");} finally {in.close();out.close();}} }

上述示例，將一個文件的內容復制到另一個文件中，具體步驟如下：

第一步：創建一個文件系統實例，給該實例傳遞新的配置。

Configuration conf = new Configuration(); FileSystem fs = FileSystem.get(conf);

第二步：獲取文件路徑

// Hadoop DFS deals with PathPath inFile = new Path(argv[0]);Path outFile = new Path(argv[1]);// Check if input/output are validif (!fs.exists(inFile))printAndExit("Input file not found");if (!fs.isFile(inFile))printAndExit("Input should be a file");if (fs.exists(outFile))printAndExit("Output already exists");

第三步：打開文件輸入輸出流，將輸入流寫到輸出流中：

// Read from and write to new fileFSDataInputStream in = fs.open(inFile);FSDataOutputStream out = fs.create(outFile);byte buffer[] = new byte[256];try {int bytesRead = 0;while ((bytesRead = in.read(buffer)) > 0) {out.write(buffer, 0, bytesRead);}} catch (IOException e) {System.out.println("Error while copying file");} finally {in.close();out.close();}

上面文件讀寫功能涉及到了文件系統FileSystem、配置文件Configuration、輸入流/輸出流FSDataInputStream/FSDataOutputStream

?4.?基本概念分析

? ? 4.1 文件系統

　　文件系統的層次結構如下所示：

　　文件系統有兩個重要的分支，一個是分布式文件系統，另一個是“本地”（映射到本地連接的磁盤）文件系統，本地磁盤適用于比較少的hadoop實例和測試。絕大部分情況下使用分布式文件系統，hadoop 分布式文件系統使用多個機器的系統，但對用戶來說只有一個磁盤。它的容錯性和大容量性使它非常有用。

　　4.2 配置文件

　　配置文件的層次結構如下：

我們關注的是HdfsConfiguration，其涉及到的配置文件有hdfs-default.xml和hdfs-site.xml：

static {addDeprecatedKeys();// adds the default resourcesConfiguration.addDefaultResource("hdfs-default.xml");Configuration.addDefaultResource("hdfs-site.xml");}

?

　　4.3 輸入/輸出流

　　　　輸入/輸出流和文件系統相對應，先看一下輸入流：

　　

其中，HdfsDataInputStream是FSDataInputStream的實現，其構造函數為：

public HdfsDataInputStream(DFSInputStream in) throws IOException {super(in);} DFSInputStream層次結構如下圖所示：

　　在了解一下輸出流：

　　

其中，重點是HdfsDataOutputStream，其構造函數為：

public HdfsDataOutputStream(DFSOutputStream out, FileSystem.Statistics stats,long startPosition) throws IOException {super(out, stats, startPosition);} DFSOutputStream 的層次結構為：

?

參考文獻：

【1】http://hadoop.apache.org/docs/r2.6.0/hadoop-project-dist/hadoop-hdfs/HdfsUserGuide.html

【2】http://hadoop.apache.org/docs/r2.6.0/hadoop-project-dist/hadoop-hdfs/HdfsDesign.html

【3】http://wiki.apache.org/hadoop/HadoopDfsReadWriteExample

【4】http://blog.csdn.net/gaoxingnengjisuan/article/details/11177049

轉載于:https://www.cnblogs.com/davidwang456/p/4772728.html

總結

以上是生活随笔為你收集整理的hdfs源码分析第一弹的全部內容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網站內容還不錯，歡迎將生活随笔推薦給好友。