hdfs fsck命令查看HDFS文件对应的文件块信息(Block)和位置信息(Locations)
生活随笔
收集整理的這篇文章主要介紹了
hdfs fsck命令查看HDFS文件对应的文件块信息(Block)和位置信息(Locations)
小編覺得挺不錯的,現(xiàn)在分享給大家,幫大家做個參考.
關(guān)鍵字：hdfs fsck、block、locations
在HDFS中，提供了fsck命令，用于檢查HDFS上文件和目錄的健康狀態(tài)、獲取文件的block信息和位置信息等。
fsck命令必須由HDFS超級用戶來執(zhí)行，普通用戶無權(quán)限。
[hadoop@dev ~]$ hdfs fsck Usage: DFSck [-list-corruptfileblocks | [-move | -delete | -openforwrite] [-files [-blocks [-locations | -racks]]]] start checking from this path -move move corrupted files to /lost+found -delete delete corrupted files -files print out files being checked -openforwrite print out files opened for write -includeSnapshots include snapshot data if the given path indicates a snapshottable directory or there are snapshottable directories under it -list-corruptfileblocks print out list of missing blocks and files they belong to -blocks print out block report -locations print out locations for every block -racks print out network topology for data-node locations下面介紹每一個選項的含義及用法。?
查看文件中損壞的塊（-list-corruptfileblocks）
[hadoop@dev ~]$ hdfs fsck /hivedata/warehouse/liuxiaowen.db/lxw_product_names/ -list-corruptfileblocks The filesystem under path '/hivedata/warehouse/liuxiaowen.db/lxw_product_names/' has 0 CORRUPT files將損壞的文件移動至/lost+found目錄（-move）
[hadoop@dev ~]$ hdfs fsck /hivedata/warehouse/liuxiaowen.db/lxw_product_names/part-00168 -move FSCK started by hadoop (auth:SIMPLE) from /172.16.212.17 for path /hivedata/warehouse/liuxiaowen.db/lxw_product_names/part-00168 at Thu Aug 13 09:36:35 CST 2015 .Status: HEALTHY Total size: 13497058 B Total dirs: 0 Total files: 1 Total symlinks: 0 Total blocks (validated): 1 (avg. block size 13497058 B) Minimally replicated blocks: 1 (100.0 %) Over-replicated blocks: 0 (0.0 %) Under-replicated blocks: 0 (0.0 %) Mis-replicated blocks: 0 (0.0 %) Default replication factor: 2 Average block replication: 2.0 Corrupt blocks: 0 Missing replicas: 0 (0.0 %) Number of data-nodes: 15 Number of racks: 1 FSCK ended at Thu Aug 13 09:36:35 CST 2015 in 1 millisecondsThe filesystem under path '/hivedata/warehouse/liuxiaowen.db/lxw_product_names/part-00168' is HEALTHY?
刪除損壞的文件（-delete）
[hadoop@dev ~]$ hdfs fsck /hivedata/warehouse/liuxiaowen.db/lxw_product_names/part-00168 -delete FSCK started by hadoop (auth:SIMPLE) from /172.16.212.17 for path /hivedata/warehouse/liuxiaowen.db/lxw_product_names/part-00168 at Thu Aug 13 09:37:58 CST 2015 .Status: HEALTHY Total size: 13497058 B Total dirs: 0 Total files: 1 Total symlinks: 0 Total blocks (validated): 1 (avg. block size 13497058 B) Minimally replicated blocks: 1 (100.0 %) Over-replicated blocks: 0 (0.0 %) Under-replicated blocks: 0 (0.0 %) Mis-replicated blocks: 0 (0.0 %) Default replication factor: 2 Average block replication: 2.0 Corrupt blocks: 0 Missing replicas: 0 (0.0 %) Number of data-nodes: 15 Number of racks: 1 FSCK ended at Thu Aug 13 09:37:58 CST 2015 in 1 millisecondsThe filesystem under path '/hivedata/warehouse/liuxiaowen.db/lxw_product_names/part-00168' is HEALTHY?
檢查并列出所有文件狀態(tài)（-files）
[hadoop@dev ~]$ hdfs fsck /hivedata/warehouse/liuxiaowen.db/lxw_product_names/ -files FSCK started by hadoop (auth:SIMPLE) from /172.16.212.17 for path /hivedata/warehouse/liuxiaowen.db/lxw_product_names/ at Thu Aug 13 09:39:38 CST 2015 /hivedata/warehouse/liuxiaowen.db/lxw_product_names/ dir /hivedata/warehouse/liuxiaowen.db/lxw_product_names/_SUCCESS 0 bytes, 0 block(s): OK /hivedata/warehouse/liuxiaowen.db/lxw_product_names/part-00000 13583807 bytes, 1 block(s): OK /hivedata/warehouse/liuxiaowen.db/lxw_product_names/part-00001 13577427 bytes, 1 block(s): OK /hivedata/warehouse/liuxiaowen.db/lxw_product_names/part-00002 13588601 bytes, 1 block(s): OK /hivedata/warehouse/liuxiaowen.db/lxw_product_names/part-00003 13479213 bytes, 1 block(s): OK /hivedata/warehouse/liuxiaowen.db/lxw_product_names/part-00004 13497012 bytes, 1 block(s): OK /hivedata/warehouse/liuxiaowen.db/lxw_product_names/part-00005 13557451 bytes, 1 block(s): OK /hivedata/warehouse/liuxiaowen.db/lxw_product_names/part-00006 13580267 bytes, 1 block(s): OK /hivedata/warehouse/liuxiaowen.db/lxw_product_names/part-00007 13486035 bytes, 1 block(s): OK /hivedata/warehouse/liuxiaowen.db/lxw_product_names/part-00008 13481498 bytes, 1 block(s): OK ...檢查并打印正在被打開執(zhí)行寫操作的文件（-openforwrite）
[hadoop@dev ~]$ hdfs fsck /hivedata/warehouse/liuxiaowen.db/lxw_product_names/ -openforwrite FSCK started by hadoop (auth:SIMPLE) from /172.16.212.17 for path /hivedata/warehouse/liuxiaowen.db/lxw_product_names/ at Thu Aug 13 09:41:28 CST 2015 .................................................................................................... .................................................................................................... .Status: HEALTHY Total size: 2704782548 B Total dirs: 1 Total files: 201 Total symlinks: 0 Total blocks (validated): 200 (avg. block size 13523912 B) Minimally replicated blocks: 200 (100.0 %) Over-replicated blocks: 0 (0.0 %) Under-replicated blocks: 0 (0.0 %) Mis-replicated blocks: 0 (0.0 %) Default replication factor: 2 Average block replication: 2.0 Corrupt blocks: 0 Missing replicas: 0 (0.0 %) Number of data-nodes: 15 Number of racks: 1 FSCK ended at Thu Aug 13 09:41:28 CST 2015 in 10 millisecondsThe filesystem under path '/hivedata/warehouse/liuxiaowen.db/lxw_product_names/' is HEALTHY打印文件的Block報告（-blocks）
需要和-files一起使用。
[hadoop@dev ~]$ hdfs fsck /logs/site/2015-08-08/lxw1234.log -files -blocks FSCK started by hadoop (auth:SIMPLE) from /172.16.212.17 for path /logs/site/2015-08-08/lxw1234.log at Thu Aug 13 09:45:59 CST 2015 /logs/site/2015-08-08/lxw1234.log 7408754725 bytes, 56 block(s): OK 0. BP-1034052771-172.16.212.130-1405595752491:blk_1075892982_2152381 len=134217728 repl=2 1. BP-1034052771-172.16.212.130-1405595752491:blk_1075892983_2152382 len=134217728 repl=2 2. BP-1034052771-172.16.212.130-1405595752491:blk_1075892984_2152383 len=134217728 repl=2 3. BP-1034052771-172.16.212.130-1405595752491:blk_1075892985_2152384 len=134217728 repl=2 4. BP-1034052771-172.16.212.130-1405595752491:blk_1075892997_2152396 len=134217728 repl=2 5. BP-1034052771-172.16.212.130-1405595752491:blk_1075892998_2152397 len=134217728 repl=2 6. BP-1034052771-172.16.212.130-1405595752491:blk_1075892999_2152398 len=134217728 repl=2 7. BP-1034052771-172.16.212.130-1405595752491:blk_1075893000_2152399 len=134217728 repl=2 8. BP-1034052771-172.16.212.130-1405595752491:blk_1075893001_2152400 len=134217728 repl=2 9. BP-1034052771-172.16.212.130-1405595752491:blk_1075893002_2152401 len=134217728 repl=2 10. BP-1034052771-172.16.212.130-1405595752491:blk_1075893003_2152402 len=134217728 repl=2 11. BP-1034052771-172.16.212.130-1405595752491:blk_1075893004_2152403 len=134217728 repl=2 12. BP-1034052771-172.16.212.130-1405595752491:blk_1075893005_2152404 len=134217728 repl=2 13. BP-1034052771-172.16.212.130-1405595752491:blk_1075893006_2152405 len=134217728 repl=2 14. BP-1034052771-172.16.212.130-1405595752491:blk_1075893007_2152406 len=134217728 repl=2 ...其中，/logs/site/2015-08-08/lxw1234.log 7408754725 bytes, 56 block(s): 表示文件的總大小和block數(shù)；
0. BP-1034052771-172.16.212.130-1405595752491:blk_1075892982_2152381 len=134217728 repl=2
1. BP-1034052771-172.16.212.130-1405595752491:blk_1075892983_2152382 len=134217728 repl=2
2. BP-1034052771-172.16.212.130-1405595752491:blk_1075892984_2152383 len=134217728 repl=2
前面的0. 1. 2.代表該文件的block索引，56的文件塊，就從0-55;
BP-1034052771-172.16.212.130-1405595752491:blk_1075892982_2152381表示block id；
len=134217728 表示該文件塊大小；
repl=2 表示該文件塊副本數(shù)；
打印文件塊的位置信息（-locations）
需要和-files -blocks一起使用。
[hadoop@dev ~]$ hdfs fsck /logs/site/2015-08-08/lxw1234.log -files -blocks -locations FSCK started by hadoop (auth:SIMPLE) from /172.16.212.17 for path /logs/site/2015-08-08/lxw1234.log at Thu Aug 13 09:45:59 CST 2015 /logs/site/2015-08-08/lxw1234.log 7408754725 bytes, 56 block(s): OK 0. BP-1034052771-172.16.212.130-1405595752491:blk_1075892982_2152381 len=134217728 repl=2 [172.16.212.139:50010, 172.16.212.135:50010] 1. BP-1034052771-172.16.212.130-1405595752491:blk_1075892983_2152382 len=134217728 repl=2 [172.16.212.140:50010, 172.16.212.133:50010] 2. BP-1034052771-172.16.212.130-1405595752491:blk_1075892984_2152383 len=134217728 repl=2 [172.16.212.136:50010, 172.16.212.141:50010] 3. BP-1034052771-172.16.212.130-1405595752491:blk_1075892985_2152384 len=134217728 repl=2 [172.16.212.133:50010, 172.16.212.135:50010] 4. BP-1034052771-172.16.212.130-1405595752491:blk_1075892997_2152396 len=134217728 repl=2 [172.16.212.142:50010, 172.16.212.139:50010] 5. BP-1034052771-172.16.212.130-1405595752491:blk_1075892998_2152397 len=134217728 repl=2 [172.16.212.133:50010, 172.16.212.139:50010] 6. BP-1034052771-172.16.212.130-1405595752491:blk_1075892999_2152398 len=134217728 repl=2 [172.16.212.141:50010, 172.16.212.135:50010] 7. BP-1034052771-172.16.212.130-1405595752491:blk_1075893000_2152399 len=134217728 repl=2 [172.16.212.144:50010, 172.16.212.142:50010] 8. BP-1034052771-172.16.212.130-1405595752491:blk_1075893001_2152400 len=134217728 repl=2 [172.16.212.133:50010, 172.16.212.138:50010] 9. BP-1034052771-172.16.212.130-1405595752491:blk_1075893002_2152401 len=134217728 repl=2 [172.16.212.140:50010, 172.16.212.134:50010] ...和打印出的文件塊信息相比，多了一個文件塊的位置信息：[172.16.212.139:50010, 172.16.212.135:50010]
打印文件塊位置所在的機(jī)架信息（-racks）
[hadoop@dev ~]$ hdfs fsck /logs/site/2015-08-08/lxw1234.log -files -blocks -locations -racks FSCK started by hadoop (auth:SIMPLE) from /172.16.212.17 for path /logs/site/2015-08-08/lxw1234.log at Thu Aug 13 09:45:59 CST 2015 /logs/site/2015-08-08/lxw1234.log 7408754725 bytes, 56 block(s): OK 0. BP-1034052771-172.16.212.130-1405595752491:blk_1075892982_2152381 len=134217728 repl=2 [/default-rack/172.16.212.139:50010, /default-rack/172.16.212.135:50010] 1. BP-1034052771-172.16.212.130-1405595752491:blk_1075892983_2152382 len=134217728 repl=2 [/default-rack/172.16.212.140:50010, /default-rack/172.16.212.133:50010] 2. BP-1034052771-172.16.212.130-1405595752491:blk_1075892984_2152383 len=134217728 repl=2 [/default-rack/172.16.212.136:50010, /default-rack/172.16.212.141:50010] 3. BP-1034052771-172.16.212.130-1405595752491:blk_1075892985_2152384 len=134217728 repl=2 [/default-rack/172.16.212.133:50010, /default-rack/172.16.212.135:50010] 4. BP-1034052771-172.16.212.130-1405595752491:blk_1075892997_2152396 len=134217728 repl=2 [/default-rack/172.16.212.142:50010, /default-rack/172.16.212.139:50010] 5. BP-1034052771-172.16.212.130-1405595752491:blk_1075892998_2152397 len=134217728 repl=2 [/default-rack/172.16.212.133:50010, /default-rack/172.16.212.139:50010] ...和前面打印出的信息相比，多了機(jī)架信息：[/default-rack/172.16.212.139:50010, /default-rack/172.16.212.135:50010]
?
轉(zhuǎn)載自：http://lxw1234.com/archives/2015/08/452.htm
總結(jié)
以上是生活随笔為你收集整理的hdfs fsck命令查看HDFS文件对应的文件块信息(Block)和位置信息(Locations)的全部內(nèi)容，希望文章能夠幫你解決所遇到的問題。
- 上一篇： CDH kerberos 认证，安全认证
- 下一篇： Storm性能简单测试