CDH Hadoop集群虛擬機本地配置搭建筆記_part7_透過Cloudera Manager(6.3)

接續前面篇章

CDH Hadoop集群虛擬機本地配置搭建筆記

part1透過VMWare創建3台虛擬機(各節點的網路配置)

part2_配置yum local repo

part3_集群NTP時間同步配置(ntpd,ntpdate)

part4_安裝Java與MySQL配置_建立CDH預設資料庫

part5_安裝Cloudera Manager(6.3) Server跟Agent(關閉透明巨頁配置)

part6_透過Cloudera Manager(6.3)_安裝自訂服務ZooKeeper

回到首頁

http://192.168.80.81:7180/cmf/home

選中 HDFS

Assign Roles

這邊將

DataNode 選擇All Hosts

NameNode為Host1

SecondaryNameNode為Host2

其他NFS Gateway , HttpFS 可以先不用特別選，比較核心重要上述三種。

Review Changes

等待綠色勾全部顯示再Continue

把一些false alarm先給supress掉

Erasure Coding Policy Verification Test

Concerning : 9 DataNodes are required for the erasure coding policies: RS-6-3-1024k. The number of DataNodes is only 3.

Java Heap Size of NameNode in Bytes is recommended to be at least 1GB for every million HDFS blocks. Suggested minimum value: 4.0 GiB

File Browser頁面

NameNode WebUI

host:9870
http://192.168.80.81:9870/

Datanode Information

Browse Directory

Name Node High Availability(高可用性)配置的必要性

HDFS的metadata基本上都存紀錄於Name Node中

個別的Data Node並不知道自身存放的HDFS Block屬於或如何組成HDFS file。

若Name Node發生問題，則整個HDFS都將無法被讀取或寫入。

若Name Node的磁碟機有錯誤發生，導致HDFS metadata遺失的話，此時所有hdfs資料等同於全數遺失，因此Name Node的高可用性是HDFS十分重要的一環。

啟動高可靠(HA)

命名Nameservice Name

NameNode Hosts 選擇一個備份的host2

JournalNode Hosts 三個都選

We recommend that JournalNodes be hosted on machines of similar hardware specifications as the NameNodes. The hosts of NameNodes and the ResourceManager are generally good options. You must have a minimum of three and an odd number of JournalNodes.

JournalNode Edits Directory

要設置三個不然會跳警示卡住不能下一步

Determines where on the local file system the NameNode should store the name table (fsimage). For redundancy, enter a comma-delimited list of directories to replicate the name table in all of the directories. Typical values are /data/N/dfs/nn where N=1..3.