當前位置：首頁 > 编程资源 > 编程问答 >内容正文

编程问答

Spark-1.4.0集群搭建

發布時間：2025/3/17 编程问答 28 豆豆

生活随笔收集整理的這篇文章主要介紹了 Spark-1.4.0集群搭建小編覺得挺不錯的,現在分享給大家,幫大家做個參考.

主要內容

Ubuntu 10.04 系統設置
ZooKeeper集群搭建
Hadoop-2.4.1集群搭建
Spark 1.4.0集群搭建

假設已經安裝好Ubuntu操作系統

Ubuntu 10.04設置

1.主機規劃

主機名IP地址進程號

SparkMaster	192.168.1.103	ResourceManager DataNode、NodeManager、JournalNode、QuorumPeerMain
SparkSlave01	192.168.1.101	ResourceManager DataNode、NodeManager、JournalNode、QuorumPeerMain NameNode、DFSZKFailoverController(zkfc)
SparkSlave02	192.168.1.102	DataNode、NodeManager、JournalNode、QuorumPeerMain NameNode、DFSZKFailoverController(zkfc)

**說明：
1.在hadoop2.0中通常由兩個NameNode組成，一個處于active狀態，另一個處于standby狀態。Active NameNode對外提供服務，而Standby NameNode則不對外提供服務，僅同步active namenode的狀態，以便能夠在它失敗時快速進行切換。
hadoop2.0官方提供了兩種HDFS HA的解決方案，一種是NFS，另一種是QJM。這里我們使用簡單的QJM。在該方案中，主備NameNode之間通過一組JournalNode同步元數據信息，一條數據只要成功寫入多數JournalNode即認為寫入成功。通常配置奇數個JournalNode
這里還配置了一個zookeeper集群，用于ZKFC（DFSZKFailoverController）故障轉移，當Active NameNode掛掉了，會自動切換Standby NameNode為standby狀態
2.hadoop-2.2.0中依然存在一個問題，就是ResourceManager只有一個，存在單點故障，hadoop-2.4.1解決了這個問題，有兩個ResourceManager，一個是Active，一個是Standby，狀態由zookeeper進行協調**

2. 修改主機名稱設置
利用vi /etc/hostname修改主機名稱

3. 修改主機IP地址
利用vi /etc/network/interfaces修改主要IP

主機/etc/network/interfaces文件內容

SparkMaster	auto loiface lo inet loopback auto eth0 iface eth0 inet static address 192.168.1.103 netmask 255.255.255.0 gateway 192.168.1.1
SparkSlave01	auto loiface lo inet loopback auto eth0 iface eth0 inet static address 192.168.1.101 netmask 255.255.255.0 gateway 192.168.1.1
SparkSlave02	auto loiface lo inet loopback auto eth0 iface eth0 inet static address 192.168.1.102 netmask 255.255.255.0 gateway 192.168.1.1

4. 修改域名解析服務器
由于需要聯網安裝OpenSSH等實現名密碼登錄，因此這邊需要配置對應的域名解析服務器

主機/etc/resolv.conf文件內容

SparkMaster	domain localdomain search localdomain nameserver 8.8.8.8
SparkSlave01	domain localdomain search localdomain nameserver 8.8.8.8
SparkSlave02	domain localdomain search localdomain nameserver 8.8.8.8

5.修改主機名與IP地址映射

主機/etc/resolv.conf文件內容

SparkMaster	127.0.0.1 SparkMaster localhost.localdomain localhost 192.168.1.101 SparkSlave01 192.168.1.102 SparkSlave02 192.168.1.103 SparkMaster ::1 localhost ip6-localhost ip6-loopback fe00::0 ip6-localnet ff00::0 ip6-mcastprefix ff02::1 ip6-allnodes ff02::2 ip6-allrouters ff02::3 ip6-allhosts
SparkSlave01	127.0.0.1 SparkSlave01 localhost.localdomain localhost 192.168.1.101 SparkSlave01 192.168.1.102 SparkSlave02 192.168.1.103 SparkMaster ::1 localhost ip6-localhost ip6-loopback fe00::0 ip6-localnet ff00::0 ip6-mcastprefix ff02::1 ip6-allnodes ff02::2 ip6-allrouters ff02::3 ip6-allhosts
SparkSlave02	127.0.0.1 SparkSlave02 localhost.localdomain localhost 192.168.1.101 SparkSlave01 192.168.1.102 SparkSlave02 192.168.1.103 SparkMaster ::1 localhost ip6-localhost ip6-loopback fe00::0 ip6-localnet ff00::0 ip6-mcastprefix ff02::1 ip6-allnodes ff02::2 ip6-allrouters ff02::3 ip6-allhosts

完成上述步驟后重新啟動機器
6.安裝SSH (三臺主機執行相同命令)

sudo apt-get install openssh-server
然后確認sshserver是否啟動了：
ps -e |grep ssh

7.設置無密碼登錄 (三臺主機執行相同命令)
執行命令：ssh-keygen -t rsa
執行完這個命令后，會生成兩個文件id_rsa（私鑰）、id_rsa.pub（公鑰）
將公鑰拷貝到要免登陸的機器上
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
或
ssh-copy-id -i SparkMaster
ssh-copy-id -i SparkSlave02
ssh-copy-id -i SparkSlave01

ZooKeeper集群搭建

本集群用的ZooKeeper版本是3.4.5，將/hadoopLearning/zookeeper-3.4.5/conf目錄下的zoo_sample.cfg文件名重命名為zoo.cfg
vi conf/zoo.cfg，在文件中填入以下內容：

# The number of milliseconds of each tick tickTime=2000 # The number of ticks that the initial # synchronization phase can take initLimit=10 # The number of ticks that can pass between # sending a request and getting an acknowledgement syncLimit=5 # the directory where the snapshot is stored. # do not use /tmp for storage, /tmp here is just # example sakes. # ZK文件存放目錄 dataDir=/hadoopLearning/zookeeper-3.4.5/zookeeper_data # the port at which the clients will connect clientPort=2181 # # Be sure to read the maintenance section of the # administrator guide before turning on autopurge. # #http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance # # The number of snapshots to retain in dataDir #autopurge.snapRetainCount=3 # Purge task interval in hours # Set to "0" to disable auto purge feature #autopurge.purgeInterval=1 server.1=SparkSlave01:2888:3888 server.2=SparkSlave02:2888:3888 server.3=SparkMaster:2888:3888

在/hadoopLearning/zookeeper-3.4.5/目錄下創建zookeeper_data
然后cd zookeeper_data進入該目錄，執行命令
touch myid
echo 3 > myid
利用scp -r zookeeper-3.4.5 root@SparkSlave01:/hadoopLearning/
scp -r zookeeper-3.4.5 root@SparkSlave02:/hadoopLearning/
將文件拷貝到其它服務器上，然后分別進入zookeeper_data目錄執行SparkSlave01服務器上echo 1> myid
SparkSlave02服務器上echo 2> myid

root@SparkMaster:/hadoopLearning/zookeeper-3.4.5/bin ./zkServer.sh start 在其它兩臺機器上執行相同操作root@SparkMaster:/hadoopLearning/zookeeper-3.4.5/bin zkServer.sh status JMX enabled by default Using config: /hadoopLearning/zookeeper-3.4.5/bin/../conf/zoo.cfg Mode: leader

至此ZooKeeper集群搭建完畢

- Hadoop-2.4.1集群搭建

將Hadoop安裝路徑HAD00P_HOME=/hadoopLearning/hadoop-2.4.1加入到環境變量

export JAVA_HOME=/hadoopLearning/jdk1.7.0_67 export JRE_HOME=${JAVA_HOME}/jre export HAD00P_HOME=/hadoopLearning/hadoop-2.4.1 export SCALA_HOME=/hadoopLearning/scala-2.10.4 export ZOOKEEPER_HOME=/hadoopLearning/zookeeper-3.4.5 export CLASS_PATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib export PATH=${JAVA_HOME}/bin:${HAD00P_HOME}/bin:${HAD00P_HOME}/sbin:${ZOOKEEPER_HOME}/bin:${SCALA_HOME}/bin:/hadoopLearning/idea-IC-141.1532.4/bin:$PATH 修改hadoo-env.shexport JAVA_HOME=/usr/java/jdk1.7.0_55 修改core-site.xml<configuration> <property><name>fs.defaultFS</name><value>hdfs://ns1</value></property><property><name>hadoop.tmp.dir</name><value>/hadoopLearning/hadoop-2.4.1/tmp</value></property><property><name>ha.zookeeper.quorum</name><value>SparkMaster:2181,SparkSlave01:2181,SparkSlave02:2181</value></property> </configuration>

修改hdfs-site.xml

<configuration><property><name>dfs.nameservices</name><value>ns1</value></property><property><name>dfs.ha.namenodes.ns1</name><value>nn1,nn2</value></property><property><name>dfs.namenode.rpc-address.ns1.nn1</name><value>SparkSlave01:9000</value></property><property><name>dfs.namenode.http-address.ns1.nn1</name><value>SparkSlave01:50070</value></property><property><name>dfs.namenode.rpc-address.ns1.nn2</name><value>SparkSlave02:9000</value></property><property><name>dfs.namenode.http-address.ns1.nn2</name><value>SparkSlave02:50070</value></property><property><name>dfs.namenode.shared.edits.dir</name><value>qjournal://SparkMaster:8485;SparkSlave01:8485;SparkSlave02:8485/ns1</value></property><property><name>dfs.journalnode.edits.dir</name><value>/hadoopLearning/hadoop-2.4.1/journal</value></property><property><name>dfs.ha.automatic-failover.enabled</name><value>true</value></property><property><name>dfs.client.failover.proxy.provider.ns1</name><value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value></property><property><name>dfs.ha.fencing.methods</name><value>sshfenceshell(/bin/true)</value></property><property><name>dfs.ha.fencing.ssh.private-key-files</name><value>/home/hadoop/.ssh/id_rsa</value></property><property><name>dfs.ha.fencing.ssh.connect-timeout</name><value>30000</value></property></configuration> 修改mapred-site.xml<configuration><property><name>mapreduce.framework.name</name><value>yarn</value></property></configuration> 修改yarn-site.xml <configuration><property><name>yarn.resourcemanager.ha.enabled</name><value>true</value></property><property><name>yarn.resourcemanager.cluster-id</name><value>SparkCluster</value></property><property><name>yarn.resourcemanager.ha.rm-ids</name><value>rm1,rm2</value></property><property><name>yarn.resourcemanager.hostname.rm1</name><value>SparkMaster</value></property><property><name>yarn.resourcemanager.hostname.rm2</name><value>SparkSlave01</value></property><property><name>yarn.resourcemanager.zk-address</name><value>SparkMaster:2181,SparkSlave01:2181,SparkSlave02:2181</value></property><property><name>yarn.nodemanager.aux-services</name><value>mapreduce_shuffle</value></property></configuration> 配置Slaves SparkMaster SparkSlave01 SparkSlave02

將配置好的hadoop-2.4.1拷到其它服務器上

scp -r /etc/profile root@SparkSlave01:/etc/profile scp -r /hadoopLearning/hadoop-2.4.1/ root@SparkSlave01:/hadoopLearning/scp -r /etc/profile root@SparkSlave02:/etc/profile scp -r /hadoopLearning/hadoop-2.4.1/ root@SparkSlave02:/hadoopLearning/ 啟動journalnodehadoop-daemons.sh start journalnode#運行jps命令檢驗，SparkMaster、SparkSlave01、SparkSlave02上多了JournalNode進程格式化HDFS#在SparkSlave01上執行命令:hdfs namenode -format#格式化后會在根據core-site.xml中的hadoop.tmp.dir配置生成個文件，這里我配置的是/hadoopLearning/hadoop-2.4.1/tmp，然后將/hadoopLearning/hadoop-2.4.1/tmp拷貝到SparkSlave02的/hadoopLearning/hadoop-2.4.1/下。 scp -r tmp/ sparkslave02:/hadoopLearning/hadoop-2.4.1/ 格式化ZK(在SparkSlave01上執行即可)hdfs zkfc -formatZK 啟動HDFS(在SparkSlave01上執行)sbin/start-dfs.sh 啟動YARN(#####注意#####：是在SparkMaster上執行start-yarn.sh，把namenode和resourcemanager分開是因為性能問題，因為他們都要占用大量資源，所以把他們分開了，他們分開了就要分別在不同的機器上啟動)sbin/start-yarn.sh

打開瀏覽器輸入：
http://sparkmaster:8088可以看到以下頁面：

輸入http://sparkslave01:50070可以看到以下頁面：

輸入http://sparkslave02:50070可以看到以下頁面

輸入以下命令上傳文件到hadoop
hadoop fs -put /etc/profile /

在active namenode上查看上傳成功的文件

至此hadoop集群搭建成功

Spark-1.4.0集群搭建
以Spark Standalone為例

1 在SparkMaster上安裝Scala 2.10.4和spark-1.4.0-bin-hadoop2.4,解壓對應安裝包到/hadoopLearning目錄，修改/etc/profile文件，內容如下：

export JAVA_HOME=/hadoopLearning/jdk1.7.0_67 export JRE_HOME=${JAVA_HOME}/jre export HAD00P_HOME=/hadoopLearning/hadoop-2.4.1 export SCALA_HOME=/hadoopLearning/scala-2.10.4 export SPARK_HOME=/hadoopLearning/spark-1.4.0-bin-hadoop2.4 export ZOOKEEPER_HOME=/hadoopLearning/zookeeper-3.4.5 export CLASS_PATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib export PATH=${JAVA_HOME}/bin:${HAD00P_HOME}/bin:${HAD00P_HOME}/sbin:${ZOOKEEPER_HOME}/bin:${SCALA_HOME}/bin:/hadoopLearning/idea-IC-141.1532.4/bin:${SPARK_HOME}/bin:${SPARK_HOME}/sbin:$PATH

2 進入/hadoopLearning/spark-1.4.0-bin-hadoop2.4/conf目錄
cp spark-defaults.conf.template spark-defaults.conf
cp spark-env.sh.template spark-env.sh
在spark-defaults.conf中添加如下內容：

export JAVA_HOME=/hadoopLearning/jdk1.7.0_67 export HADOOP_CONF_DIR=/hadoopLearning/hadoop-2.4.1/etc/hadoop

在spark-defaults.conf中添加如下內容：

spark.master=spark://sparkmaster:7077 spark.eventLog.enabled=true //hdfs://ns1是前面core-site.xml中定義的hdfs名稱 spark.eventLog.dir=hdfs://ns1/user/spark/applicationHistory

3 將sparkmaster中的安裝配置拷由到sparkslave01,sparkslave02上

scp -r /hadoopLearning/scala-2.10.4 sparkslave01:/hadoopLearning/scp -r /hadoopLearning/scala-2.10.4 sparkslave02:/hadoopLearning/scp -r /hadoopLearning/spark-1.4.0-bin-hadoop2.4 sparkslave01:/hadoopLearning/scp -r /hadoopLearning/spark-1.4.0-bin-hadoop2.4 sparkslave02:/hadoopLearning/scp -r /etc/profile sparkslave01:/etc/profile scp -r /etc/profile sparkslave02:/etc/profile

4 將sparkmaster中的 /hadoopLearning/spark-1.4.0-bin-hadoop2.4/sbin中執行以下命令：

./start-all.sh

利用jps在各主要上查看，可以看到sparkmaster上多了進程master，而sparkslave01,sparkslave02多了進程worker

在瀏覽器中輸入http://sparkmaster:8080,可以看到如下界面：

該圖中顯示了集群的運行相關信息，說明集群初步搭建成功

5 spark-1.4.0 集群程序運行測試
上傳 README.md文件到hdfs /user/root目錄root@sparkmaster:/hadoopLearning/spark-1.4.0-bin-hadoop2.4# hadoop fs -put README.md /user/root

在sparkmaster節點，進入 /hadoopLearning/spark-1.4.0-bin-hadoop2.4/bin目錄，執行spark-shell，刷新http://sparkmaster:8080后可以看到以下內容：

輸入下列語句：

val textCount = sc.textFile(“README.md”).filter(line => line.contains(“Spark”)).count()

程序結果如下：

至此，Spark集群搭建成功

總結

以上是生活随笔為你收集整理的Spark-1.4.0集群搭建的全部內容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網站內容還不錯，歡迎將生活随笔推薦給好友。

集群
Spark

上一篇： HDU 2955 Robberies
下一篇：（九）模型驱动和属性驱动