當前位置：首頁 > 编程资源 > 编程问答 >内容正文

编程问答

大数据流式计算 apache storm 学习笔记 01 ---汪文君

發布時間：2023/12/9 编程问答 42 豆豆

生活随笔收集整理的這篇文章主要介紹了大数据流式计算 apache storm 学习笔记 01 ---汪文君小編覺得挺不錯的,現在分享給大家,幫大家做個參考.

汪文君Apache Storm1.2.2實戰

9個小時
https://www.bilibili.com/video/BV1et41147xq?p=1

P1汪文君Apache Storm1.2.2實戰-01講-Apache Storm的介紹 20:53
P2汪文君Apache Storm1.2.2實戰-02講-Apache Storm關鍵組件的詳細講解 14:42
P3汪文君Apache Storm1.2.2實戰-03講-Apache Storm集群環境搭建詳解 33:09
P4汪文君Apache Storm1.2.2實戰-04講-第一個Topology程序開發本地模式 23:53
P5汪文君Apache Storm1.2.2實戰-05講-第一個Topology程序開發集群模式 23:26
P6汪文君Apache Storm1.2.2實戰-06講-Storm并行度核心概念講解 42:54
P7汪文君Apache Storm1.2.2實戰-07講-通過程序運行分析Topology的并行度-上 29:56
P8汪文君Apache Storm1.2.2實戰-08講-通過程序運行分析Topology的并行度-下 17:39
P9汪文君Apache Storm1.2.2實戰-09講-Storm Topology的Rebalance 10:41
P10汪文君Apache Storm1.2.2實戰-10講-分析Storm Topology并行度內容補充 11:26
P11汪文君Apache Storm1.2.2實戰-11講-shuffle grouping數據分組詳解 25:46
P12汪文君Apache Storm1.2.2實戰-12講-shuffle grouping數據分組詳解（補充） 05:49
P13汪文君Apache Storm1.2.2實戰-13講-fields grouping數據分組詳解（戴耳機聽） 23:29
P14汪文君Apache Storm1.2.2實戰-14講-all grouping講解前的分析過程 20:26
P15汪文君Apache Storm1.2.2實戰-15講-all grouping詳細講解 14:16
P16汪文君Apache Storm1.2.2實戰-16講-global grouping詳細講解 25:51
P17汪文君Apache Storm1.2.2實戰-17講-direct grouping詳細講解 26:17
P18汪文君Apache Storm1.2.2實戰-18講-none grouping& localOrShuffe grouping 04:52
P19汪文君Apache Storm1.2.2實戰-19講-如何自定義storm grouping詳解 33:33
P20汪文君Apache Storm1.2.2實戰-20講-綜合案例之電信號碼主叫被叫實時統計 24:01
P21汪文君Apache Storm1.2.2實戰-21講-綜合案例之Word Count 25:05
P22汪文君Apache Storm1.2.2實戰-22講-Storm擔保數據被處理ack,failed,timeout,exeption講解（戴耳機聽） 26:53
P23汪文君Apache Storm1.2.2實戰-23講-Storm Fully擔保數據被處理的案例分析 15:38
P24汪文君Apache Storm1.2.2實戰-24講-一些擔保數據被處理的方案講解 16:30
P25汪文君Apache Storm1.2.2實戰-25講-高級Tuple的使用（Tick Tuple） 16:42

指令

-----啟動nimbus storm nimbus & --- 修改日志級別 ------方便查看日志比如將日志從 info 降級為 warn [root@storm51 log4j2]# vim /usr/local/storm/log4j2/worker.xml ---- 查看日志記錄 [root@storm51 logs]# tail -f 200 /usr/local/storm/logs/nimbus.log --- 查看 topology 的日志 [root@storm53 workers-artifacts]# pwd /usr/local/storm/logs/workers-artifacts [root@storm53 workers-artifacts]# ll total 0 drwxr-xr-x. 3 root root 18 Apr 10 02:13 tp1-1-1618046001 drwxr-xr-x. 3 root root 18 Apr 10 02:51 tp2-2-1618048284 drwxr-xr-x. 3 root root 18 Apr 10 03:51 tp6-7-1618051908

P1汪文君Apache Storm1.2.2實戰-01講-Apache Storm的介紹 20:53

項目Value

1	Java8 In Action
1	Apache Flume
1	PowerMock
1	Concordion
1	Mockito
1	Apache Sqoop
1	Java Concurrency
1	Google Guava
1	Scala In Action
1	Apache Kafka 0.11.x
1	Metrics
1	JMH
1

官網： https://storm.apache.org/

流式計算產品 Streaming Compute System

項目Value

1	Apache Spark Streaming
1	Apache Flink
1	Apache Kafka Streaming
1	Apache Storm
1	… …

課程大綱

P2汪文君Apache Storm1.2.2實戰-02講-Apache Storm關鍵組件的詳細講解 14:42

A Storm cluster follows a master-slave model

1.Master主節點-------Nimbus
The Nimbus node is the master in a Storm cluster.
The Nimbus is stateless and stores all of its data in Zookeeper.There is a single Nimbus node in a Storm cluster.

2.Supervisor Nodes 工作節點

3.Tuples --The Storm data model

4.Storm Topology
5.Spout&Stream&Bolt
5.1 Spout: A spout is the source of tuples in a Storm Topology
5.2 Bolt: A bolt is the processing powerhouse of a Storm topology , and is responsible for transforming a stream
5.3 Stream: The key abstraction in Storm is tha of a stream.

P3汪文君Apache Storm1.2.2實戰-03講-Apache Storm集群環境搭建詳解 33:09

視頻作者用時30分鐘講完，leo用時4個小時搭建完成（20210405），作者用1.2.2 ，Leo用 1.2.3

linux centos 7.6 安裝 Apache Storm1.2.3
https://blog.csdn.net/wei198621/article/details/115449855

P4汪文君Apache Storm1.2.2實戰-04講-第一個Topology程序開發本地模式 23:53

spout 生成隨機數交個 bolt 顯示 ****outputCollector 用于發送數據 step1. RandomStringSpout.java ------------- SpoutOutputCollector.emit(new Tuple(...)); 將數據通過collector 以tuple 格式發送 ------------- OutputFieldsDeclarer.declare(new Fields("stream")); 命名發送的tuple 為stream step2. WrapStarBolt.java Tuple.getStringByField("stream"); 獲取spout 發過來的名為 stream 的 tuple step3. WrapWellBolt.java Tuple.getStringByField("stream"); 獲取spout 發過來的名為 stream 的 tuple step4. RandomStringTopologyLocal.java 4.1 topoBuilder= new TopologyBuilder() 將 1 2 3 步驟的 spout Bolt 賦值給此 TopologyBuilder 4.2 cluster = new LocalCluster() 定義cluster 4.3 conf = new Config() 定義cluster 的配置 4.4 cluster.submitTopology( *, conf, topoBuilder); cluster 提交 TopologyBuilder

P5汪文君Apache Storm1.2.2實戰-05講-第一個Topology程序開發集群模式 23:26

啟動zookeeper storm 集群

[root@centos7-7 bin]# ll total 24 -rwxr--r--. 1 root root 466 Apr 8 06:06 ctlkafkaauto.sh -rwxr--r--. 1 root root 133 Apr 8 09:23 ctlstorm.sh -rwxr--r--. 1 root root 394 Apr 8 06:06 ctlzookeeperauto.sh lrwxrwxrwx. 1 root root 36 Apr 8 06:09 jps -> /usr/local/java/jdk1.8.0_251/bin/jps -rwxr--r--. 1 root root 154 Apr 8 06:06 xcallkafka.sh -rwxr--r--. 1 root root 153 Apr 8 08:31 xcallstorm.sh -rwxr--r--. 1 root root 145 Apr 8 06:06 xcallzk.sh [root@centos7-7 bin]# ctlzookeeperauto.sh start ... 暫時還不可以批量執行需要手動執行 [root@centos7-7 bin]# cat ctlstorm.sh ssh storm51 "storm ui 1>/dev/null 2>&1" ssh storm51 "storm nimbus 1>/dev/null 2>&1" ssh storm52 "storm supervisor 1>/dev/null 2>&1" ssh storm53 "storm supervisor 1>/dev/null 2>&1" [root@centos7-7 bin]# xcallzk.sh jps ============= zk1 jps ============= 9360 QuorumPeerMain 29191 Jps ============= zk2 jps ============= 29269 Jps 9384 QuorumPeerMain ============= zk3 jps ============= 9181 QuorumPeerMain 29085 Jps[root@centos7-7 bin]# xcallstorm.sh jps ============= storm51 jps ============= 12126 core 13054 nimbus 17663 Jps ============= storm52 jps ============= 30615 Supervisor 31870 Jps ============= storm53 jps ============= 30256 worker 27252 Supervisor 31726 Jps

啟動 storm ui
http://192.168.121.51:8080/index.html

啟動腳本

[root@storm51 ~]# storm jar storm-1.0-SNAPSHOT-jar-with-dependencies.jar com.tiza.leo.bigdata.storm.test01Random.RandomStringTopologyRemote 2340 [main] WARN o.a.s.u.Utils - STORM-VERSION new 1.2.3 old 1.2.3 3033 [main] INFO o.a.s.StormSubmitter - Finished submitting topology: RandomStringTopologyRemote [root@storm51 ~]#

查看topology是否啟動腳本

[root@storm51 ~]# storm list Running: .../usr/local/apache-storm-1.2.3/bin org.apache.storm.command.list 5402 [main] INFO o.a.s.u.NimbusClient - Found leader nimbus : master:6627 Topology_name Status Num_tasks Num_workers Uptime_secs ------------------------------------------------------------------- RandomStringTopologyRemote ACTIVE 12 3 98 [root@storm51 ~]#

此時查看jps運行進程

[root@centos7-7 bin]# xcallstorm.sh jps ============= storm51 jps ============= 12126 core 13054 nimbus 17663 Jps ============= storm52 jps ============= 30615 Supervisor 31870 Jps ============= storm53 jps ============= 30256 worker 27252 Supervisor 30214 worker 30215 worker 30185 LogWriter 30186 LogWriter 30221 LogWriter 31726 Jps

敦化暫停一個 topology

storm deactivate RandomStringTopologyRemote

P6汪文君Apache Storm1.2.2實戰-06講-Storm并行度核心概念講解 42:54

更改日志節點為warn ,fang變查看

[root@storm52 log4j2]# vim /usr/local/storm/log4j2/worker.xml [root@storm53 log4j2]# vim /usr/local/storm/log4j2/worker.xml

P7汪文君Apache Storm1.2.2實戰-07講-通過程序運行分析Topology的并行度-上 29:56

到 zhan 筆記本上面用腳本啟動 zookeerper storm
http://192.168.121.51:8080/index.html 確認一切OK ,
編譯上一步的文件，給到 storm 主節點 storm 51

測試1

[root@storm51 ~]# storm jar storm-1.0-SNAPSHOT-jar-with-dependencies.jar com.tiza.leo.bigdata.storm.test02Parallel.SimpleTopology ------ 報錯，需要參數 ====================================================================================[root@storm51 ~]# storm jar storm-1.0-SNAPSHOT-jar-with-dependencies.jar com.tiza.leo.bigdata.storm.test02Parallel.SimpleTopology tp1 tp1 1 2 1 2 1 {topologyName='tp1', prefix='tp1', workers=1, spoutParallelHint=2, spoutTasks=1, boltParallelHint=2, boltTasks=1}------------------------------------------------------------------------------- 結論并行度(parallelHint) Num executors = 1worker + 1spoutTask + 1boltTask =3 ====================================================================================

測試2

[root@storm51 ~]# storm jar storm-1.0-SNAPSHOT-jar-with-dependencies.jar com.tiza.leo.bigdata.storm.test02Parallel.SimpleTopology tp2 tp2 2 2 1 2 1 {topologyName='tp2', prefix='tp2', workers=2, spoutParallelHint=2, spoutTasks=1, boltParallelHint=2, boltTasks=1}------------------------------------------------------------------------------- 結論并行度(parallelHint) Num executors = 2 worker + 1spoutTask + 1boltTask =4

測試3

==================================================================================== [root@storm51 ~]# storm jar storm-1.0-SNAPSHOT-jar-with-dependencies.jar com.tiza.leo.bigdata.storm.test02Parallel.SimpleTopology tp3 tp3 1 2 2 2 2 {topologyName='tp3', prefix='tp3', workers=1, spoutParallelHint=2, spoutTasks=2, boltParallelHint=2, boltTasks=2} ------------------------------------------------------------------------------- 結論并行度(parallelHint) Num executors = 1 worker + 2 spoutTask + 2 boltTask =5

測試4

==================================================================================== [root@storm51 ~]# storm jar storm-1.0-SNAPSHOT-jar-with-dependencies.jar com.tiza.leo.bigdata.storm.test02Parallel.SimpleTopology tp4 tp4 3 3 1 3 2 {topologyName='tp4', prefix='tp4', workers=3, spoutParallelHint=3, spoutTasks=1, boltParallelHint=3, boltTasks=2} ------------------------------------------------------------------------------- 推測結論并行度(parallelHint) Num executors = 3 worker + 1 spoutTask + 2 boltTask =6 實際結論并行度(parallelHint) Num executors = 3 worker + 2 spoutTask + 2 boltTask =7 why ? why ? why ? 2021 10 30 ，原因是 task 數據量=（worker+ + boltTask*2 ）

測試5

==================================================================================== [root@storm51 ~]# storm jar storm-1.0-SNAPSHOT-jar-with-dependencies.jar com.tiza.leo.bigdata.storm.test02Parallel.SimpleTopology tp5 tp5 3 2 2 2 2 {topologyName='tp5', prefix='tp5', workers=3, spoutParallelHint=2, spoutTasks=2, boltParallelHint=2, boltTasks=2} ------------------------------------------------------------------------------- 結論并行度(parallelHint) Num executors = 3 worker + 2 spoutTask + 2 boltTask =7

測試6

==================================================================================== [root@storm51 ~]# storm jar storm-1.0-SNAPSHOT-jar-with-dependencies.jar com.tiza.leo.bigdata.storm.test02Parallel.SimpleTopology tp6 tp6 1 2 4 2 4 {topologyName='tp6', prefix='tp6', workers=1, spoutParallelHint=2, spoutTasks=4, boltParallelHint=2, boltTasks=4} ------------------------------------------------------------------------------- 結論并行度(parallelHint) Num executors = 1 worker + 4 spoutTask + 4 boltTask =9

測試7

==================================================================================== [root@storm51 ~]# storm jar storm-1.0-SNAPSHOT-jar-with-dependencies.jar com.tiza.leo.bigdata.storm.test02Parallel.SimpleTopology tp7 tp7 1 1 4 1 4 {topologyName='tp7', prefix='tp7', workers=1, spoutParallelHint=1, spoutTasks=4, boltParallelHint=1, boltTasks=4} ------------------------------------------------------------------------------- 結論并行度(parallelHint) Num executors = 1 worker + 4 spoutTask + 4 boltTask =9

之前的參數是 -n 1 -e tp7-SimpleSpout=4 -e tp7-SimpleBolt=4 修改為 storm reblance tp7 -n 2 -e tp7-SimpleSpout=8 -e tp7-SimpleBolt=8

----------------------------topo名稱-------------2個;----spout 4個 -----bolt 4 個 ----- bin/storm rebalance SampleStormClusterTopology -n 2 -e SampleSpout=4 -e SapmleBolt=4 [root@storm51 ~]# storm rebalance tp7 -n 2 -e tp7-SimpleSpout=8 -e tp7-SimpleBolt=8

P8汪文君Apache Storm1.2.2實戰-08講-通過程序運行分析Topology的并行度-下 17:39

見上

P9汪文君Apache Storm1.2.2實戰-09講-Storm Topology的Rebalance 10:41

見上

P10汪文君Apache Storm1.2.2實戰-10講-分析Storm Topology并行度內容補充 11:26

P11汪文君Apache Storm1.2.2實戰-11講-shuffle grouping數據分組詳解 25:46

P12汪文君Apache Storm1.2.2實戰-12講-shuffle grouping數據分組詳解（補充） 05:49

P13汪文君Apache Storm1.2.2實戰-13講-fields grouping數據分組詳解（戴耳機聽） 23:29

P14汪文君Apache Storm1.2.2實戰-14講-all grouping講解前的分析過程 20:26

P15汪文君Apache Storm1.2.2實戰-15講-all grouping詳細講解 14:16

P16汪文君Apache Storm1.2.2實戰-16講-global grouping詳細講解 25:51

P17汪文君Apache Storm1.2.2實戰-17講-direct grouping詳細講解 26:17

P18汪文君Apache Storm1.2.2實戰-18講-none grouping& localOrShuffe grouping 04:52

P19汪文君Apache Storm1.2.2實戰-19講-如何自定義storm grouping詳解 33:33

P20汪文君Apache Storm1.2.2實戰-20講-綜合案例之電信號碼主叫被叫實時統計 24:01

P21汪文君Apache Storm1.2.2實戰-21講-綜合案例之Word Count 25:05

P22汪文君Apache Storm1.2.2實戰-22講-Storm擔保數據被處理ack,failed,timeout,exeption講解（戴耳機聽） 26:53

P23汪文君Apache Storm1.2.2實戰-23講-Storm Fully擔保數據被處理的案例分析 15:38

P24汪文君Apache Storm1.2.2實戰-24講-一些擔保數據被處理的方案講解 16:30

P25汪文君Apache Storm1.2.2實戰-25講-高級Tuple的使用（Tick Tuple） 16:42

總結

以上是生活随笔為你收集整理的大数据流式计算 apache storm 学习笔记 01 ---汪文君的全部內容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網站內容還不錯，歡迎將生活随笔推薦給好友。

上一篇：关于spring boot多张表建立外健
下一篇： oracle中日期处理方法汇总