本地部署

安裝

在官網安裝Flink，并解壓到/usr/local/flink

• sudo tar -zxf flink-1.6.2-bin-hadoop27-scala_2.11.tgz -C /usr/local cd /usr/local

• 54388226982

修改文件名字，并設置權限

• sudo mv ./flink-*/ ./flink sudo chown -R hadoop:hadoop ./flink

修改配置文件

• Flink對于本地模式是開箱即用的，如果要修改Java運行環境，可修改conf/flink-conf.yaml中的env.java.home，設置為本地java的絕對路徑

添加環境變量

vim ~/.bashrc export FLINK_HOME=/usr/local/flink export PATH=$FLINK_HOME/bin:$PATH

54388242695

啟動Flink

start-cluster.sh

• 可以通過觀察logs目錄下的日志來檢測系統是否正在運行了

tail log/flink--jobmanager-.log

54388315301

• JobManager同時會在8081端口上啟動一個web前端，通過http://localhost:8081來訪問

54388290147

可以發現flink已經正常啟動

運行示例

使用Maven創建Flink項目，在pom.xml中添加以下依賴：

<dependencies><dependency><groupId>org.apache.flink</groupId><artifactId>flink-java</artifactId><version>1.6.2</version></dependency><dependency><groupId>org.apache.flink</groupId><artifactId>flink-streaming-java_2.11</artifactId><version>1.6.2</version></dependency><dependency><groupId>org.apache.flink</groupId><artifactId>flink-clients_2.11</artifactId><version>1.6.2</version></dependency></dependencies>

批處理運行WordCount

官方示例

可以直接在/usr/local/flink/examples/batch中運行WordCount程序，并且這里還有更多示例：

54388437325

運行：

flink run WordCount.jar

54388443638

代碼

WordCountData

提供原始數據

import org.apache.flink.api.java.DataSet; import org.apache.flink.api.java.ExecutionEnvironment;public class WordCountData {public static final String[] WORDS=new String[]{"To be, or not to be,--that is the question:--", "Whether \'tis nobler in the mind to suffer", "The slings and arrows of outrageous fortune", "Or to take arms against a sea of troubles,", "And by opposing end them?--To die,--to sleep,--", "No more; and by a sleep to say we end", "The heartache, and the thousand natural shocks", "That flesh is heir to,--\'tis a consummation", "Devoutly to be wish\'d. To die,--to sleep;--", "To sleep! perchance to dream:--ay, there\'s the rub;", "For in that sleep of death what dreams may come,", "When we have shuffled off this mortal coil,", "Must give us pause: there\'s the respect", "That makes calamity of so long life;", "For who would bear the whips and scorns of time,", "The oppressor\'s wrong, the proud man\'s contumely,", "The pangs of despis\'d love, the law\'s delay,", "The insolence of office, and the spurns", "That patient merit of the unworthy takes,", "When he himself might his quietus make", "With a bare bodkin? who would these fardels bear,", "To grunt and sweat under a weary life,", "But that the dread of something after death,--", "The undiscover\'d country, from whose bourn", "No traveller returns,--puzzles the will,", "And makes us rather bear those ills we have", "Than fly to others that we know not of?", "Thus conscience does make cowards of us all;", "And thus the native hue of resolution", "Is sicklied o\'er with the pale cast of thought;", "And enterprises of great pith and moment,", "With this regard, their currents turn awry,", "And lose the name of action.--Soft you now!", "The fair Ophelia!--Nymph, in thy orisons", "Be all my sins remember\'d."};public WordCountData() {}public static DataSet<String> getDefaultTextLineDataset(ExecutionEnvironment env){return env.fromElements(WORDS);} }

WordCountTokenizer

切分句子

import org.apache.flink.api.common.functions.FlatMapFunction; import org.apache.flink.api.java.tuple.Tuple2; import org.apache.flink.util.Collector;public class WordCountTokenizer implements FlatMapFunction<String, Tuple2<String,Integer>>{public WordCountTokenizer(){}public void flatMap(String value, Collector<Tuple2<String, Integer>> out) throws Exception {String[] tokens = value.toLowerCase().split("\\W+");int len = tokens.length;for(int i = 0; i<len;i++){String tmp = tokens[i];if(tmp.length()>0){out.collect(new Tuple2<String, Integer>(tmp,Integer.valueOf(1)));}}} }

WordCount

主函數

import org.apache.flink.api.java.DataSet; import org.apache.flink.api.java.ExecutionEnvironment; import org.apache.flink.api.java.operators.AggregateOperator; import org.apache.flink.api.java.utils.ParameterTool;public class WordCount {public WordCount(){}public static void main(String[] args) throws Exception {ParameterTool params = ParameterTool.fromArgs(args);ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();env.getConfig().setGlobalJobParameters(params);Object text;//如果沒有指定輸入路徑，則默認使用WordCountData中提供的數據if(params.has("input")){text = env.readTextFile(params.get("input"));}else{System.out.println("Executing WordCount example with default input data set.");System.out.println("Use -- input to specify file input.");text = WordCountData.getDefaultTextLineDataset(env);}AggregateOperator counts = ((DataSet)text).flatMap(new WordCountTokenizer()).groupBy(new int[]{0}).sum(1);//如果沒有指定輸出，則默認打印到控制臺if(params.has("output")){counts.writeAsCsv(params.get("output"),"\n", " ");env.execute();}else{System.out.println("Printing result to stdout. Use --output to specify output path.");counts.print();}} }

首先打包成JAR包，這里需要使用-c指定main函數：

flink run -c WordCount WordCount.jar

流處理運行WordCount

官方示例

可以直接在/usr/local/flink/examples/streaming中運行WordCount程序，并且這里還有更多示例：

54388669798

代碼

SocketWindowWordCount

import org.apache.flink.api.common.functions.FlatMapFunction; import org.apache.flink.api.java.utils.ParameterTool; import org.apache.flink.streaming.api.datastream.DataStream; import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;import java.sql.Time; import java.util.stream.Collector;public class SocketWindowWordCount {public static void main(String[] args) throws Exception {// the port to connect tofinal int port;try {final ParameterTool params = ParameterTool.fromArgs(args);port = params.getInt("port");} catch (Exception e) {System.err.println("No port specified. Please run 'SocketWindowWordCount --port <port>'");return;}// get the execution environmentfinal StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();// get input data by connecting to the socketDataStream<String> text = env.socketTextStream("localhost", port, "\n");// parse the data, group it, window it, and aggregate the countsDataStream<WordWithCount> windowCounts = text.flatMap(new FlatMapFunction<String, WordWithCount>() {@Overridepublic void flatMap(String value, Collector<WordWithCount> out) {for (String word : value.split("\\s")) {out.collect(new WordWithCount(word, 1L));}}}).keyBy("word").timeWindow(Time.seconds(5), Time.seconds(1)).reduce(new ReduceFunction<WordWithCount>() {@Overridepublic WordWithCount reduce(WordWithCount a, WordWithCount b) {return new WordWithCount(a.word, a.count + b.count);}});// print the results with a single thread, rather than in parallelwindowCounts.print().setParallelism(1);env.execute("Socket Window WordCount");}// Data type for words with countpublic static class WordWithCount {public String word;public long count;public WordWithCount() {}public WordWithCount(String word, long count) {this.word = word;this.count = count;}@Overridepublic String toString() {return word + " : " + count;}} }

首先打包成JAR包，然后啟動netcat：

nc -l 9000

將終端啟動netcat作為輸入流：

提交Jar包：

flink run -c SocketWindowWordCount WordCountSteaming.jar --port 9000

這樣終端會一直等待netcat的輸入流

54388822906

在netcat中輸入字符流：

54388825265

可以在WebUI中查看運行結果：

54388897680

作者：zealscott
鏈接：https://www.jianshu.com/p/bbaa8d72cfcf/
來源：簡書
簡書著作權歸作者所有，任何形式的轉載都請聯系作者獲得授權并注明出處。

總結

以上是生活随笔為你收集整理的Flink安装及使用的全部內容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網站內容還不錯，歡迎將生活随笔推薦給好友。