根據(jù)tomcat日志計算url訪問了情況，具體的url如下，
要求：區(qū)別統(tǒng)計GET和POST URL訪問量
結(jié)果為：訪問方式、URL、訪問量

?

輸入文件：

196.168.2.1 - - [03/Jul/2014:23:36:38 +0800] "GET /course/detail/3.htm HTTP/1.0" 200 38435 0.038
182.131.89.195 - - [03/Jul/2014:23:37:43 +0800] "GET /html/notes/20140617/888.html HTTP/1.0" 301 - 0.000
196.168.2.1 - - [03/Jul/2014:23:38:27 +0800] "POST /service/notes/addViewTimes_23.htm HTTP/1.0" 200 2 0.003
196.168.2.1 - - [03/Jul/2014:23:39:03 +0800] "GET /html/notes/20140617/779.html HTTP/1.0" 200 69539 0.046
196.168.2.1 - - [03/Jul/2014:23:43:00 +0800] "GET /html/notes/20140318/24.html HTTP/1.0" 200 67171 0.049
196.168.2.1 - - [03/Jul/2014:23:43:59 +0800] "POST /service/notes/addViewTimes_779.htm HTTP/1.0" 200 1 0.003
196.168.2.1 - - [03/Jul/2014:23:45:51 +0800] "GET /html/notes/20140617/888.html HTTP/1.0" 200 70044 0.060
196.168.2.1 - - [03/Jul/2014:23:46:17 +0800] "GET /course/list/73.htm HTTP/1.0" 200 12125 0.010
196.168.2.1 - - [03/Jul/2014:23:46:58 +0800] "GET /html/notes/20140609/542.html HTTP/1.0" 200 94971 0.077
196.168.2.1 - - [03/Jul/2014:23:48:31 +0800] "POST /service/notes/addViewTimes_24.htm HTTP/1.0" 200 2 0.003
196.168.2.1 - - [03/Jul/2014:23:48:34 +0800] "POST /service/notes/addViewTimes_542.htm HTTP/1.0" 200 2 0.003
196.168.2.1 - - [03/Jul/2014:23:49:31 +0800] "GET /notes/index-top-3.htm HTTP/1.0" 200 53494 0.041

?

//textFile() 加載數(shù)據(jù)
val data = sc.textFile("/spark/seven.txt")

//filter 過濾長度小于0， 過濾不包含GET與POST的URL
val filtered = data.filter(_.length()>0).filter( line => (line.indexOf("GET")>0 || line.indexOf("POST")>0) )

//轉(zhuǎn)換成鍵值對操作
val res = filtered.map( line => {
if(line.indexOf("GET")>0){ //截取 GET 到URL的字符串
(line.substring(line.indexOf("GET"),line.indexOf("HTTP/1.0")).trim,1)
}else{?? //截取 POST 到URL的字符串
(line.substring(line.indexOf("POST"),line.indexOf("HTTP/1.0")).trim,1)
}//最后通過reduceByKey求sum
}).reduceByKey(_+_)

//觸發(fā)action事件執(zhí)行
res.collect()

?

?

輸出文件：

(POST?/service/notes/addViewTimes_542.htm,2),???

(POST?/service/notes/addViewTimes_544.htm,1),???

(GET?/html/notes/20140609/542.html,2),???

(POST?/service/notes/addViewTimes_23.htm,1),???

(GET?/html/notes/20140617/888.html,3),???

(POST?/service/notes/addViewTimes_24.htm,1),???

(GET?/course/detail/3.htm,1),???

(GET?/course/list/73.htm,2),???

(GET?/html/notes/20140617/779.html,1),???

(GET?/html/notes/20140620/872.html,1)?

?

轉(zhuǎn)載于:https://www.cnblogs.com/energy1010/p/6933914.html

總結(jié)

以上是生活随笔為你收集整理的Spark日志分析的全部內(nèi)容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網(wǎng)站內(nèi)容還不錯，歡迎將生活随笔推薦給好友。