當(dāng)前位置：首頁 > 运维知识 > 数据库 >内容正文

数据库

MapReduce读取数据库操作

發(fā)布時間：2024/8/1 数据库 51 豆豆

生活随笔收集整理的這篇文章主要介紹了 MapReduce读取数据库操作小編覺得挺不錯的,現(xiàn)在分享給大家,幫大家做個參考.

MapReduce讀取數(shù)據(jù)庫操作

MapReduce
- MapReduce 實體類
- MapReduce DBMapper類（讀數(shù)據(jù)庫操作）
- MapReduce DBReduce類（讀數(shù)據(jù)庫操作）
- MapReduce DBDriver類（讀數(shù)據(jù)庫操作驅(qū)動類）
- MapReduce 數(shù)據(jù)庫表

MapReduce

MapReduce 實體類

讀取數(shù)據(jù)庫操作，首先創(chuàng)建實體類。實體類需要繼承 Writable，DBWritable兩個類。重寫Writable的 write（序列化方法），readFields（反序列化方法）方法。重寫 DBWritable的 write（序列化方法），readFields（反序列化方法）。兩個繼承類的重寫的方法名一樣但是參數(shù)不同。
下面是GoodsBean實體類：

import org.apache.hadoop.io.Writable; import org.apache.hadoop.mapreduce.lib.db.DBWritable;import java.io.DataInput; import java.io.DataOutput; import java.io.IOException; import java.sql.PreparedStatement; import java.sql.ResultSet; import java.sql.SQLException;public class GoodsBean implements Writable, DBWritable {private long goodsId;//商品IDprivate String goodsSn;//商品編號private String goodsName;//商品名稱private double marketPrice;//市場價private double shopPrice;//門店價private long saleNum;//總銷售量public long getGoodsId() {return goodsId;}public void setGoodsId(long goodsId) {this.goodsId = goodsId;}public String getGoodsSn() {return goodsSn;}public void setGoodsSn(String goodsSn) {this.goodsSn = goodsSn;}public String getGoodsName() {return goodsName;}public void setGoodsName(String goodsName) {this.goodsName = goodsName;}public double getMarketPrice() {return marketPrice;}public void setMarketPrice(double marketPrice) {this.marketPrice = marketPrice;}public double getShopPrice() {return shopPrice;}public void setShopPrice(double shopPrice) {this.shopPrice = shopPrice;}public long getSaleNum() {return saleNum;}public void setSaleNum(long saleNum) {this.saleNum = saleNum;}/*** 無參構(gòu)造方法*/public GoodsBean() {}/*** 有參構(gòu)造方法*/public GoodsBean(long goodsId, String goodsSn, String goodsName, double marketPrice, double shopPrice, long saleNum) {this.goodsId = goodsId;this.goodsSn = goodsSn;this.goodsName = goodsName;this.marketPrice = marketPrice;this.shopPrice = shopPrice;this.saleNum = saleNum;}/*** set方法用于對象賦值*/public void set(long goodsId, String goodsSn, String goodsName, double marketPrice, double shopPrice, long saleNum) {this.goodsId = goodsId;this.goodsSn = goodsSn;this.goodsName = goodsName;this.marketPrice = marketPrice;this.shopPrice = shopPrice;this.saleNum = saleNum;}//此處省略setter getter方法/*** 序列化方法*/@Overridepublic void write(DataOutput out) throws IOException {out.writeLong(goodsId);out.writeUTF(goodsSn);out.writeUTF(goodsName);out.writeDouble(marketPrice);out.writeDouble(shopPrice);out.writeLong(saleNum);}/*** 反序列化方法*/@Overridepublic void readFields(DataInput in) throws IOException {this.goodsId = in.readLong();this.goodsSn = in.readUTF();this.goodsName = in.readUTF();this.marketPrice = in.readDouble();this.shopPrice = in.readDouble();this.saleNum = in.readLong();}/*** 在PreparedStatement中設(shè)置對象的字段。寫數(shù)據(jù)庫*/@Overridepublic void write(PreparedStatement ps) throws SQLException {ps.setLong(1, goodsId);ps.setString(2, goodsSn);ps.setString(3, goodsName);ps.setDouble(4, marketPrice);ps.setDouble(5, shopPrice);ps.setLong(6, saleNum);}/*** 從ResultSet中讀取對象的字段。讀數(shù)據(jù)庫*/@Overridepublic void readFields(ResultSet rs) throws SQLException {this.goodsId = rs.getLong(1);this.goodsSn = rs.getString(2);this.goodsName = rs.getString(3);this.marketPrice = rs.getDouble(4);this.shopPrice = rs.getDouble(5);this.saleNum = rs.getLong(6);}@Overridepublic String toString() {return goodsId + "\t" + goodsSn + "\t" + goodsName + "\t" + marketPrice + "\t" + shopPrice + "\t" + saleNum;}}

MapReduce DBMapper類（讀數(shù)據(jù)庫操作）

此處為讀取數(shù)據(jù)庫操作需要繼承Mapper<KeyIn,ValueIn,KeyOut,ValueOut>重寫map方法
KeyIn:一般為行數(shù)坐標(biāo) LongWritable
ValueIn ：為讀取數(shù)據(jù)庫的行數(shù)據(jù) GoodBean
KeyOut：輸出的key LongWritable
ValueOut：輸出的值，將數(shù)據(jù)庫寫入文檔當(dāng)中所以用字符串格式 Text

import org.apache.hadoop.io.LongWritable; import org.apache.hadoop.io.Text; import org.apache.hadoop.mapreduce.Mapper;import java.io.IOException;public class DBMapper extends Mapper<LongWritable, GoodsBean, LongWritable, Text> {Text outValue = new Text();@Overrideprotected void map(LongWritable key, GoodsBean value, Mapper<LongWritable, GoodsBean, LongWritable, Text>.Context context) throws IOException, InterruptedException {outValue.set(value.toString());//此處的輸出key‘和輸入key一樣所以直接使用輸入的keycontext.write(key,outValue);} }

MapReduce DBReduce類（讀數(shù)據(jù)庫操作）

因為需求簡單沒有進行Reduce合并的需求可以不寫此類

MapReduce DBDriver類（讀數(shù)據(jù)庫操作驅(qū)動類）

import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.Path; import org.apache.hadoop.io.LongWritable; import org.apache.hadoop.io.Text; import org.apache.hadoop.mapreduce.Job; import org.apache.hadoop.mapreduce.lib.db.DBConfiguration; import org.apache.hadoop.mapreduce.lib.db.DBInputFormat; import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;import java.io.IOException;public class DBDriver {public static void main(String[] args) throws IOException, InterruptedException, ClassNotFoundException {Configuration conf = new Configuration();//數(shù)據(jù)庫配置DBConfiguration.configureDB(conf, "com.mysql.cj.jdbc.Driver", "jdbc:mysql://localhost/shop?serverTimezone=GMT%2B8&Unicode=true&characterEncoding=utf-8&useSSL=false", "root", "123456");//創(chuàng)建作業(yè)的job類Job job = Job.getInstance(conf, DBDriver.class.getSimpleName());//驅(qū)動類job.setJarByClass(DBDriver.class);//mapper類job.setMapperClass(DBMapper.class);//輸出key和輸出value的類型job.setOutputKeyClass(LongWritable.class);job.setOutputValueClass(Text.class);//輸出的文件路徑FileOutputFormat.setOutputPath(job, new Path("E://mysqlOut"));//因為沒有寫Reduce類所以設(shè)置ReduceTask數(shù)為0job.setNumReduceTasks(0);//sql語句job.setInputFormatClass(DBInputFormat.class);DBInputFormat.setInput(job, GoodsBean.class, "select goodsId,goodsSn,goodsName,marketPrice,shopPrice,saleNum from itheima_goods", "select count(*) from itheima_goods");boolean b = job.waitForCompletion(true);//退出System.exit(b ? 0 : 1);} }

下一篇: 寫入數(shù)據(jù)庫操作鏈接.

MapReduce 數(shù)據(jù)庫表

CREATE TABLE `goods` (`goodsId` bigint NOT NULL AUTO_INCREMENT COMMENT '商品id',`goodsSn` varchar(20) NOT NULL COMMENT '商品編號',`goodsName` varchar(200) NOT NULL COMMENT '商品名稱',`marketPrice` decimal(11,2) NOT NULL DEFAULT '0.00' COMMENT '市場價',`shopPrice` decimal(11,2) NOT NULL DEFAULT '0.00' COMMENT '門店價',`saleNum` int NOT NULL DEFAULT '0' COMMENT '總銷售量',PRIMARY KEY (`goodsId`) ) ENGINE=InnoDB AUTO_INCREMENT=115909 DEFAULT CHARSET=utf8mb3;

附上100條數(shù)據(jù)

INSERT INTO `goods` VALUES ('100101', '155083444927602', '四川果凍橙6個約180g/個', '45.00', '38.00', '0'); INSERT INTO `goods` VALUES ('100102', '155083493976803', '鮮豐水果秭歸臍橙中華紅橙9斤家庭裝單果130g—220g4500g', '70.00', '59.90', '0'); INSERT INTO `goods` VALUES ('100103', '155083608286049', '紅顏奶油草莓約重500g/15-20顆新鮮水果', '60.00', '49.90', '0'); INSERT INTO `goods` VALUES ('100104', '155083629928386', '智利進口車?yán)遄覬級1磅裝果徑約26-28mm新鮮水果', '100.00', '79.00', '0'); INSERT INTO `goods` VALUES ('100105', '155083696154773', 'Zespri佳沛新西蘭陽光金奇異果6個裝經(jīng)典36-39號果單果重約80-100g新鮮水果', '700.00', '59.90', '0'); INSERT INTO `goods` VALUES ('100106', '155084645314975', '花果山云南甜玉米水果玉米約2.5kg6-9根燒烤食材新鮮蔬菜', '35.00', '26.80', '0'); INSERT INTO `goods` VALUES ('100107', '155084660300312', '福建六鰲紅薯蜜薯地瓜2.5kg紅蜜薯單果重約150g-500g新鮮蔬菜', '30.00', '26.90', '0'); INSERT INTO `goods` VALUES ('100108', '155084691120340', '花果山17.5°橙3kg裝鉑金果新鮮水果禮盒', '50.00', '45.90', '0'); INSERT INTO `goods` VALUES ('100109', '155084738764170', '花果山甜鴨梨6-8個凈重約1.8kg新鮮水果', '35.00', '24.90', '0'); INSERT INTO `goods` VALUES ('100110', '155084761584559', '花果山煙臺紅富士蘋果12個凈重2.1kg以上單果160-190g一二級混裝自營水果', '41.00', '34.90', '0'); INSERT INTO `goods` VALUES ('100111', '155084782094837', '黃冠梨4個單果約250g-300g新鮮水果', '35.00', '299.00', '0'); INSERT INTO `goods` VALUES ('100112', '155084798412816', '花果山球生菜約400g火鍋食材新鮮蔬菜', '13.00', '10.80', '0'); INSERT INTO `goods` VALUES ('100113', '155084813931944', '上海青小油菜小青菜約400g火鍋食材新鮮蔬菜', '10.00', '7.90', '0'); INSERT INTO `goods` VALUES ('100114', '155084830858516', '花果山香菇蘑菇約400g火鍋食材新鮮蔬菜', '20.00', '15.80', '0'); INSERT INTO `goods` VALUES ('100115', '155084851974657', '花果山農(nóng)家新鮮大蒜2500g裝蒜頭', '22.00', '20.50', '0'); INSERT INTO `goods` VALUES ('100116', '155084880330968', '花果山杭椒約300g自營蔬菜', '10.00', '8.50', '0'); INSERT INTO `goods` VALUES ('100117', '155084916882945', '廣西荔浦砂糖桔砂糖橘甜橘子2.5kg禮盒裝新鮮水果', '58.00', '49.90', '0'); INSERT INTO `goods` VALUES ('100118', '155084929348686', '紅旗坡新疆阿克蘇蘋果果徑80mm-85mm約5kg新鮮水果', '120.00', '99.00', '0'); INSERT INTO `goods` VALUES ('100119', '155084948775196', '進口牛油果6個裝單果重約130-180g新鮮水果', '50.00', '39.90', '0'); INSERT INTO `goods` VALUES ('100120', '155084971919785', '九家原越南進口大玉芒4斤單果約180-230g', '25.00', '19.90', '0'); INSERT INTO `goods` VALUES ('100121', '155085021605526', '三只松鼠開口松子休閑零食堅果炒貨東北手剝紅松子100g', '45.00', '39.90', '0'); INSERT INTO `goods` VALUES ('100122', '155085043559648', '三只松鼠草莓干蜜餞果干果脯辦公室休閑零食水果干106g/袋', '30.00', '25.90', '0'); INSERT INTO `goods` VALUES ('100123', '155085057770371', '三只松鼠堅果炒貨零食特產(chǎn)每日堅果開心果100g/袋', '45.00', '39.90', '0'); INSERT INTO `goods` VALUES ('100124', '155085070063201', '滿199減120三只松鼠黃桃干106g休閑零食特產(chǎn)果脯蜜餞水果干桃肉紅色', '25.00', '19.90', '0'); INSERT INTO `goods` VALUES ('100125', '155085092481916', '三只松鼠無核白葡萄干280gx2袋零食蜜餞果干新疆特產(chǎn)提子干', '50.00', '42.70', '0'); INSERT INTO `goods` VALUES ('100126', '155085103438739', '三只松鼠休閑零食特產(chǎn)混合什錦果蔬菜干蔬果干田園果蔬脆100g/袋', '25.00', '21.90', '0'); INSERT INTO `goods` VALUES ('100127', '155085120796574', '三只松鼠黃秋葵脆秋葵干健康營養(yǎng)休閑零食清新即食嘎嘣清脆蔬菜干40g/袋', '20.00', '16.90', '0'); INSERT INTO `goods` VALUES ('100128', '155085134595386', '三只松鼠鳳梨干菠蘿干蜜餞果干零食果脯106g/袋', '23.00', '19.80', '0'); INSERT INTO `goods` VALUES ('100129', '155085152523279', '三只松鼠堅果特產(chǎn)休閑零食核桃肉琥珀核桃仁165g/袋', '33.00', '29.90', '0'); INSERT INTO `goods` VALUES ('100130', '155085164041384', '三只松鼠肩扛大零食巨型堅果零食大禮包3195g30包抖音一箱零食豬飼料送女友年貨禮盒混合口味', '200.00', '189.00', '0'); INSERT INTO `goods` VALUES ('100131', '155085192227531', '三只松鼠堅果炒貨零食特產(chǎn)即食板栗開口甘栗120g/袋', '24.00', '16.90', '0'); INSERT INTO `goods` VALUES ('100132', '155085237245488', '同仁堂（TRT）【同仁堂旗艦店】同仁堂紅糖姜茶10g*12袋*1盒速溶姜茶大姨媽可飲用', '100.00', '89.90', '0'); INSERT INTO `goods` VALUES ('100133', '155085249050414', '同仁堂（TRT）阿膠糕225g禮盒裝即食阿膠糕原味阿膠固元膏北京同仁堂', '130.00', '106.00', '0'); INSERT INTO `goods` VALUES ('100134', '155085262909301', '同仁堂TRT玫瑰花玫瑰花茶花草茶80g/瓶', '40.00', '35.00', '0'); INSERT INTO `goods` VALUES ('100135', '155085275777627', '同仁堂牛蒡茶170g', '50.00', '45.00', '0'); INSERT INTO `goods` VALUES ('100136', '155085295660683', '同仁堂(TRT)玫瑰花茶45g', '40.00', '35.00', '0'); INSERT INTO `goods` VALUES ('100137', '155085311727405', '北京同仁堂中華老字號黑果枸杞袋泡茶（代用茶）2g/袋*30', '170.00', '159.00', '0'); INSERT INTO `goods` VALUES ('100138', '155085316644196', '同仁堂（TRT）黑苦蕎茶273g(7g*39袋)', '68.00', '57.00', '0'); INSERT INTO `goods` VALUES ('100139', '155085327982623', '同仁堂陳皮茶120g', '30.00', '27.90', '0'); INSERT INTO `goods` VALUES ('100140', '155085358959309', '芙麗芳絲freeplus洗面奶女深層清潔泡沫氨基酸溫和潔面乳男專柜特價凈潤洗面霜100g敏感肌適用', '140.00', '120.00', '0'); INSERT INTO `goods` VALUES ('100141', '155085380178651', '資生堂珊珂（SENKA）綿潤泡沫潔面乳120g（日本進口洗面奶深層清潔清爽滋養(yǎng)男女適用）', '56.00', '45.00', '0'); INSERT INTO `goods` VALUES ('100142', '155085391329574', '資生堂水之密語（AQUAIR）凈澄水活(倍潤型)洗護套裝(洗發(fā)水600ml*2+護發(fā)素600ml*2)', '145.00', '139.00', '0'); INSERT INTO `goods` VALUES ('100143', '155085403436449', '夏依女性專用洗液237ml（敏感肌膚）*2支裝護理液私處私密護理美國原裝進口', '100.00', '89.00', '0'); INSERT INTO `goods` VALUES ('100144', '155085421243310', '婦炎潔婦科洗液女性私處護理液伊斯蜜粉嫩縮陰緊致私密護理凝膠潤滑油劑潔陰洗陰內(nèi)道沖洗器洗陰液私密洗液200ml+洗液380ml送沖洗器2個', '89.00', '75.00', '0'); INSERT INTO `goods` VALUES ('100145', '155085444960210', '蜜絲佛陀（MaxFactor）無暇持妝隔離霜30mlSPF20（彩妝妝前乳防曬裸妝控油持久）', '110.00', '99.00', '0'); INSERT INTO `goods` VALUES ('100146', '155085452741648', '謎尚（MISSHA）魅力潤顏修容霜SPF42/PA+++[21號]50ml（氣墊bb霜粉底遮瑕保濕防曬）（新老包裝隨機發(fā)貨）', '120.00', '99.00', '0'); INSERT INTO `goods` VALUES ('100147', '155085467032901', '歐萊雅（L\'OREAL）男士水能護膚品洗面奶套裝面霜補水保濕滋潤膚乳液擦臉油護臉?biāo)駨姖櫵?#43;酷爽水凝露120ml', '120.00', '89.00', '0'); INSERT INTO `goods` VALUES ('100148', '155085509783922', '五糧液52度普五500ML（新老包裝隨機發(fā)貨）', '1200.00', '1050.00', '0'); INSERT INTO `goods` VALUES ('100149', '155085531427345', '瀘州老窖國窖157352度濃香型白酒500ml（百年品牌瀘州老窖榮譽出品）', '1100.00', '969.00', '0'); INSERT INTO `goods` VALUES ('100150', '155085549028442', '劍南春水晶劍52度單瓶裝白酒500ml口感濃香型', '1000.00', '898.00', '0'); INSERT INTO `goods` VALUES ('100151', '155085560443863', '長城（GreatWall）紅酒特選5年橡木桶解百納干紅葡萄酒整箱裝750ml*6瓶', '500.00', '399.00', '0'); INSERT INTO `goods` VALUES ('100152', '155085570890755', '法國進口紅酒拉菲（LAFITE）傳奇波爾多干紅葡萄酒整箱裝750ml*6瓶（ASC）', '650.00', '498.00', '0'); INSERT INTO `goods` VALUES ('100153', '155085593740896', '法國原瓶進口紅酒凱旋干紅葡萄酒禮盒750ml整箱6支裝', '305.00', '279.00', '0'); INSERT INTO `goods` VALUES ('100154', '155085607093157', '送2香檳杯尚尼酒莊魔幻云星空酒起泡葡萄酒4支組合整箱火焰酒女士鐘愛的配制甜紅酒氣泡果酒4*750ml', '200.00', '158.00', '0'); INSERT INTO `goods` VALUES ('100155', '155088550521760', '勁牌勁酒中國勁酒35度1.5L', '125.00', '99.00', '0'); INSERT INTO `goods` VALUES ('100156', '155088572544924', '勁牌勁酒中國勁酒35度125ml*12瓶禮盒裝', '200.00', '179.00', '0'); INSERT INTO `goods` VALUES ('100157', '155088588416731', 'HE小酒JOY紀(jì)念版洋河+習(xí)酒+古井貢酒+勁牌+黃金酒大牌暢飲盒', '310.00', '279.00', '0'); INSERT INTO `goods` VALUES ('100158', '155088605646015', '洋河藍色經(jīng)典夢之藍M352度禮盒裝500ml*2瓶白酒口感綿柔濃香型', '15000.00', '1098.00', '0'); INSERT INTO `goods` VALUES ('100159', '155088623897765', '郎酒郎牌郎酒醬香型53度整箱裝500ml*6瓶', '1300.00', '1128.00', '0'); INSERT INTO `goods` VALUES ('100160', '155088647094285', '優(yōu)露清（Explution）油煙機清洗劑瓷磚浴室清潔劑廚房油污浴室玻璃清洗液廚衛(wèi)清潔套裝1L', '90.00', '89.70', '0'); INSERT INTO `goods` VALUES ('100161', '155088662493170', '優(yōu)露清（Explution）頑固污漬干洗劑羽絨服清洗劑衣服去油漬布藝沙發(fā)免水洗去污清潔劑1瓶裝', '87.50', '93.00', '0'); INSERT INTO `goods` VALUES ('100162', '155088679433830', '優(yōu)露清（Explution）便攜式去污筆去漬筆衣服免水洗去油漬干洗劑應(yīng)急去污魔法筆隨身攜帶一盒裝', '35.00', '26.90', '0'); INSERT INTO `goods` VALUES ('100163', '155088696774534', '優(yōu)露清【榜單商品】油污清潔劑油煙機清洗劑廚房去油污強力清潔劑瓷磚去油污凈500ml', '55.00', '49.90', '0'); INSERT INTO `goods` VALUES ('100164', '155088710371201', '威猛先生（MrMuscle）【領(lǐng)券滿65減50】家用廚衛(wèi)清潔組合套裝灶臺重油污凈油煙', '55.00', '46.90', '0'); INSERT INTO `goods` VALUES ('100165', '155088726129516', '立白茶籽洗潔精雙瓶省心裝1.45kg*2瓶', '29.00', '23.90', '0'); INSERT INTO `goods` VALUES ('100166', '155088744403257', '立白檸檬去油洗潔精（清新檸檬）1.5kg/瓶', '15.00', '13.50', '0');