接上一篇：(企業內部) ETL異構數據源Datax_部署前置環境_01
https://gblfy.blog.csdn.net/article/details/118081253

文章目錄

• • • • • 一、直接下載DataX
        • 二、下載DataX源碼，自己編譯
        • • 2.1.下載DataX源碼
          • 2.2. 通過maven打包
        • 三、配置示例
        • • 3.1. 創建作業的配置文件（json格式）
          • 3.2. 根據模板配置json
          • 3.3. 執行測試

一、直接下載DataX

：DataX下載地址

下載后解壓至本地某個目錄，進入bin目錄，即可運行同步作業：

tar zxvf datax.tar.gz cd {YOUR_DATAX_HOME}/bin python datax.py {YOUR_JOB.json}

自檢腳本：

python {YOUR_DATAX_HOME}/bin/datax.py {YOUR_DATAX_HOME}/job/job.json

二、下載DataX源碼，自己編譯

DataX源碼

2.1.下載DataX源碼

git clone git@github.com:alibaba/DataX.git

2.2. 通過maven打包

cd {DataX_source_code_home} mvn -U clean package assembly:assembly -Dmaven.test.skip=true

打包成功，日志顯示如下：

[INFO] BUILD SUCCESS [INFO] ----------------------------------------------------------------- [INFO] Total time: 08:12 min [INFO] Finished at: 2021-12-13T16:26:48+08:00 [INFO] Final Memory: 133M/960M [INFO] -----------------------------------------------------------------

打包成功后的DataX包位于 {DataX_source_code_home}/target/datax/datax/ ，結構如下：

bin conf job lib log log_perf plugin script tmp

三、配置示例

從stream讀取數據并打印到控制臺

3.1. 創建作業的配置文件（json格式）

可以通過命令查看配置模板： python datax.py -r {YOUR_READER} -w {YOUR_WRITER} cd {YOUR_DATAX_HOME}/bin python datax.py -r streamreader -w streamwriter---------------------------------------------------------- # 查看常用作業的配置文件模板python datax.py -r streamreader -w streamwriter python datax.py -r oraclereader -w mysqlwriter python datax.py -r mysqlreader -w oraclewriter控制臺輸出 ```bash DataX (DATAX-OPENSOURCE-3.0), From Alibaba ! Copyright (C) 2010-2017, Alibaba Group. All Rights Reserved.Please refer to the streamreader document:https://github.com/alibaba/DataX/blob/master/streamreader/doc/streamreader.md Please refer to the streamwriter document:https://github.com/alibaba/DataX/blob/master/streamwriter/doc/streamwriter.md Please save the following configuration as a json file and usepython {DATAX_HOME}/bin/datax.py {JSON_FILE_NAME}.json to run the job.{"job": {"content": [{"reader": {"name": "streamreader", "parameter": {"column": [], "sliceRecordCount": ""}}, "writer": {"name": "streamwriter", "parameter": {"encoding": "", "print": true}}}], "setting": {"speed": {"channel": ""}}} }

3.2. 根據模板配置json

vim stream2stream.json添加內容如下：{"job": {"content": [{"reader": {"name": "streamreader","parameter": {"sliceRecordCount": 10,"column": [{"type": "long","value": "10"},{"type": "string","value": "hello，你好，世界-DataX"}]}},"writer": {"name": "streamwriter","parameter": {"encoding": "UTF-8","print": true}}}],"setting": {"speed": {"channel": 5}}} }

3.3. 執行測試

啟動DataX

cd {YOUR_DATAX_DIR_BIN} python datax.py ./stream2stream.json 同步結束，顯示日志如下：... 2021-06-23 09:43:14.869 [job-0] INFO StandAloneJobContainerCommunicator - Total 50 records, 950 bytes | Speed 95B/s, 5 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 0.000s | All Task WaitReaderTime 0.000s | Percentage 100.00% 2021-06-23 09:43:14.872 [job-0] INFO JobContainer - 任務啟動時刻 : 2021-06-23 09:43:04 任務結束時刻 : 2021-06-23 09:43:14 任務總計耗時 : 10s 任務平均流量 : 95B/s 記錄寫入速度 : 5rec/s 讀出記錄總數 : 50 讀寫失敗總數 : 0

總結

以上是生活随笔為你收集整理的ETL异构数据源Datax_工具部署_02的全部內容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網站內容還不錯，歡迎將生活随笔推薦給好友。