python 機器學習管道

Below are the usual steps involved in building the ML pipeline:

以下是構建ML管道所涉及的通常步驟：

Import Data

匯入資料

Exploratory Data Analysis (EDA)

探索性數據分析(EDA)

Missing Value Imputation

缺失價值估算

Outlier Treatment

離群值處理

Feature Engineering

特征工程

Model Building

建筑模型

Feature Selection

功能選擇

Model Interpretation

模型解釋

Save the model

保存模型

Model Deployment *

模型部署*

問題陳述和數據獲取 (Problem Statement and Getting the Data)

I’m using a relatively bigger and more complicated data set to demonstrate the process. Refer to the Kaggle competition — IEEE-CIS Fraud Detection.

我正在使用相對較大和較復雜的數據集來演示該過程。 請參閱Kaggle競賽-IEEE-CIS欺詐檢測 。

Navigate to Data Explorer and you will see something like this:

導航到“數據資源管理器”，您將看到以下內容：

Select train_transaction.csv and it will show you a glimpse of what data looks like. Click on the download icon highlighted by a red arrow to get the data.

選擇train_transaction.csv ，它將向您展示數據的外觀。 單擊以紅色箭頭突出顯示的下載圖標以獲取數據。

Other than the usual library import statements, you will need to check for 2 additional libraries —

除了通常的庫導入語句外，您還需要檢查另外兩個庫-

pip安裝pyarrow (pip install pyarrow)

點安裝fast_ml (pip install fast_ml)

主要亮點 (Key Highlights)

This is the first article in the series of building machine learning pipeline. In this article, we will focus on optimizations around importing the data in Jupyter notebook and executing things faster.

這是構建機器學習管道系列文章中的第一篇。 在本文中，我們將專注于圍繞Jupyter Notebook中導入數據并更快地執行操作的優化。

There are 3 key things to note in this article —

本文中有3個關鍵要注意的地方-

Python zipfile

Python壓縮檔

Reducing the memory usage of the dataset

減少數據集的內存使用量

A faster way of saving/loading working datasets

保存/加載工作數據集的更快方法

1：導入數據 (1: Import Data)

After you have downloaded the zipped file. It is so much better to use python to unzip the file.

下載壓縮文件后。 使用python解壓縮文件要好得多。

Tip 1: Use function from python zipfile library to unzip the file.

提示1： 使用python zipfile庫中的函數來解壓縮文件。

import zipfilewith zipfile.ZipFile('train_transaction.csv.zip', mode='r') as zip_ref:
zip_ref.extractall('data/')

This will create a folder data and unzip the CSV file train_transaction.csv in that folder.

這將創建一個文件夾data ，并將CSV文件train_transaction.csv解壓縮到該文件夾??中。

We will use pandas read_csv method to load the data set into Jupyter notebook.

我們將使用pandas read_csv方法將數據集加載到Jupyter筆記本中。

%time trans = pd.read_csv('train_transaction.csv')df_size = trans.memory_usage().sum() / 1024**2
print(f'Memory usage of dataframe is {df_size} MB')print (f'Shape of dataframe is {trans.shape}')---- Output ----
CPU times: user 23.2 s, sys: 7.87 s, total: 31 s
Wall time: 32.5 sMemory usage of dataframe is 1775.1524047851562 MB
Shape of dataframe is (590540, 394)

This data is ~1.5 GB with more than half a million rows.

該數據約為1.5 GB，具有超過一百萬行。

Tip 2: We will use a function from fast_ml to reduce this memory usage.

提示2： 我們將使用fast_ml中的函數來減少此內存使用量。

from fast_ml.utilities import reduce_memory_usage%time trans = reduce_memory_usage(trans, convert_to_category=False)---- Output ----
Memory usage of dataframe is 1775.15 MB
Memory usage after optimization is: 542.35 MB
Decreased by 69.4%CPU times: user 2min 25s, sys: 2min 57s, total: 5min 23s
Wall time: 5min 56s

This step took almost 5 mins but it has reduced the memory size by almost 70% that’s quite a significant reduction

此步驟花費了將近5分鐘，但已將內存大小減少了將近70％，這是一個相當大的減少

For further analysis, we will create a sample dataset of 200k records just so that our data processing steps won’t take a long time to run.

為了進行進一步的分析，我們將創建一個包含20萬條記錄的樣本數據集，以便我們的數據處理步驟不會花費很長時間。

# Take a sample of 200k records
%time trans = trans.sample(n=200000)#reset the index because now index would have shuffled
trans.reset_index(inplace = True, drop = True)df_size = trans.memory_usage().sum() / 1024**2
print(f'Memory usage of sample dataframe is {df_size} MB')---- Output ----
CPU times: user 1.39 s, sys: 776 ms, total: 2.16 s
Wall time: 2.43 s
Memory usage of sample dataframe is 185.20355224609375 MB

Now, we will save this in our local drive — CSV Format

現在，我們將其保存在本地驅動器中-CSV格式

import osos.makedirs('data', exist_ok=True) trans.to_feather('data/train_transaction_sample')

Tip 3: use the feather format instead of csv

提示3： 使用羽毛格式而不是csv

import osos.makedirs('data', exist_ok=True)
trans.to_feather('data/train_transaction_sample')

Once you load the data from these 2 sources you will observe the significant performance improvements.

從這兩個來源加載數據后，您將觀察到顯著的性能改進。

Load the saved sample data — CSV Format

加載保存的樣本數據-CSV格式

%time trans = pd.read_csv('data/train_transaction_sample.csv')df_size = tras.memory_usage().sum() / 1024**2
print(f'Memory usage of dataframe is {df_size} MB')
print (f'Shape of dataframe is {trans.shape}')---- Output ----
CPU times: user 7.37 s, sys: 1.06 s, total: 8.42 s
Wall time: 8.5 sMemory usage of dataframe is 601.1964111328125 MB
Shape of dataframe is (200000, 394)

Load the saved sample data — Feather Format

加載保存的樣本數據—羽毛格式

%time trans = pd.read_feather('tmp/train_transaction_sample')df_size = trans.memory_usage().sum() / 1024**2
print(f'Memory usage of dataframe is {df_size} MB')
print (f'Shape of dataframe is {trans.shape}')---- Output ----
CPU times: user 1.32 s, sys: 930 ms, total: 2.25 s
Wall time: 892 msMemory usage of dataframe is 183.67779541015625 MB
Shape of dataframe is (200000, 394)

注意這里兩件事： (Notice 2 things here :)

i. The amount of time it took to load the CSV file is almost 10 times the time it took for loading feather format data.

一世。 加載CSV文件所花費的時間幾乎是加載羽毛格式數據所花費的時間的10倍。

ii. Size of the data set loaded was retained in feather format whereas in CSV format the data set is again consuming high memory and we will have to run the reduce_memory_usage function again.

ii。 加載的數據集的大小以羽毛格式保留，而在CSV格式中，數據集再次占用大量內存，我們將不得不再次運行reduce_memory_usage函數。

結束語： (Closing Notes:)

• Please feel free to write your thoughts/suggestions/feedback.
  請隨時寫下您的想法/建議/反饋。
• We will use the new sample data set what we created for our further analysis.
  我們將使用我們創建的新樣本數據集進行進一步分析。
• We will talk about Exploratory Data Analysis in our next article.
  在下一篇文章中，我們將討論探索性數據分析。

• Github link

  Github鏈接

翻譯自: https://towardsdatascience.com/building-a-machine-learning-pipeline-part-1-b19f8c8317ae

python 機器學習管道

總結

以上是生活随笔為你收集整理的python 机器学习管道_构建机器学习管道-第1部分的全部內容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網站內容還不錯，歡迎將生活随笔推薦給好友。