During COVID19 quarantine I decided to build my own implementation of a mask-detector able to detect whether a person is wearing a mask in images or videos just for fun.

在COVID19隔離期間，我決定構(gòu)建自己的口罩檢測器實現(xiàn)，該檢測器能夠檢測人是否只是出于娛樂目的在圖像或視頻中戴著口罩。

Like every Machine Learning project, the very first step is to collect the necessary data. As we’re trying to build a mask detector model which should be able to output “mask” or “no mask”, given an input face image, so we need to collect images of people wearing and not wearing a mask.

像每個機器學(xué)習(xí)項目一樣，第一步是收集必要的數(shù)據(jù)。 由于我們正在嘗試構(gòu)建一個遮罩檢測器模型，在給定輸入面部圖像的情況下，該模型應(yīng)該能夠輸出“遮罩”或“無遮罩”，因此我們需要收集佩戴和未佩戴遮罩的人的圖像。

收集資料 (Collecting the data)

I just came to ask all my friends to send me a selfie where they were wearing a mask and another where they were not wearing it. I was able to collect around 200 images which seems to be very poor for training an accurate Machine Learning model, however the results were quite acceptable.

我只是來問我所有的朋友，給我發(fā)一張自帶自拍照的自拍照，他們戴著口罩，另一個不戴口罩。 我能夠收集大約200張圖像，這對于訓(xùn)練準(zhǔn)確的機器學(xué)習(xí)模型來說似乎很差，但是結(jié)果還是可以接受的。

構(gòu)建解決方案 (Structuring the solution)

To build a mask detector let’s first split the problem into 2 main steps:

要構(gòu)建遮罩檢測器，我們首先將問題分為兩個主要步驟：

1. Given an input image we need to detect faces on it, this is a task called “Object Detection” in the world of Computer Vision. Object Detection is the task of detecting object positions and their types over images as the example below:

1.給定輸入圖像，我們需要檢測其上的面部，這是計算機視覺界中稱為“對象檢測”的任務(wù)。 對象檢測是檢測圖像上的對象位置及其類型的任務(wù)，如下例所示：

In our problem we need to detect only faces and output their bounding boxes delimiting their positions, so we can pass them to the next step:

在我們的問題中，我們只需要檢測面Kong并輸出界定其位置的邊界框，因此可以將其傳遞到下一步：

2. Given one or more face images, we need to classify them into “mask” or “no mask”. In the Machine Learning vocabulary this is called “binary classification”, where we need to classify some input data into 2 possible classes (in this case [“mask”, “no mask”]). Our input data will be the RGB image representation of human faces.

2.給定一個或多個面部圖像，我們需要將其分類為“遮罩”或“無遮罩”。 在機器學(xué)習(xí)詞匯中，這被稱為“二進制分類”，在這里我們需要將一些輸入數(shù)據(jù)分類為2種可能的類別(在這種情況下為[mask]，“ no mask”)。 我們的輸入數(shù)據(jù)將是人臉的RGB圖像表示。

So, given the 2 steps mentioned before, here we’re building a pipeline of processing, the first step takes an input image and outputs the bounding boxes of human faces found in that image, and the second step takes that cropped face images delimited by the bounding boxes and classifies them into “mask” or “nomask”.

因此，考慮到前面提到的兩個步驟，我們在這里建立處理流程，第一步獲取輸入圖像并輸出在該圖像中找到的人臉邊界框，第二步獲取由邊界框并將其分類為“遮罩”或“無遮罩”。

Let’s start by talking about the second step: “The classification problem” as it is the main focus of this article.

讓我們首先討論第二步：“分類問題”，因為這是本文的主要重點。

關(guān)鍵概念 (Key Concepts)

Transfer Learning: Transfer learning is a research problem in machine learning that focuses on storing knowledge gained while solving one problem and applying it to a different but related problem. For example, knowledge gained while learning to recognize cars could apply when trying to recognize trucks. Wikipedia

轉(zhuǎn)移學(xué)習(xí) ：轉(zhuǎn)移學(xué)習(xí)是機器學(xué)習(xí)中的一個研究問題，著重于存儲在解決一個問題時獲得的知識并將其應(yīng)用于另一個但相關(guān)的問題。 例如，在嘗試識別卡車時可以應(yīng)用在學(xué)習(xí)識別汽車時獲得的知識。 維基百科

Data Augmentation: Data augmentation is a strategy that enables practitioners to significantly increase the diversity of data available for training models, without actually collecting new data. Data augmentation techniques such as cropping, padding, and horizontal flipping are commonly used to train large neural networks

數(shù)據(jù)增強 ：數(shù)據(jù)增強是一種策略，使從業(yè)人員可以顯著增加可用于訓(xùn)練模型的數(shù)據(jù)的多樣性，而無需實際收集新數(shù)據(jù) 。 諸如裁剪，填充和水平翻轉(zhuǎn)之類的數(shù)據(jù)增強技術(shù)通常用于訓(xùn)練大型神經(jīng)網(wǎng)絡(luò)

預(yù)處理人臉圖像 (Preprocessing Face Images)

In order to build the mask detector model which takes faces as input and detects masks i needed to crop the faces from the collected images. Am not copying/pasting the code here to avoid making this article too large, but you can find the code here: labelling_images.ipynb

為了建立面具檢測器模型，該模型將人臉作為輸入并檢測面具，我需要從收集的圖像中裁剪人臉。 不要在此處復(fù)制/粘貼代碼以避免使本文過大，但是您可以在此處找到代碼： labelling_images.ipynb

訓(xùn)練模型 (Training the model)

Let’s start by coding step by step the model training algorithm. I used Python and Jupyter notebooks in a Google Colab environment with GPU, but you can also run the code in whatever python environment you prefer.

讓我們開始逐步編碼模型訓(xùn)練算法。 我在帶有GPU的Google Colab環(huán)境中使用了Python和Jupyter筆記本，但是您也可以在自己喜歡的任何python環(huán)境中運行代碼。

The training notebook can be found here in my Github repository if you prefer to see the entire code directly: https://github.com/fnandocontreras/mask-detector/blob/master/training_model.ipynb

如果您希望直接查看整個代碼，可以在我的Github存儲庫中找到培訓(xùn)筆記本： https : //github.com/fnandocontreras/mask-detector/blob/master/training_model.ipynb

Let’s import some dependencies:

讓我們導(dǎo)入一些依賴項：

import tensorflow as tfimport pathlibimport numpy as npimport IPython.display as displayfrom PIL import Imageimport matplotlib.pyplot as pltimport osfrom keras.preprocessing.image import ImageDataGeneratorfrom keras.callbacks import EarlyStopping

I’m using Google drive to store the training images, but feel free to use your local machine if you want to run your code locally.Let’s mount the Google drive storage into the notebook and set the path to the collected images to base_path and get a directory info object with pathlib as follows:

我正在使用Google驅(qū)動器來存儲訓(xùn)練圖像，但是如果要在本地運行代碼，請隨時使用本地計算機。讓我們將Google驅(qū)動器存儲設(shè)備安裝到筆記本中，并將收集到的圖像的路徑設(shè)置為base_path并獲取具有pathlib的目錄信息對象，如下所示：

from google.colab import drive
drive.mount(‘/content/drive’)base_path = '/content/drive/My Drive/Colab Notebooks/mask-detector/'
data_dir_collected = pathlib.Path(os.path.join(base_path, 'data/training'))
data_dir_test = pathlib.Path(os.path.join(base_path, 'data/test'))image_count_collected = len(list(data_dir_collected.glob('**/*.jpg')))
test_images_count = len(list(data_dir_test.glob('**/*.jpg')))
print('images collected', image_count_collected)
print('test images', test_images_count)

In my google drive the images are stored with the following folder structure:A folder named “mask” containing all the images with masksA folder named “nomask” containing all the images without masks

在我的Google驅(qū)動器中，圖像使用以下文件夾結(jié)構(gòu)存儲：名為“ mask”的文件夾，其中包含所有帶有掩碼的圖像；名為“ nomask”的文件夾，其中包含所有沒有掩碼的圖像

let’s check that we are loading the images in the right path:

讓我們檢查一下我們是否在正確的路徑中加載圖像：

CLASS_NAMES = np.array([item.name for item in data_dir_collected.glob('*')])
print(CLASS_NAMES)

this should print: [‘nomask’ ‘mask’]

這應(yīng)該打印：['nomask''mask']

let’s define some constants and create the tensorflow image generator that will load images and feed them to the model training process:

讓我們定義一些常量并創(chuàng)建tensorflow圖像生成器，該生成器將加載圖像并將其提供給模型訓(xùn)練過程：

IMG_WIDTH = 160
IMG_HEIGHT = 160
IMG_SIZE = (IMG_WIDTH, IMG_HEIGHT)image_generator = ImageDataGenerator(rescale=1./255)
train_data_gen = image_generator.flow_from_directory(directory=str(data_dir_collected), shuffle=True, target_size=IMG_SIZE, classes = list(CLASS_NAMES))

Let’s show the images being loaded by the training data generator:

讓我們展示訓(xùn)練數(shù)據(jù)生成器正在加載的圖像：

def show_batch(image_batch, label_batch):
plt.figure(figsize=(10,10))
for n in range(25):
ax = plt.subplot(5,5,n+1)
plt.imshow(image_batch[n])
plt.title(CLASS_NAMES[label_batch[n]==1][0].title())
plt.axis('off')
image_batch, label_batch = next(train_data_gen)
show_batch(image_batch, label_batch)

Let’s define some auxiliar methods that will be useful for data preprocessing:

讓我們定義一些對數(shù)據(jù)預(yù)處理有用的輔助方法：

def get_label(file_path):
# convert the path to a list of path components
parts = tf.strings.split(file_path, os.path.sep)
# The second to last is the class-directory
return parts[-2] == CLASS_NAMESdef decode_img(img):
# convert the compressed string to a 3D uint8 tensor
img = tf.image.decode_jpeg(img, channels=3)
# Use `convert_image_dtype` to convert to floats in the [0,1] range.
img = tf.image.convert_image_dtype(img, tf.float32)
# resize the image to the desired size.
return tf.image.resize(img, [IMG_WIDTH, IMG_HEIGHT])def process_path(file_path):
label = get_label(file_path)
# load the raw data from the file as a string
img = tf.io.read_file(file_path)
img = decode_img(img)
return img, label

Let’s load now all the images from the storage, including the test images that we will use to evaluate the model:

現(xiàn)在從存儲中加載所有圖像，包括將用于評估模型的測試圖像：

list_ds_collected = tf.data.Dataset.list_files(str(data_dir_collected/'*/*'))
list_ds_test = tf.data.Dataset.list_files(str(data_dir_test/'*/*'))

Let’s now apply the preprocessing functions defined before to the loaded images:

現(xiàn)在讓我們將之前定義的預(yù)處理功能應(yīng)用于加載的圖像：

train_ds = list_ds_collected.map(process_path)
validation_ds = list_ds_collected.map(process_path)

Let’s define an ImageDataGenerator, this will define a generator class that will perform Data Augmentation over the loaded images, this will allow us to train the model over a bigger distribution of data, it performs some operations over the images like: zoom, rotation, horizontal flip, etc.

讓我們定義一個ImageDataGenerator，這將定義一個生成器類，它將對加載的圖像執(zhí)行數(shù)據(jù)增強，這將使我們能夠在更大的數(shù)據(jù)分布上訓(xùn)練模型，并對圖像執(zhí)行一些操作，例如：縮放，旋轉(zhuǎn)，水平翻轉(zhuǎn)等

Data augmentation is useful in many cases, when we don’t have enough training data or in the cases when the model is overfitting the training dataset. Intuitively we’re training a model to predict whether people are wearing masks, and our training data containing faces might be augmented by applying transformations to every face picture. In fact these transformations don’t modify the resulting class (“mask”, “nomask”), so let’s go for it.

在許多情況下，當(dāng)我們沒有足夠的訓(xùn)練數(shù)據(jù)時，或者在模型過度擬合訓(xùn)練數(shù)據(jù)集的情況下，數(shù)據(jù)增強很有用。 直觀地，我們正在訓(xùn)練一個模型來預(yù)測人們是否戴著口罩，并且可以通過將變換應(yīng)用于每張臉部圖片來增強包含臉部的訓(xùn)練數(shù)據(jù)。 實際上，這些轉(zhuǎn)換不會修改結(jié)果類(“ mask”，“ nomask”)，因此讓我們繼續(xù)吧。

def get_data_generator():
return ImageDataGenerator(
rotation_range=20,
zoom_range=0.10,
width_shift_range=0.1,
height_shift_range=0.1,
shear_range=0.15,
horizontal_flip=True,
fill_mode="nearest")

Let’s generate a batch of images containing all the collected/training and validation images and fit the image generator with them.

讓我們生成一批包含所有收集/訓(xùn)練和驗證圖像的圖像，并將它們與圖像生成器配合。

train_ds = train_ds.batch(image_count_collected)
validation_ds = validation_ds.batch(test_images_count)datagen = get_data_generator()
image_batch, label_batch = next(iter(train_ds))
datagen.fit(image_batch)

Now we can start building our Deep Learning model. As we mentioned before, we’re using Transfer Learning, which is the task of using a pre-trained model as part of our final model, this allows us to take advantage of the parameters learned by a general purpose computer vision model to build our model adapted to our requirements.

現(xiàn)在，我們可以開始構(gòu)建深度學(xué)習(xí)模型。 如前所述，我們正在使用轉(zhuǎn)移學(xué)習(xí)，這是將預(yù)先訓(xùn)練的模型用作最終模型的一部分的任務(wù)，這使我們能夠利用通用計算機視覺模型學(xué)到的參數(shù)來構(gòu)建我們的模型。符合我們要求的模型。

In the code block below we’re loading the MobileNET V2, you can find the research paper here if you want to know deeper details about the network architecture: https://arxiv.org/abs/1801.04381

在下面正在加載MobileNET V2的代碼塊中，如果您想了解有關(guān)網(wǎng)絡(luò)體系結(jié)構(gòu)的更詳細的信息，可以在這里找到研究論文： https : //arxiv.org/abs/1801.04381

IMG_SHAPE = (IMG_WIDTH, IMG_HEIGHT, 3)# Create the base model from the pre-trained model MobileNet V2
base_model = tf.keras.applications.MobileNetV2(input_shape=IMG_SHAPE,
include_top=False,
weights='imagenet')
base_model.trainable = False

We’re setting the model propery: “traianble” to False, because we don’t want to retrain that model.

我們正在設(shè)置模型屬性：“ traianble”為False，因為我們不想重新訓(xùn)練該模型。

So now that we have the base model let’s complete it by adding some layers that we’ll need for our prediction outputs:

現(xiàn)在，有了基本模型，讓我們通過添加預(yù)測輸出所需的一些層來完成它：

model = tf.keras.Sequential([
base_model,
GlobalAveragePooling2D(),
Dense(len(CLASS_NAMES), activation='softmax')
])
model.summary()

You should see the model summary as follows:

您應(yīng)該看到模型摘要如下：

Model: “sequential_3” _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= mobilenetv2_1.00_160 (Model) (None, 5, 5, 1280) 2257984 _________________________________________________________________ global_average_pooling2d_4 ( (None, 1280) 0 _________________________________________________________________ dense_4 (Dense) (None, 2) 2562 ================================================================= Total params: 2,260,546 Trainable params: 2,562 Non-trainable params: 2,257,984

We are adding to the base model’s output a Global Average Pooling layer and a Dense layer with a “softmax” activation function, please, find more details about this layers in the oficial Keras documentation:GlobalAveragePooling2DDense

我們將在基本模型的輸出中添加全局平均池化層和具有“ softmax”激活功能的密集層，請在官方Keras文檔中找到有關(guān)此層的更多詳細信息： GlobalAveragePooling2D Dense

The model’s output is “(None, 2)” where: “None” represents the batch size which might vary, and “2" is the size of the softmax layer, corresponding to the number of classes. A softmax layer outputs a probability distribution of the possible output classes.

模型的輸出為“(None，2)”，其中：“ None”表示可能變化的批次大小，“ 2”是softmax層的大小，對應(yīng)于類數(shù)。softmax層輸出概率分布可能的輸出類別。

Now that we have the model, let’s proceed to the training. Let’s iterate through the training images 50 times to generate the images that we will transform afterwards with our DataAugmentation generator:

現(xiàn)在我們有了模型，讓我們繼續(xù)進行培訓(xùn)。 讓我們遍歷訓(xùn)練圖像50次以生成圖像，然后使用DataAugmentation生成器將其轉(zhuǎn)換：

reps = 50
training_ds = train_ds.repeat(reps)
X_training, Y_training = next(iter(training_ds))
for X, Y in training_ds:
X_training = tf.concat([X, X_training], axis=0)
Y_training = tf.concat([Y, Y_training], axis=0)

Now we can compile and fit our model:We’re using here “Adam” as the optimisation algorithm during training, we are training the model for 10 epochs and we’re using early stopping which will stop the training process if the accuracy doesn’t get higher through 6 training iterations.

現(xiàn)在我們可以編譯和擬合模型了：我們在訓(xùn)練過程中使用“ Adam”作為優(yōu)化算法，我們正在訓(xùn)練模型10個紀(jì)元，并且我們正在使用提前停止功能，如果精度不高，則會停止訓(xùn)練過程。通過6次訓(xùn)練迭代獲得更高的結(jié)果。

model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])
initial_epochs = 10
earlystopping = EarlyStopping(monitor='val_loss', patience=6)
batch_size = 100history = model.fit(
datagen.flow(X_training, Y_training, batch_size=batch_size),
steps_per_epoch=len(X_training) / batch_size,
epochs=initial_epochs,
validation_data=validation_ds,
use_multiprocessing=False)

At the end of the training i had the following results:Epoch 10/10 98/97 [==============================] — 46s 466ms/step — loss: 0.0018 — accuracy: 1.0000 — val_loss: 6.2493e-04 — val_accuracy: 1.0000

在訓(xùn)練結(jié)束時，我得到了以下結(jié)果： 時代10/10 98/97 [=============================== ] — 46s 466ms / step —損耗：0.0018 —精度：1.0000 — val_loss：6.2493e-04 — val_accuracy：1.0000

Accuracy: 1.00 means the model predicted the test dataset with 100% of accuracy. It doesn’t mean though that the model performs the same in every data set. Remember that for this experiment i’am using only near of 200 images during training, which isn’t enough for the most of Machine Learning problems.

準(zhǔn)確性：1.00表示模型以100％的準(zhǔn)確性預(yù)測測試數(shù)據(jù)集。 這并不意味著該模型在每個數(shù)據(jù)集中執(zhí)行相同的操作。 請記住，對于這個實驗，我在訓(xùn)練過程中僅使用了近200張圖像，對于大多數(shù)機器學(xué)習(xí)問題而言，這還不夠。

In order to build a model that performs well and generalizes correctly in most of the cases we would need maybe thousands of images. However for the purpose of a proof of concept, it’s more than enough and the model seems to work very well on several people i used to test in real time using the webcam video stream.

為了建立一個在大多數(shù)情況下都能良好運行并正確歸納的模型，我們可能需要成千上萬張圖像。 但是，出于概念驗證的目的，它已綽綽有余，并且該模型似乎可以很好地用于我以前使用網(wǎng)絡(luò)攝像頭視頻流進行實時測試的幾個人。

Now that we have the trained model let’s save it and see how we can use it:

現(xiàn)在我們有了訓(xùn)練有素的模型，讓我們保存它，看看如何使用它：

model.save(os.path.join(base_path,'model.h5'))

Now you can load the saved model from another python program like this:

現(xiàn)在，您可以從另一個python程序中加載保存的模型，如下所示：

from tensorflow.keras.models import load_model
model = load_model(‘model.h5’)

Now that we have a mask detector model, we need the first part of our pipeline: “a face detector”. Object detection is one of the main tasks of Computer Vision. You can find a lot of pretrained models out there for object detection with sometimes several thousands of different classes. Here i used MTCNN which stands for “Multi Task Convolutional Neural Network”. You can find the github repository here: https://github.com/ipazc/mtcnn

現(xiàn)在我們有了一個遮罩檢測器模型，我們需要管道的第一部分：“面部檢測器”。 對象檢測是計算機視覺的主要任務(wù)之一。 您可以在其中找到許多用于對象檢測的預(yù)訓(xùn)練模型，有時甚至有數(shù)千種不同的類。 在這里，我使用了MTCNN，它代表“多任務(wù)卷積神經(jīng)網(wǎng)絡(luò)”。 您可以在這里找到github存儲庫： https : //github.com/ipazc/mtcnn

let’s import MTCNN and create an instance of face detector:

讓我們導(dǎo)入MTCNN并創(chuàng)建一個面部檢測器實例：

from mtcnn import MTCNN
detector = MTCNN()

Let’s load an image with opencv for testing:

讓我們使用opencv加載圖像進行測試：

import cv2
from matplotlib import pyplot as plt
test_img_path = os.path.join(data_dir_test, 'macron.jpg')
img = cv2.imread(test_img_path,cv2.IMREAD_COLOR)
plt.imshow(img[:,:,::-1])

Let’s run the face detector on the image:

讓我們在圖像上運行面部檢測器：

face_boxes = detector.detect_faces(img)
print(face_boxes)

[{ ‘box’: [463, 187, 357, 449], ‘confidence’: 0.9995754361152649, ‘keypoints’: { ‘left_eye’: (589, 346), ‘right_eye’: (750, 357), ‘nose’: (678, 442), ‘mouth_left’: (597, 525), ‘mouth_right’: (733, 537) }}]

[{'box'：[463，187，357，449]，'confidence'：0.9995754361152649，'keypoints'：{'left_eye'：(589，346)，'right_eye'：(750，357)，'nose' ：(678，442)，'mouth_left'：(597，525)，'mouth_right'：(733，537)}}]

We can see that the face is detected and we have all the relevant information like bounding box, and position of points of interest. In this case we only need the bounding box, which will help us to crop the image delimiting the face.

我們可以看到檢測到了人臉，并獲得了所有相關(guān)信息，例如邊界框和興趣點的位置。 在這種情況下，我們只需要邊界框，這將幫助我們裁剪界定人臉的圖像。

And now let’s see how the model performs with the french president:Am using some auxiliary functions for cropping and drawing faces that you can find in tools.py

現(xiàn)在讓我們來看一下該模型與法國總統(tǒng)的表現(xiàn)：使用一些輔助功能裁剪和繪制可在tools.py中找到的面Kong

face_boxes, faces = extract_faces(face_boxes, img)
preds = model.predict(tf.data.Dataset.from_tensors(faces))
probas = preds.max(axis=1)
y_preds = [CLASS_NAMES[c] for c in preds.argmax(axis=1)]
draw_boxes(img, face_boxes, (y_preds, probas))
plt.imshow(img[:,:,::-1])

And voilà: the model says the french president Macron is wearing a mask.

還有，模型：模特說法國總統(tǒng)馬克龍戴著口罩。

You can try it yourself with you own images, you can find the whole code and the trained model saved in Github: https://github.com/fnandocontreras/mask-detector

您可以使用自己的圖像自己嘗試，可以找到完整的代碼以及保存在Github中的經(jīng)過訓(xùn)練的模型： https : //github.com/fnandocontreras/mask-detector

You can run it in real time using opencv and your webcam, for details on how to run the program, please find instructions in readme.md

您可以使用opencv和網(wǎng)絡(luò)攝像頭實時運行它，有關(guān)如何運行該程序的詳細信息，請在readme.md中找到說明。

That’s all for this tutorial, remember this is just an experiment, it is not intended to be used in a real life environment because of its limitations. One important limitation is the fact that the face detector in many cases fails to detect masked faces, so it breaks the first step of the pipeline and it will fail to work as intended.

這只是本教程的全部內(nèi)容，請記住，這只是一個實驗，由于其局限性，不適合在現(xiàn)實生活中使用。 一個重要的限制是，在很多情況下，面部檢測器無法檢測到被遮蓋的面部，因此它中斷了流水線的第一步，并且將無法按預(yù)期工作。

I hope you enjoyed reading.

希望您喜歡閱讀。

Keep calm and wear a mask to help stop #covid19

保持鎮(zhèn)靜并戴口罩以幫助阻止＃covid19

翻譯自: https://medium.com/analytics-vidhya/building-a-covid19-mask-detector-with-opencv-keras-and-tensorflow-fc5e311071f9

總結(jié)

以上是生活随笔為你收集整理的使用OpenCV，Keras和Tensorflow构建Covid19掩模检测器的全部內(nèi)容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網(wǎng)站內(nèi)容還不錯，歡迎將生活随笔推薦給好友。