好的數(shù)據(jù)集質(zhì)量，決定后續(xù)模型的上限 (Better data leads to better models)，那么怎么快速評估數(shù)據(jù)集的質(zhì)量了？

本文分享的Facets，是一款由Google開源、快速評估數(shù)據(jù)集質(zhì)量的神器;

Facets包含2個(gè)組件：

• facets overview：outlier檢測、數(shù)據(jù)集間各特征分布比較

• facets dive：交互式探索某一特定數(shù)據(jù)細(xì)節(jié)。

---

安裝

pip?install?facets-overview

---

facets overview

以一個(gè)案例簡單介紹使用方法，

#?1、生成數(shù)據(jù)源 import?pandas?as?pdfeatures?=?["Age",?"Workclass",?"fnlwgt",?"Education",?"Education-Num","Marital?Status",?"Occupation",?"Relationship",?"Race",?"Sex","Capital?Gain",?"Capital?Loss",?"Hours?per?week",?"Country",?"Target" ] train_data?=?pd.read_csv("https://archive.ics.uci.edu/ml/machine-learning-databases/adult/adult.data",names=features,sep=r'\s*,\s*',engine='python',na_values="?") test_data?=?pd.read_csv("https://archive.ics.uci.edu/ml/machine-learning-databases/adult/adult.test",names=features,sep=r'\s*,\s*',skiprows=[0],engine='python',na_values="?")#?2、GenericFeatureStatisticsGenerator()和ProtoFromDataFrames()函數(shù)存儲(chǔ)數(shù)據(jù)集的所有統(tǒng)計(jì)信息 from?facets_overview.generic_feature_statistics_generator?import?GenericFeatureStatisticsGenerator import?base64gfsg?=?GenericFeatureStatisticsGenerator() proto?=?gfsg.ProtoFromDataFrames([{'name':?'train','table':?train_data },?{'name':?'test','table':?test_data }]) protostr?=?base64.b64encode(proto.SerializeToString()).decode("utf-8")#?3、生成HTML并可視化結(jié)果 from?IPython.core.display?import?display,?HTMLHTML_TEMPLATE?=?"""<script?src="https://cdnjs.cloudflare.com/ajax/libs/webcomponentsjs/1.3.3/webcomponents-lite.js"></script><link?rel="import"?href="https://raw.githubusercontent.com/PAIR-code/facets/1.0.0/facets-dist/facets-jupyter.html"?><facets-overview?id="elem"></facets-overview><script>document.querySelector("#elem").protoInput?=?"{protostr}";</script>""" html?=?HTML_TEMPLATE.format(protostr=protostr) display(HTML(html))

以上結(jié)果可非常方便的展示train//test數(shù)據(jù)集的偏斜情況、缺失值情況等等。

---

facets dive

同樣以一個(gè)案例簡單介紹使用方法，

import?base64 import?urllib.request import?os import?pandas?as?pd#?數(shù)據(jù)準(zhǔn)備 img_url?=?"https://storage.googleapis.com/what-if-tool-resources/misc-resources/fmnist_sprite_atlas.png" img_name?=?os.path.basename(img_url) urllib.request.urlretrieve(img_url,?img_name)df_fmnist?=?pd.read_csv("https://storage.googleapis.com/what-if-tool-resources/misc-resources/fmnist.csv" ) with?open("fmnist_sprite_atlas.png",?"rb")?as?image_file:encoded_string?=?base64.b64encode(image_file.read())#?生成HTML并可視化展示 from?IPython.core.display?import?display,?HTMLjsonstr?=?df_fmnist.to_json(orient='records') HTML_TEMPLATE?=?"""<script?src="https://cdnjs.cloudflare.com/ajax/libs/webcomponentsjs/1.3.3/webcomponents-lite.js"></script><link?rel="import"?href="https://raw.githubusercontent.com/PAIR-code/facets/1.0.0/facets-dist/facets-jupyter.html">??????<facets-dive?id="elem"?height="1000"?sprite-image-width="28"?sprite-image-height="28"?atlas-url="data:image/png;base64,{encoded_string}"></facets-dive>?#調(diào)用facets-dive?<script>var?data?=?{jsonstr};document.querySelector("#elem").data?=?data;</script>""" html?=?HTML_TEMPLATE.format(jsonstr=jsonstr,encoded_string=encoded_string.decode("utf-8")) display(HTML(html))

參考&進(jìn)一步學(xué)習(xí)👉👉https://github.com/PAIR-code/facets

往期精彩回顧適合初學(xué)者入門人工智能的路線及資料下載機(jī)器學(xué)習(xí)及深度學(xué)習(xí)筆記等資料打印機(jī)器學(xué)習(xí)在線手冊深度學(xué)習(xí)筆記專輯《統(tǒng)計(jì)學(xué)習(xí)方法》的代碼復(fù)現(xiàn)專輯 AI基礎(chǔ)下載黃海廣老師《機(jī)器學(xué)習(xí)課程》視頻課黃海廣老師《機(jī)器學(xué)習(xí)課程》711頁完整版課件

本站qq群554839127，加入微信群請掃碼：

總結(jié)

以上是生活随笔為你收集整理的【机器学习】Facets：评估机器学习数据集质量利器 (来自Google、可交互、可可视化)...的全部內(nèi)容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網(wǎng)站內(nèi)容還不錯(cuò)，歡迎將生活随笔推薦給好友。