神碼ai人工智能寫作機(jī)器人

A brief overview of the state-of-the-art in training ML models on devices. For a more comprehensive survey, read our full paper on this topic.

關(guān)于在設(shè)備上訓(xùn)練ML模型的最新技術(shù)的簡要概述。 要進(jìn)行更全面的調(diào)查，請閱讀 有關(guān)此主題的完整論文 。

We are surrounded by smart devices: from mobile phones and watches to glasses, jewelry, and even clothes. But while these devices are small and powerful, they are merely the tip of a computing iceberg that starts at your fingertips and ends in giant data and compute centers across the world. Data is transmitted from devices to the cloud where it is used to train models that are then transmitted back to be deployed back on the device. Unless used for learning simple concepts like wake words or recognizing your face to unlock your phone, machine learning is computationally expensive and data has no choice but to travel these thousands of miles before it can be turned into useful information.

智能設(shè)備包圍著我們 ：從手機(jī)和手表到眼鏡，珠寶，甚至衣服。 但是，盡管這些設(shè)備體積小巧，功能強(qiáng)大，但它們僅是計算冰山的一角，它觸手可及，并遍布全球的巨型數(shù)據(jù)和計算中心。 數(shù)據(jù)從設(shè)備傳輸?shù)皆?#xff0c;在云中用于訓(xùn)練模型，然后將模型傳輸回去再部署回設(shè)備上。 除非用于學(xué)習(xí)諸如喚醒單詞之類的簡單概念或識別您的面部以解鎖手機(jī)，否則機(jī)器學(xué)習(xí)在計算上是昂貴的，數(shù)據(jù)別無選擇，只能經(jīng)過數(shù)千英里才能轉(zhuǎn)化為有用的信息。

This journey from device to data center and back to device has its drawbacks. The privacy and security of user data is probably the most obvious as this data needs to be transmitted to the cloud and stored there, most often, indefinitely. Transmission of user data is open to interference and capture, and stored data leaves open the possibility of unauthorized access. But there are other significant drawbacks. Cloud-based AI and ML models have higher latencies, cost more to implement, lack autonomy, and, depending on the frequency of model updates, are often less personalized.

從設(shè)備到數(shù)據(jù)中心再回到設(shè)備的過程有其缺點(diǎn)。 用戶數(shù)據(jù)的隱私性和安全性可能是最明顯的，因?yàn)樵摂?shù)據(jù)需要無限期地傳輸?shù)皆撇⒋鎯υ谠浦小?用戶數(shù)據(jù)的傳輸容易受到干擾和捕獲，存儲的數(shù)據(jù)使未經(jīng)授權(quán)的訪問成為可能。 但是還有其他重大缺陷。 基于云的AI和ML模型具有更高的延遲，更高的實(shí)現(xiàn)成本，缺乏自治性，并且根據(jù)模型更新的頻率，通常不那么個性化。

As devices become more powerful, it becomes possible to address the drawbacks of the cloud model by moving some or all of the model development onto the device itself. This transfer of model development on to the device is usually referred to as Edge Learning or On-device Learning. The biggest roadblock to doing Edge Learning is model training which is the most computationally expensive part of the model development process especially in the age of deep learning. Speeding up training is possible either by adding more resources to the device or using these resources more effectively or some combination of the two.

隨著設(shè)備功能越來越強(qiáng)大，可以通過將部分或全部模型開發(fā)移至設(shè)備本身來解決云模型的缺點(diǎn)。 將模型開發(fā)轉(zhuǎn)移到設(shè)備上的過程通常稱為邊緣學(xué)習(xí)或設(shè)備上學(xué)習(xí)。 進(jìn)行邊緣學(xué)習(xí)的最大障礙是模型訓(xùn)練，這是模型開發(fā)過程中計算上最昂貴的部分，尤其是在深度學(xué)習(xí)時代。 通過向設(shè)備添加更多資源或更有效地使用這些資源或兩者的某種組合，可以加快培訓(xùn)速度。

This transfer of model development on to the device is usually referred to as Edge Learning or On-device Learning.

將模型開發(fā)轉(zhuǎn)移到設(shè)備上的過程通常稱為邊緣學(xué)習(xí)或設(shè)備上學(xué)習(xí)。

Fig 1: A hierarchical view of the various approaches to edge/on-device learning. The boxes in grey are the topics covered in this article and corresponding paper. Image by Author圖1：邊緣/設(shè)備上學(xué)習(xí)的各種方法的層次結(jié)構(gòu)圖。 灰色框是本文和相應(yīng)論文中涉及的主題。 圖片作者

Fig 1 gives a hierarchical view of the ways to improve model training on devices. On the left are the hardware approaches that work with the actual chipsets. Fundamental research in this area aims at improving existing chip design (by developing chips with more compute and memory, and lower power consumption and footprint) or developing new designs with novel architectures that speed up model training. While hardware research is a fruitful avenue for improving on-device learning, it is an expensive process that requires large capital expenditure to build laboratories and fabrication facilities, and usually involves long timescales for development.

圖1給出了改進(jìn)設(shè)備模型訓(xùn)練的方法的分層視圖。 左側(cè)是與實(shí)際芯片組配合使用的硬件方法。 該領(lǐng)域的基礎(chǔ)研究旨在改進(jìn)現(xiàn)有芯片設(shè)計(通過開發(fā)具有更多計算和內(nèi)存以及更低功耗和占用空間的芯片)或開發(fā)具有新穎架構(gòu)的新設(shè)計來加快模型訓(xùn)練的速度。 雖然硬件研究是改善設(shè)備上學(xué)習(xí)的有效途徑，但它是一個昂貴的過程，需要大量資本支出來建立實(shí)驗(yàn)室和制造設(shè)施，并且通常涉及較長的開發(fā)時間。

Software approaches encompass a large part of current work in this field. Every machine learning algorithm depends on a small set of computing libraries for efficient execution of a few key operations (such as Multiply-Add in the case of neural networks). The libraries that support these operations are the interface between the hardware and the algorithms and allow for algorithm development that is not based on any specific hardware architecture. However, these libraries are heavily tuned to the unique aspects of the hardware on which the operations are executed. This dependency limits the amount of improvement that can be gained by new libraries. The algorithms part of software approaches gets the most attention when it comes to improving ML on the edge as it involves the development and improvement of the machine learning algorithms themselves.

軟件方法涵蓋了該領(lǐng)域當(dāng)前的大部分工作。 每種機(jī)器學(xué)習(xí)算法都依賴于一小組計算庫來有效執(zhí)行一些關(guān)鍵操作(例如在神經(jīng)網(wǎng)絡(luò)的情況下為乘加)。 支持這些操作的庫是硬件和算法之間的接口，并允許不基于任何特定硬件體系結(jié)構(gòu)的算法開發(fā)。 但是，這些庫在很大程度上針對執(zhí)行操作的硬件的獨(dú)特方面進(jìn)行了調(diào)整。 這種依賴性限制了新庫可以實(shí)現(xiàn)的改進(jìn)量。 當(dāng)涉及到邊緣機(jī)器學(xué)習(xí)的改進(jìn)時，軟件方法的算法部分將引起最多的關(guān)注，因?yàn)樗婕皺C(jī)器學(xué)習(xí)算法本身的開發(fā)和改進(jìn)。

Finally, theoretical approaches help direct new research on ML algorithms. These approaches improve our understanding of existing techniques and their generalizability to new problems, environments, and hardware.

最后，理論方法有助于指導(dǎo)有關(guān)ML算法的新研究。 這些方法提高了我們對現(xiàn)有技術(shù)的理解以及它們對新問題，環(huán)境和硬件的一般性。

This article focuses on developments in algorithms and theoretical approaches. While hardware and computing libraries are equally important, given the long lead times for novel hardware and the interdependency between hardware and libraries, the state-of-the-art changes faster in the algorithms and theoretical spaces.

本文重點(diǎn)介紹算法和理論方法的發(fā)展。 盡管硬件和計算庫同等重要，但鑒于新型硬件的交貨時間較長以及硬件和庫之間的相互依賴關(guān)系，最新技術(shù)在算法和理論空間方面的變化更快。

演算法 (Algorithms)

Most of the work in on-device ML has been on deploying models. Deployment focuses on improving model size and inference speed using techniques like model quantization and model compression. For training models on devices, there needs to be advances in areas such as model optimization and Hyperparameter Optimization (HPO). But, advances in these fields improve accuracy and the rate of convergence, often at the expense of compute and memory usage. To improve model training on devices, it is important to have training techniques that are aware of the resource constraints under which these techniques will be run.

設(shè)備上ML的大部分工作都是在部署模型上進(jìn)行的。 部署著重于使用模型量化和模型壓縮等技術(shù)來提高模型大小和推理速度。 對于在設(shè)備上訓(xùn)練模型，需要在模型優(yōu)化和超參數(shù)優(yōu)化(HPO)等領(lǐng)域取得進(jìn)步。 但是，這些領(lǐng)域的進(jìn)步通常會以計算和內(nèi)存使用為代價，提高準(zhǔn)確性和收斂速度。 為了改善設(shè)備上的模型訓(xùn)練，重要的是要有訓(xùn)練技術(shù)，這些訓(xùn)練技術(shù)應(yīng)知道將在這些資源下運(yùn)行這些資源。

To improve model training on devices, it is important to have training techniques that are aware of the resource constraints under which these techniques will be run.

為了改善設(shè)備上的模型訓(xùn)練，重要的是要有訓(xùn)練技術(shù)，這些訓(xùn)練技術(shù)應(yīng)知道將在這些資源下運(yùn)行這些資源。

The mainstream approach to doing such resource-aware model training is to design ML algorithms that satisfy a surrogate software-centric resource constraint instead of a standard loss function. Such surrogate measures are designed to approximate the hardware constraints through asymptotic analysis, resource profiling, or resource modeling. For a given software-centric resource constraint, state-of-art algorithm designs adopt one of the following approaches:

進(jìn)行這種資源感知模型訓(xùn)練的主流方法是設(shè)計滿足替代軟件中心資源約束而不是標(biāo)準(zhǔn)損失函數(shù)的ML算法。 此類替代措施旨在通過漸近分析，資源配置文件或資源建模來近似估計硬件約束。 對于給定的以軟件為中心的資源約束，最新的算法設(shè)計采用以下方法之一：

Lightweight ML Algorithms — Existing algorithms, such as linear/logistic regression or SVMs, have low resource footprints and need no additional modifications for resource constrained model building. This low footprint makes these techniques an easy and obvious starting point for building resource-constrained learning models. However, in cases where the available device’s resources are smaller than the resource footprint of the selected lightweight algorithm, this approach will fail. Additionally, in many cases, lightweight ML algorithms result in models with low complexity that may fail to fully capture the underlying process resulting in underfitting and poor performance.

輕量級ML算法 -現(xiàn)有的算法(例如線性/邏輯回歸或SVM)具有較低的資源占用量，并且無需進(jìn)行其他修改即可構(gòu)建資源受限的模型。 這種低占用空間使這些技術(shù)成為構(gòu)建資源受限的學(xué)習(xí)模型的簡單而明顯的起點(diǎn)。 但是，如果可用設(shè)備的資源小于所選輕量算法的資源占用量，則此方法將失敗。 此外，在許多情況下，輕量級ML算法導(dǎo)致模型的復(fù)雜度較低，可能無法完全捕獲基礎(chǔ)過程，從而導(dǎo)致擬合不足和性能不佳。

Reducing Model complexity — A better approach to control the size (memory footprint) and computation complexity of the learning algorithm is by constraining the model architecture (for e.g. by selecting a smaller hypothesis class). This approach has the added advantage that these models can be trained using traditional optimization routines. Apart from model building, this is one of the dominant approaches for deploying resource efficient models for model inference. Most importantly, this approach extends to even Deep Neural Networks (DNNs) where, as evidenced by Fig 2, there has been a slow but steady progression towards smaller, faster, leaner architectures. This progression has been helped by the increased use of Neural Architecture Search (NAS) techniques that show a preference for smaller, more efficient networks. Compared to the lightweight ML algorithms approach, model complexity reduction techniques can accommodate a broader class of ML algorithms and can more effectively capture the underlying process.

降低模型復(fù)雜度 —控制學(xué)習(xí)算法的大小(內(nèi)存占用)和計算復(fù)雜度的更好方法是約束模型架構(gòu)(例如，通過選擇較小的假設(shè)類別)。 這種方法的另一個優(yōu)點(diǎn)是可以使用傳統(tǒng)的優(yōu)化例程來訓(xùn)練這些模型。 除模型構(gòu)建外，這是為模型推理部署資源高效模型的主要方法之一。 最重要的是，這種方法甚至擴(kuò)展到了深度神經(jīng)網(wǎng)絡(luò)(DNN)，如圖2所示，在向更小，更快，更精簡的架構(gòu)發(fā)展的過程中，進(jìn)展緩慢但穩(wěn)定。 越來越多地使用神經(jīng)體系結(jié)構(gòu)搜索(NAS)技術(shù)幫助實(shí)現(xiàn)了這一進(jìn)步，這些技術(shù)顯示出對更小，更高效的網(wǎng)絡(luò)的偏愛。 與輕量級ML算法相比，模型復(fù)雜度降低技術(shù)可以容納更多種類的ML算法，并且可以更有效地捕獲底層過程。

Fig 2. Ball chart of the chronological evolution of model complexity. Top-1 accuracy is measured on the ImageNet dataset. The model complexity is represented by FLOPS and reflected by the ball size. The accuracy and FLOPS are taken from original publications of the models. The time of the model is when the associated publication is first made available online. Image by Junyao Guo.圖2.模型復(fù)雜度按時間順序演變的球狀圖。 Top-1準(zhǔn)確性是在ImageNet數(shù)據(jù)集上測量的。 模型的復(fù)雜性由FLOPS表示，并由球的大小反映出來。 精度和FLOPS取自模型的原始出版物。 模型的時間是相關(guān)的出版物首次在線提供的時間。 郭俊堯

Modifying optimization routines — The most significant of the algorithmic advances is the design of optimization routines specifically for resource-efficient model building where resource constraints are incorporated during the model building (training) phase. Instead of limiting the model architectures beforehand, these approaches can adapt optimization routines to fit the resource constraints for any given model architecture (hypothesis class).

修改優(yōu)化例程 -最先進(jìn)的算法是專門針對資源效率較高的模型構(gòu)建的優(yōu)化例程的設(shè)計，在模型構(gòu)建(訓(xùn)練)階段將資源約束納入其中。 這些方法可以預(yù)先使用優(yōu)化例程以適應(yīng)任何給定模型體系結(jié)構(gòu)(假設(shè)類)的資源約束，而不是預(yù)先限制模型體系結(jié)構(gòu)。

Resource-constrained model-centric optimization routines focus on improving the performance of models that will be quantized after training either through stochastic rounding, weight initialization, or by introducing quantization error into gradient updates. Also prevalent are layer-wise training and techniques that trade computation for memory, both of which try to reduce the computational requirements associated with training DNNs. In certain cases, this approach can also dynamically modify the architecture to fit the resource constraints. Although this approach provides a wider choice of the class of models, the design process is still tied to a specific problem type (classification, regression, etc.) and depends on the selected method/loss function (linear regression, ridge regression for regression problems).

資源受限的以模型為中心的優(yōu)化例程著重于提高模型的性能，這些模型將在訓(xùn)練后通過隨機(jī)舍入，權(quán)重初始化或?qū)⒘炕`差引入梯度更新來進(jìn)行量化。 分層訓(xùn)練和以內(nèi)存換取計算的技術(shù)也很普遍，它們都試圖減少與訓(xùn)練DNN相關(guān)的計算要求。 在某些情況下，此方法還可以動態(tài)修改體系結(jié)構(gòu)以適應(yīng)資源限制。 盡管此方法提供了更多的模型類別選擇，但設(shè)計過程仍與特定的問題類型(分類，回歸等)相關(guān)，并且取決于所選的方法/損失函數(shù)(線性回歸，針對回歸問題的嶺回歸) )。

Resource-constrained generic optimization routines such as Buckwild! And SWALP focuses on reducing the resource-footprint for model training by using low-precision arithmetic for gradient computations. An alternative line of work involves implementing fixed point Quadratic Programs (QP) such as QSGD or QSVRG for solving linear Model Predictive Control (MPC). Most of these algorithms involve modifying fast gradient methods for convex optimization to obtain a suboptimal solution in a finite number of iterations under resource-constrained settings .

資源受限的通用優(yōu)化例程，例如Buckwild !！ SWALP致力于通過使用低精度算術(shù)進(jìn)行梯度計算來減少模型訓(xùn)練的資源占用。 另一種工作方式是實(shí)施定點(diǎn)二次程序(QP)，例如QSGD或QSVRG，以解決線性模型預(yù)測控制(MPC)。 這些算法大多數(shù)都涉及修改快速梯度方法以進(jìn)行凸優(yōu)化，以在資源受限的設(shè)置下以有限次數(shù)的迭代獲得次優(yōu)解。

Data Compression — Rather than constraining the model size/complexity, data compression approaches target building models on compressed data. The goal is to limit the memory usage via reduced data storage and computation through fixed per-sample computation cost. A more generic approach includes adopting advanced learning settings that accommodates algorithms with smaller sample complexity. However, this is a broader research topic and is not just limited to on-device learning.

數(shù)據(jù)壓縮 - 數(shù)據(jù)壓縮不是限制模型的大小/復(fù)雜性，而是針對壓縮數(shù)據(jù)構(gòu)建目標(biāo)模型。 目的是通過減少數(shù)據(jù)存儲和通過固定的每樣本計算成本進(jìn)行計算來限制內(nèi)存使用。 更為通用的方法包括采用高級學(xué)習(xí)設(shè)置，以適應(yīng)樣本復(fù)雜度較小的算法。 但是，這是一個更廣泛的研究主題，而不僅限于設(shè)備上的學(xué)習(xí)。

New protocols for data observation — Finally, completely novel approaches are possible that completely change the traditional data observation protocol (like the availability of i.i.d data in batch or online settings). These approaches are guided by an underlying resource-constrained learning theory which captures the interplay between resource constraints and the goodness of the model in terms of the generalization capacity. Compared to the above approaches, this framework provides a generic mechanism to design resource-constrained algorithms for a wider range of learning problems applicable to any method/loss function targeting that problem type.

數(shù)據(jù)觀察的新協(xié)議 —最后，完全新穎的方法可能會徹底改變傳統(tǒng)的數(shù)據(jù)觀察協(xié)議(例如，批量或在線設(shè)置中的iid數(shù)據(jù)的可用性)。 這些方法以一種潛在的資源受限學(xué)習(xí)理論為指導(dǎo)，該理論從泛化能力的角度捕獲了資源約束與模型優(yōu)度之間的相互作用。 與上述方法相比，此框架提供了一種通用機(jī)制來設(shè)計資源受限算法，用于更廣泛的學(xué)習(xí)問題，適用于針對該問題類型的任何方法/損失函數(shù)。

ChallengesThe major challenge in algorithms research is proper software-centric characterization of the hardware constraints and the appropriate use of this characterization for better metric designs. If hardware dependencies are not properly abstracted away, the same model and algorithm can have very different performance profiles on different hardware. While novel loss functions can take such dependencies into account, it is still a relatively new field of study. The assumption in many cases is that the resource budget available for training does not change but that is usually never the case. Our everyday devices are often multi-tasking — checking emails, social media, messaging people, playing videos… the list goes on. Each of these apps and services are constantly vying for resources at any given moment in time. Taking this changing resource landscape into account is an important challenge for effective model training on the edge.

挑戰(zhàn)算法研究中的主要挑戰(zhàn)是對硬件約束進(jìn)行正確的以軟件為中心的表征，以及為更好的度量設(shè)計而適當(dāng)使用此表征。 如果沒有正確抽象出硬件依賴關(guān)系，則相同的模型和算法在不同的硬件上可能具有非常不同的性能。 盡管新穎的損失函數(shù)可以考慮這種依賴性，但它仍然是一個相對較新的研究領(lǐng)域。 許多情況下的假設(shè)是，可用于培訓(xùn)的資源預(yù)算不會改變，但通常永遠(yuǎn)不會改變。 我們的日常設(shè)備通常是多任務(wù)處理的-檢查電子郵件，社交媒體，消息傳遞者，播放視頻...等等。 這些應(yīng)用程序和服務(wù)中的每一個都在任何給定的時間不斷爭奪資源。 考慮到這種不斷變化的資源格局，這是在邊緣進(jìn)行有效模型訓(xùn)練的重要挑戰(zhàn)。

Finally, improved methods for model profiling are needed to more accurately calculate an algorithm’s resource consumption. Current approaches to such measurements are abstract and focus on applying software engineering principles such as asymptotic analysis or low-level measures like FLOPS or MACs (Multiply-Add Computations). None of these approaches give a holistic idea of resource requirements and in many cases represent an insignificant portion of the total resources required by the system during learning.

最后，需要用于模型分析的改進(jìn)方法來更準(zhǔn)確地計算算法的資源消耗。 當(dāng)前進(jìn)行此類測量的方法是抽象的，并且側(cè)重于應(yīng)用軟件工程原理，例如漸近分析或諸如FLOPS或MAC(乘加計算)的低級測量。 這些方法都沒有一個全面的資源需求概念，并且在許多情況下，它們在學(xué)習(xí)過程中只占系統(tǒng)所需總資源的很小一部分。

理論 (Theory)

Every learning algorithm is based on an underlying theory that guarantees certain aspects of its performance. Research in this area focuses mainly on Learnability — the development of frameworks to analyze the statistical aspects (i.e. error guarantees) of algorithms. While traditional machine learning theories underlie most current approaches, developing newer notions of learnability that include resource constraints will help us better understand and predict how algorithms will perform under resource-constrained settings. There are two broad categories of theories into which most of the existing resource-constrained algorithms can be divided

每種學(xué)習(xí)算法均基于保證其性能某些方面的基礎(chǔ)理論。 該領(lǐng)域的研究主要集中在可學(xué)習(xí)性上 —開發(fā)用于分析算法統(tǒng)計方面(即錯誤保證)的框架。 盡管傳統(tǒng)的機(jī)器學(xué)習(xí)理論是大多數(shù)當(dāng)前方法的基礎(chǔ)，但發(fā)展包括資源約束在內(nèi)的更新的可學(xué)習(xí)性概念將有助于我們更好地理解和預(yù)測算法在資源受限的環(huán)境下的性能。 有兩大類理論可將大多數(shù)現(xiàn)有的資源受限算法分為

Traditional Learning Theories — Most existing resource-constrained algorithms are designed following traditional machine learning theory (like PAC Learning Theory, Mistake Bounds, Statistical Query). A limitation of this approach is that such theories are built mainly for analyzing the error guarantees of the algorithm used for model estimation. The effect of resource constraints on the generalization capability of the algorithm is not directly addressed through such theories. For example, algorithms developed using the approach of reducing the model complexity typically adopts a two-step approach. First, the size of the hypothesis class is constrained beforehand to those that use fewer resources. Next, an algorithm is designed guaranteeing the best-in-class model within that hypothesis class. What is missing in such frameworks is the direct interplay between the error guarantees and the resource constraints.

傳統(tǒng)學(xué)習(xí)理論 -大多數(shù)現(xiàn)有資源受限的算法都是按照傳統(tǒng)的機(jī)器學(xué)習(xí)理論(如PAC學(xué)習(xí)理論，誤區(qū)，統(tǒng)計查詢)設(shè)計的。 這種方法的局限性在于，建立這種理論主要是為了分析用于模型估計的算法的誤差保證。 通過這種理論不能直接解決資源約束對算法泛化能力的影響。 例如，使用降低模型復(fù)雜度的方法開發(fā)的算法通常采用兩步法。 首先，假設(shè)類的大小事先限制為使用較少資源的類。 接下來，設(shè)計一種算法，以確保該假設(shè)類別內(nèi)的同類最佳模型。 這種框架中缺少的是錯誤保證和資源約束之間的直接相互作用。

Resource-constrained learning theories — Newer learning theories try to overcome the drawbacks of traditional theories especially since new research has shown that it may be impossible to learn a hypothesis class under resource constrained settings. Most of the algorithms from earlier that assume new protocols for data observation fall in this category of resource-constrained theories. Typically, such approaches modify the traditional assumption of i.i.d data being presented in a batch or streaming fashion and introduces a specific protocol of data observability that limits the memory/space footprint used by the approach. These theories provide a platform to utilize existing computationally efficient algorithms under memory-constrained settings to build machine learning models with strong error guarantees. Prominent resource-constrained learning theories include Restricted Focus of Attention (RFA), newer Statistical Query (SQ) based learning paradigms, and graph-based approaches that model the hypothesis class as a hypothesis graph. Branching programs translate the learning algorithm under resource constraints (memory) in the form of a matrix (as opposed to a graph) where there is a connection between the stability of the matrix norm (in the form of an upper bound on its maximum singular value) and the learnability of the hypothesis class with limited memory. Although such theory-motivated design provides a generic framework through which algorithms can be designed for a wide range of learning problems, to date, very few algorithms based on these theories have been developed.

資源受限的學(xué)習(xí)理論 -較新的學(xué)習(xí)理論試圖克服傳統(tǒng)理論的弊端，特別是因?yàn)樾碌难芯勘砻?#xff0c;在資源受限的環(huán)境下學(xué)習(xí)假設(shè)類是不可能的。 早先的大多數(shù)假設(shè)使用新協(xié)議進(jìn)行數(shù)據(jù)觀察的算法都屬于這種資源受限的理論。 通常，此類方法修改了以批量或流方式呈現(xiàn)iid數(shù)據(jù)的傳統(tǒng)假設(shè)，并引入了數(shù)據(jù)可觀察性的特定協(xié)議，該協(xié)議限制了該方法使用的內(nèi)存/空間占用量。 這些理論提供了一個平臺，可以在內(nèi)存受限的設(shè)置下利用現(xiàn)有的高效計算算法來構(gòu)建具有強(qiáng)大錯誤保證的機(jī)器學(xué)習(xí)模型。 突出的資源受限學(xué)習(xí)理論包括限制注意力集中(RFA)，更新的基于統(tǒng)計查詢(SQ)的學(xué)習(xí)范例，以及將假設(shè)類別建模為假設(shè)圖的基于圖的方法。 分支程序在資源約束(內(nèi)存)下以矩陣(而不是圖)的形式轉(zhuǎn)換學(xué)習(xí)算法，其中矩陣范數(shù)的穩(wěn)定性(以其最大奇異值的上限形式)之間存在聯(lián)系)和假設(shè)類的可記憶性有限。 盡管這種基于理論的設(shè)計提供了一個通用的框架，通過該框架可以針對各種學(xué)習(xí)問題設(shè)計算法，但迄今為止，基于這些理論的算法很少。

ChallengesPerhaps the biggest drawback to theoretical research is that while it is flexible enough to apply across classes of algorithms and hardware systems, it is limited due to the inherent difficulty of such research and the need to implement a theory in the form of an algorithm before its utility can be realized.

挑戰(zhàn)理論研究的最大缺點(diǎn)可能是，盡管它具有足夠的靈活性以適用于各種算法和硬件系統(tǒng)，但由于這種研究的固有困難以及需要以算法的形式實(shí)現(xiàn)理論而受到限制。它的效用可以實(shí)現(xiàn)。

結(jié)論 (Conclusion)

A future full of smart devices was the stuff of science fiction when we slipped the first iPhones into our pockets. Thirteen years later, devices have become much more capable and now promise the power of AI and ML right at our fingertips. However, these new-found capabilities are a facade propped up by massive computational resources (data centers, compute clusters, 4G/5G networks etc) that bring AI and ML to life. But devices can only be truly powerful on their own when it is possible to sever the lifeline that extends between them and the cloud. And that requires the ability to train machine learning models on these devices rather than in the cloud.

當(dāng)我們將第一批iPhone放入口袋時，科幻小說充滿了未來的智能設(shè)備。 十三年后，設(shè)備變得更加強(qiáng)大，現(xiàn)在可以在指尖獲得AI和ML的強(qiáng)大功能。 但是，這些新發(fā)現(xiàn)的功能是使AI和ML栩栩如生的大量計算資源(數(shù)據(jù)中心，計算集群，4G / 5G網(wǎng)絡(luò)等)支撐的立面。 但是，只有在可以切斷在設(shè)備與云之間延伸的生命線的情況下，設(shè)備才能真正發(fā)揮真正的強(qiáng)大功能。 這就要求能夠在這些設(shè)備上而不是在云中訓(xùn)練機(jī)器學(xué)習(xí)模型。

Training ML models on a device has so far remained an academic pursuit, but with the increasing number of smart devices and improved hardware, there is interest in performing learning on the device itself. In the industry, this interest is fueled mainly by hardware manufacturers promoting AI-specific chipsets that are optimized for certain mathematical operations, and startups providing ad hoc solutions to certain niche domains mostly in computer vision and IoT. From an AI/ML perspective, most of the activity lies in two areas — the development of algorithms that can train models under resource constraints and the development of theoretical frameworks that provide guarantees about the performance of such algorithms.

迄今為止，在設(shè)備上訓(xùn)練ML模型仍是一項(xiàng)學(xué)術(shù)追求，但是隨著智能設(shè)備和硬件的改進(jìn)，人們對在設(shè)備上進(jìn)行學(xué)習(xí)感興趣。 在行業(yè)中，這種興趣主要是由硬件制造商推動的，這些制造商推廣了針對特定數(shù)學(xué)運(yùn)算進(jìn)行了優(yōu)化的AI專用芯片組，并且初創(chuàng)公司主要在計算機(jī)視覺和IoT中為某些特定領(lǐng)域提供了臨時解決方案。 從AI / ML的角度來看，大多數(shù)活動都在兩個領(lǐng)域中：可以在資源約束下訓(xùn)練模型的算法的開發(fā)以及為此類算法的性能提供保證的理論框架的開發(fā)。

At the algorithmic level, it is clear that current efforts are mainly targeted at either utilizing already lightweight machine learning algorithms or modifying existing algorithms in ways that reduce resource utilization. There are a number of challenges before we can consistently train models on the edge including the need for decoupling algorithms from the hardware, and designing effective loss functions and metrics that capture resource constraints. Also important are an expanded focus on traditional as well as advanced ML algorithms with low sample complexity and dealing with situations where the resource budget is dynamic rather than static. Finally, the availability of an easy and reliable way to profile algorithm behavior under resource constraints will speed up the entire development process.

在算法級別，很明顯，當(dāng)前的工作主要針對利用已經(jīng)很輕量級的機(jī)器學(xué)習(xí)算法或以減少資源利用的方式修改現(xiàn)有算法。 在我們不斷地在邊緣訓(xùn)練模型之前，存在許多挑戰(zhàn)，包括需要將算法與硬件解耦，以及設(shè)計有效的損失函數(shù)和指標(biāo)以捕獲資源約束的需求。 同樣重要的是，應(yīng)進(jìn)一步關(guān)注具有低樣本復(fù)雜度的傳統(tǒng)以及高級ML算法，并應(yīng)對資源預(yù)算是動態(tài)而非靜態(tài)的情況。 最后，在資源限制下提供一種簡單可靠的方法來描述算法行為的方法將加速整個開發(fā)過程。

Learning theory for resource-constrained algorithms is focused on the un-learnability of an algorithm under resource constraints. The natural step forward is to identify techniques that can instead provide guarantees on the learnability of an algorithm and the associated estimation error. Existing theoretical techniques also mainly focus on the space(memory) complexity of these algorithms and not their compute requirements. Even in cases where an ideal hypothesis class can be identified that satisfies resource constraints, further work is needed to select the optimal model from within that class.

資源受限算法的學(xué)習(xí)理論集中于資源約束下算法的不可學(xué)習(xí)性。 向前邁出的自然步伐是確定可以替代地為算法的可學(xué)習(xí)性和相關(guān)的估計誤差提供保證的技術(shù)。 現(xiàn)有的理論技術(shù)也主要集中在這些算法的空間(存儲器)復(fù)雜度上，而不是它們的計算要求上。 即使在可以確定滿足資源約束的理想假設(shè)類別的情況下，也需要進(jìn)一步的工作以從該類別中選擇最佳模型。

Despite these difficulties, the future for machine learning on the edge is exciting. Model sizes, even for deep neural networks, have been trending down. Major platforms such as Apple’s Core/CreateML support the retraining of models on the device. While the complexity and training regimen of models continue to grow, it is within the realm of possibility that we will continue to see a push to offload computation from the cloud to the device for reasons of privacy and security, cost, latency, autonomy, and better personalization.

盡管存在這些困難，但是邊緣機(jī)器學(xué)習(xí)的未來還是令人興奮的。 甚至對于深度神經(jīng)網(wǎng)絡(luò)，模型的大小一直在下降。 蘋果的Core / CreateML等主要平臺支持對設(shè)備上的模型進(jìn)行再培訓(xùn)。 盡管模型的復(fù)雜性和訓(xùn)練方案不斷增長，但出于隱私和安全性，成本，延遲，自治性和安全性的考慮，我們將繼續(xù)看到將計算從云上卸載到設(shè)備的可能性。更好的個性化。

This article was written with contributions from Sauptik Dhar, Junyao Guo, Samarth Tripathi, Jason Liu, Vera Serdiukova, and Mohak Shah.

本文是由Sauptik Dhar，Guunyao Guo，Samarth Tripathi，Jason Liu，Vera Serdiukova和Mohak Shah撰寫的。

If you are interested in a more comprehensive survey of edge learning, read our full paper on this topic.

如果您對邊緣學(xué)習(xí)的更全面的研究感興趣，請閱讀我們 關(guān)于該主題的全文 。

翻譯自: https://towardsdatascience.com/from-cloud-to-device-the-future-of-ai-and-machine-learning-on-the-edge-78009d0aee9

神碼ai人工智能寫作機(jī)器人

總結(jié)

以上是生活随笔為你收集整理的神码ai人工智能写作机器人_从云到设备到边缘的人工智能和机器学习的未来的全部內(nèi)容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網(wǎng)站內(nèi)容還不錯，歡迎將生活随笔推薦給好友。