在機器學習中有很多調優的方式機器學習——超參數調優，深度學習中也存在同樣的方式，接下來，介紹一下深度學習框架里邊的自動調參模塊。

1. 基于Tensorflow的Keras tuner

官方教程如下：Introduction to the Keras Tuner ?|? TensorFlow Core (google.cn)

官網API極其更多細節：HyperParameters - Keras Tuner (keras-team.github.io)

超參數分為兩種類型：

• 模型超參數（例如隱藏層的權重和數量）
• 算法超參數（例如SGD的學習率，KNN的K值等）

（1）定義模型

當建立模型進行超參數優化時，我們需要根據模型建立超參數搜索空間，建立hypermodel通常是兩個途徑：

• 使用模型建立函數（model builder function）
• 使用Keras Tuner API中HyperModel的子類HyperXception?and?HyperResNet

Keras Tuner API使用hp進行參數遍歷，常用方法有

• hp.Int?

hp_units = hp.Int('units', min_value=32, max_value=512, step=32)

? ? ? ? ? ? name: Str. 參數的名字，必須唯一
? ? ? ? ? ? min_value: Int. 范圍的最小值
? ? ? ? ? ? max_value: Int.?范圍的最大值
? ? ? ? ? ? step: 步長

• hp.Choice?

hp_learning_rate = hp.Choice('learning_rate', values=[1e-2, 1e-3, 1e-4])

? ? ? ? ? ? name: Str. 參數的名字，必須唯一
? ? ? ? ? ? values: 可能值的list. 值的類型可以是int, float,str, or bool. 所有的值必須是一個類型
? ? ? ? ? ? ordered: Whether the values passed should be considered to
? ? ? ? ? ? ? ? have an ordering. This defaults to `True` for float/int
? ? ? ? ? ? ? ? values. Must be `False` for any other values.

• hp.Float

hp_units = hp.Float('units', min_value=32.0, max_value=512.0, step=32.0)

? ? ? ? ? ? name: Str. 參數的名字，必須唯一
? ? ? ? ? ? min_value: Float. 范圍的最小值
? ? ? ? ? ? max_value: Float. 范圍的最大值
? ? ? ? ? ? step: Optional. Float, e.g. 0.1.
? ? ? ? ? ? ? ? smallest meaningful distance between two values.
? ? ? ? ? ? ? ? Whether step should be specified is Oracle dependent,
? ? ? ? ? ? ? ? since some Oracles can infer an optimal step automatically.

• hp.Boolean

? ? ? ? ? ? name: Str. 參數的名字，必須唯一
? ? ? ? ? ? default: Default value to return for the parameter.
? ? ? ? ? ? ? ? If unspecified, the default value will be False.

• hp.Fixed

? ? ?? ? ? ?name: Str. 參數的名字，必須唯一
? ? ? ? ? ? value: Value to use (can be any JSON-serializable
? ? ? ? ? ? ? ? Python type).

建立builder function，主要對units和learning_rate兩個參數進行了調優，代碼如下：

def model_builder(hp):model = keras.Sequential()model.add(keras.layers.Flatten(input_shape=(28, 28)))# Tune the number of units in the first Dense layer# Choose an optimal value between 32-512hp_units = hp.Int('units', min_value=32, max_value=512, step=32)model.add(keras.layers.Dense(units=hp_units, activation='relu'))model.add(keras.layers.Dense(10))# Tune the learning rate for the optimizer# Choose an optimal value from 0.01, 0.001, or 0.0001hp_learning_rate = hp.Choice('learning_rate', values=[1e-2, 1e-3, 1e-4])model.compile(optimizer=keras.optimizers.Adam(learning_rate=hp_learning_rate),loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),metrics=['accuracy'])return model

（2）實例化調優器并進行調優

The Keras Tuner有四個調優器，分別是RandomSearch,?Hyperband,?BayesianOptimization, and?Sklearn，這里主要是用了Hyperband，需要指定objective和max_epochs等參數，并在directory='my_dir'/project_name='intro_to_kt'的文件中記錄訓練細節。代碼如下：

tuner = kt.Hyperband(model_builder,objective='val_accuracy',max_epochs=10,factor=3,directory='my_dir',project_name='intro_to_kt')#早停 stop_early = tf.keras.callbacks.EarlyStopping(monitor='val_loss', patience=5)tuner.search(img_train, label_train, epochs=50, validation_split=0.2, callbacks=[stop_early])# Get the optimal hyperparameters best_hps=tuner.get_best_hyperparameters(num_trials=1)[0]print(f""" The hyperparameter search is complete. The optimal number of units in the first densely-connected layer is {best_hps.get('units')} and the optimal learning rate for the optimizer is {best_hps.get('learning_rate')}. """)

（3）模型調優之后，可以通過最優參數進行重新training。

#利用搜索得到的超參數，找出最優epoch數來訓練模型。 # Build the model with the optimal hyperparameters and train it on the data for 50 epochs model = tuner.hypermodel.build(best_hps) history = model.fit(img_train, label_train, epochs=50, validation_split=0.2)val_acc_per_epoch = history.history['val_accuracy'] best_epoch = val_acc_per_epoch.index(max(val_acc_per_epoch)) + 1 print('Best epoch: %d' % (best_epoch,))#重新實例化超模型，從上面用最優的紀元數訓練它。 hypermodel = tuner.hypermodel.build(best_hps)# Retrain the model hypermodel.fit(img_train, label_train, epochs=best_epoch, validation_split=0.2)

拓展：其他優化器的用法

import kerastuner as ktfrom sklearn import ensemblefrom sklearn import datasetsfrom sklearn import linear_modelfrom sklearn import metricsfrom sklearn import model_selectiondef build_model(hp):model_type = hp.Choice('model_type', ['random_forest', 'ridge'])if model_type == 'random_forest':model = ensemble.RandomForestClassifier(n_estimators=hp.Int('n_estimators', 10, 50, step=10),max_depth=hp.Int('max_depth', 3, 10))else:model = linear_model.RidgeClassifier(alpha=hp.Float('alpha', 1e-3, 1, sampling='log'))return modeltuner = kt.tuners.Sklearn(oracle=kt.oracles.BayesianOptimization(objective=kt.Objective('score', 'max'),max_trials=10),hypermodel=build_model,scoring=metrics.make_scorer(metrics.accuracy_score),cv=model_selection.StratifiedKFold(5),directory='.',project_name='my_project')X, y = datasets.load_iris(return_X_y=True)X_train, X_test, y_train, y_test = model_selection.train_test_split(X, y, test_size=0.2)tuner.search(X_train, y_train)best_model = tuner.get_best_models(num_models=1)[0]

其他兩個：

tuner = kt.tuners.BayesianOptimization(hypermodel=model_builder,objective='val_accuracy',max_trials=4,directory='.',project_name='my_project') tuner = kt.tuners.RandomSearch(hypermodel=model_builder,objective='val_accuracy',max_trials=4,directory='.',project_name='my_project')

2.RAY tune

RAY現在僅支持Mac和Linux，Windows的wheels還沒有。

（1）安裝：

pip install 'ray[tune]'

（2）優勢：

• 支持多平臺包括PyTorch, XGBoost, MXNet和Keras
• 自動管理檢查點和記錄到TensorBoard

（3）關鍵點

Ray tune的使用核心點主要如下圖構成：

• Trainable：包裝目標函數

#損失函數 def objective(x, a, b):return a * (x ** 0.5) + b #wrapper def trainable(config):# config (dict): A dict of hyperparameters.for x in range(20):score = objective(x, config["a"], config["b"])tune.report(score=score) # This sends the score to Tune.

• tune.run:執行參數優化

# Pass in a Trainable class or function to tune.run. tune.run(trainable)

• search space：建立搜索空間

config = {"uniform": tune.uniform(-5, -1), # Uniform float between -5 and -1"quniform": tune.quniform(3.2, 5.4, 0.2), # Round to increments of 0.2"loguniform": tune.loguniform(1e-4, 1e-2), # Uniform float in log space"qloguniform": tune.qloguniform(1e-4, 1e-1, 5e-4), # Round to increments of 0.0005"randn": tune.randn(10, 2), # Normal distribution with mean 10 and sd 2"qrandn": tune.qrandn(10, 2, 0.2), # Round to increments of 0.2"randint": tune.randint(-9, 15), # Random integer between -9 and 15"qrandint": tune.qrandint(-21, 12, 3), # Round to increments of 3 (includes 12)"choice": tune.choice(["a", "b", "c"]), # Choose one of these options uniformly"func": tune.sample_from(lambda spec: spec.config.uniform * 0.01), # Depends on other value"grid": tune.grid_search([32, 64, 128]) # Search over all these values }

• search? Algorithms（Search Algorithms (tune.suggest) — Ray v1.2.0所支持的搜索算法）

# Be sure to first run `pip install hyperopt`import hyperopt as hp from ray.tune.suggest.hyperopt import HyperOptSearch# Create a HyperOpt search space config = {"a": tune.uniform(0, 1),"b": tune.uniform(0, 20)# Note: Arbitrary HyperOpt search spaces should be supported!# "foo": tune.randn(0, 1)) }# Specify the search space and maximize score hyperopt = HyperOptSearch(metric="score", mode="max")# Execute 20 trials using HyperOpt and stop after 20 iterations tune.run(trainable,config=config,search_alg=hyperopt,num_samples=20,stop={"training_iteration": 20} )

• Trial Schedulers：查看訓練過程
• Analysis:分析訓練過程

（4）使用

from ray import tunedef objective(step, alpha, beta):return (0.1 + alpha * step / 100)**(-1) + beta * 0.1def training_function(config):# Hyperparametersalpha, beta = config["alpha"], config["beta"]for step in range(10):# Iterative training function - can be any arbitrary training procedure.intermediate_score = objective(step, alpha, beta)# Feed the score back back to Tune.tune.report(mean_loss=intermediate_score)analysis = tune.run(training_function,config={"alpha": tune.grid_search([0.001, 0.01, 0.1]),"beta": tune.choice([1, 2, 3])})print("Best config: ", analysis.get_best_config(metric="mean_loss", mode="min"))# Get a dataframe for analyzing trial results. df = analysis.results_df

?

創作挑戰賽新人創作獎勵來咯，堅持創作打卡瓜分現金大獎

總結

以上是生活随笔為你收集整理的深度学习-超参数调优的全部內容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網站內容還不錯，歡迎將生活随笔推薦給好友。