2.1 Hyperparameters

Hyperparameters are the variables of model to determine its structure and behavior. Recently, hyperparameters have received remarkable attention to deal with the computational complexity of the models. Each ML algorithm has its own specific set of hyperparameters that need to be tuned. Studies such as [34–36] explain the hyperparameters in detail. There are two major categories of hyperparameters: model and optimizer hyperparameters. Model hyperparameters designed the structure of model while the training optimization algorithms are optimizer hyperparameters.

The activation function is a kind of model-specific hyperparameter that makes a strong perception of critical and non-linear complicated functions to convert the input signals. The activation functions such as Sigmoid, Softmax, Tanh, and Rectified Linear Units (ReLU) are commonly used ⁽¹⁶⁾. ReLU is suggested as a default activation function that rectifies the vanishing gradient problem and converges six times faster than the Tanh function. The learning rate (LR) is a well-known hyperparameter that quantifies the rate and speed of the networks during learning. LR is used to adjust the weights of the hidden layers where the overall structure of networks that extract the complex features directly are represented. Optimal LR values need to be selected in order not to miss the local minimum. Recently, the LR annealing approach has been developed to find optimal value during training ⁽¹⁷⁾, ⁽¹⁸⁾. However, in most cases, the users manually set LR values. Data augmentation is a data generation technique by creating fake copies of the training data to perform specific functions such as transformation, rotation, cropping, etc. It helps the NNs during the learning to improve their performance.

Optimization algorithms play a prominent role in the tuning of neural networks (NNs). They make the networks learnable to perform better. In DL, Gradient Descent (GD), Root Mean Square Propagation (RMSprop), and Adaptive momentum estimation (Adam) ⁽¹⁹⁾–⁽²¹⁾ are widely used optimizers. Gradient Descent minimizes the cost function by updating the models’ parameters to reach the standard bias values in each step. It updates the weights and bias iteratively to get the local minima by adjusting the gradient. RMSprop optimizes the gradient by balancing the momentum (step size) and decreasing the step size for large gradients ⁽²⁰⁾. Another state-of-the-art optimizer is Adam which combines the two optimizers, Adaptive Gradient Descent (AdaGrad) ⁽²²⁾ and RMSprop. It computes the adaptive LR for each parameter and is used as the default optimization algorithm for the training.

2.2 HPO Techniques

In this section, we overview the HPO techniques such as GS, RS, BO, Genetic Algorithm (GA), and TPE [8, 9, 31-33].

3.1 Optuna

Optuna ⁽¹⁰⁾ is a newly developed open-source tool by a Japanese AI company for ML and DL applications in 2019. Optuna framework is built on the python language and guarantees the efficient optimization of complex problems ⁽¹⁰⁾. The existing HPO tools have many limitations, such as constructing a search space for each model individually, a lack of pruning approach, and handling the previous techniques within allocated resources ⁽⁶⁾. Optuna addressed these problems and provided a better solution by building the framework. It is a next-generation strategy that allows users to create a search space dynamically. Optuna delivers a lot of user-customized sampling, searching, and pruning algorithms for an efficient implementation. The versatile architecture of Optuna itself is easy to set up and can be deployed for different types of problems, ranging from scalable to lightweight experiments. The overall framework of Optuna for an ML model is shown in fig 1, where it automatically finds optimal combinations of hyperparameters using specific HPO sampling algorithms (GS, RS, BO, Hyperband). Later, the ML model is evaluated with a validation strategy to produce the final results.

Optuna is designed to solve the problems and limitations of black-box optimization frameworks. The implementation of Optuna is based on study and trial. The study requires an objective function that decides the number of samples in each trial and returns the optimal values for the specific hyperparameters. In contrast, a trial is the execution of the objective function. The key features and available HPO algorithms of Optuna are highlighted in table 2.

3.2 HyperOpt

HyperOpt ⁽¹¹⁾ is a python-based HPO tool based on sequential model-based optimization. This tool provides a user-friendly interface to configure the variables from the search space of hyperparameters and evaluate the objective function. The configured variables have continuous or discrete values, conditional ones, and sensitivity (uniform, log scaling). HyperOpt helps find the best deals on selected variables. These selected variables define the optimal hyperparameter configuration search space that minimizes the objective function as shown in fig 2.

HyperOpt key factors are a search space, objective function, and optimization algorithm. A search space in HyperOpt is chosen with random variables or parameters whose distributions have

higher combinations of prior probability. It includes the functions and operators in python language to combine the random combinations of parameters for the specific objective function. An objective function is defined with any conditional structure to map the sampling of different random parameter values to minimize the selective optimization algorithm score. HyperOpt supports the following optimization algorithms RS, TPE, and Adaptive TPE algorithms. HyperOpt search functions select the optimization algorithms, configure the best-performing optimal hyperparameters, and store the configuration results (see table 2).

3.3 Keras Tuner

In ML, there is no fixed way to select the optimal parameters such as the number of layers, and kernel size, and the optimizing parameter such as LR, decay, normalization, etc. to build the model. Keras is an open-source python API that led to the development of Keras Tuner, an HPO framework ⁽¹²⁾, to select the search space of hyperparameters and finds the optimal combinations of the values for training ML models. These hyperparameters play a vital role in generalizing the models to perform better.

Keras tuner is a library to tune the set of hyperparameters to obtain high performance for the imaging study ⁽³⁷⁾. The idea of the Keras tuner is to define the range of values of hyperparameters and get the optimal combination that improves the validation evaluation of the model, as shown in fig 3. Moreover, it helps to build a lightweight and efficient ML model that selects the optimal search space configuration to perform tuning (see table 2). Keras tuner has various built-in HPO methods: RS, GS, BO, hyperband, and evolutionary algorithms. It allows the researchers to conduct experiments with different techniques. RS, GS, and BO methods are explained in detail in section 3. The Keras Tuner-based hyperband is an extended version of the RS with an early stopping function to optimize the speed ⁽³⁸⁾. This framework is helpful in the tuning of the CNN model and allows the tuning of different numbers of the model's hyperparameters (convolutional layers, number of neurons and epochs, learning rate, etc.). Keras Tuner works as an HPO framework by optimizing the long-short-term memory (LSTM) network to predict the earthquake ⁽³⁹⁾.

4.1 Experimental Settings

For the performance evaluation, we selected four publicly available datasets named dry beans ⁽⁴⁰⁾, raisin ⁽⁴¹⁾, nomao ⁽⁴²⁾, and CIFAR-10 ⁽⁴³⁾ dataset. The selection of datasets is based on the real-world scenarios. The details of each dataset, including the number of samples, classes, and numerical and categorical features are summarized in table 3. CIFAR-10 is the only image dataset. It consists of 60K images with ten different classes with 10K images per each type. This dataset is extracted from the 80 million tiny images database where each RGB image is 32 X 32 pixels. The dataset was split into training and testing sets with 50K and 10K, respectively.

We used Python programming language with all the libraries needed to conduct the experiments. The training and testing of ML models were performed on a computer system with an Intel® Core™ i7-8700 CPU with a clock speed of 3.20 GHz, 16 GB of DRAM, and an NVIDIA GeForce GTX 1060 GPU with 8GB GDRAM. We evaluated accuracy, F1 score, and computing time as the performance evaluation metrics for the models.

4.2 Machine Learning Classifiers and the Experimental Results

We implemented three machine learning classification models including RF, XGB, and SVM, and evaluated their performance on the classification datasets (dry beans, raisin, nomao).

Random Forest (RF)

Random forest is a learning method that can analyze the classification and regression problems. It is an extension of decision tree algorithms and creates multiple trees where each tree is trained on the subset of the data. The results for all the trees are merged to build the final prediction resulting in a more robust and accurate model. RF can efficiently overcome the problem of overfitting. It has multiple hyperparameters to tune as discussed in table 4.

Extreme Gradient Boosting (XGB)

XGB is a type of supervised ML algorithm to improve traditional gradient boosting using decision trees. It is a highly efficient classifier that won numerous data science challenges. It builds the sequence of decision trees iteratively. Each new tree in the sequence is used to correct the error of the previous tree that results in performance improvement. The key hyperparameter of XGB to be optimized are gamma, n_estimators, and max_depth.

Support Vector Machine (SVM)

SVM is a ML method that can be commonly used for classification and regression analysis. It finds the best boundary to separate the classes in the data. New data points fall on the sides of the boundary and can be classified easily. The data points closer to the boundary are called support vectors. The SVM classifier works well for the smaller dataset and may struggle to handle high-dimensional data.

Each classifier has its own configuration space of hyperparameters that need to be tuned. The configuration space includes the tuned hyperparameters, type, space, and the range of values for each hyperparameter (see table 4). The complete process involves the following steps: data pre-processing, selection of search space with values, and selection of algorithm to perform the implementation. In data pre-processing, we handled the missing values by imputation (mean, forward, backward), one-hot encoding for the categorical features, use of label encoder to target features, and Min-Max scaling to all the numerical features in the dataset. The dataset was split into training and testing using the K-fold cross-validation technique. We used 5-fold cross-validation and 50 iterations to tune the hyperparameters in each fold. Hyperparameter tuning was performed with BO, Optuna, and HyperOpt HPO framework for machine learning tasks. Previous research suggests that Keras Tuner is primarily used for image classification rather than continuous classification tasks, thus we did not consider it. As mentioned earlier, our experiments used accuracy, F1 score, and the computing time as performance metrics.

Experiments were conducted on the benchmark datasets for the classification. The configuration search space of each classifier is based on the list of hyperparameters provided in table 4. In each experiment, models with high-accuracy results were returned and highlighted in table 5. HyperOpt-TPE performed the best and achieved the highest accuracy score in the case of dry beans and nomao with a similar score with others for the raisin dataset. On the other hand, the BO had the lowest score.

Comparing the performance of HyperOpt and Optuna in terms of accuracy, both frameworks performed well and achieved almost the similar results, while HyperOpt had an advantage with a little margin: HyperOpt achieved an accuracy of 94.12% and Optuna got 93.97%. Both frameworks used TPE to tune their hyperparameters. TPE requires upper and lower limits and a distribution that makes optimization easier. Compared with RS, GS, BO, and many other optimization algorithms, TPE can better utilize the resources by evaluating the hyperparameters efficiently, can handle non-linear relationships between selected hyperparameters and objective functions more effectively, and require lower computational costs than other algorithms for finding the best hyperparameters.

The raisin dataset has a smaller number of instances compared with other datasets. All frameworks had satisfactory accuracy scores. BO showed a higher accuracy score of 87.44% despite a longer run time than others. The use of the acquisition function to find the optimal combinations for smaller datasets led to better accuracy and optimization. The results from table 5highlight that BO achieved quite a reasonable accuracy of 87.17% and 87.44% for the dry beans and raisin datasets. Optuna and HyperOpt achieved the same results for the raisin dataset as shown in fig 4.

In the high dimensional nomao dataset, BO had a lower score while HyperOpt had a higher score with a slightly longer runtime. HyperOpt achieved an accuracy score of 94.12%. Each framework took a long time to optimize compared with other datasets due to its large size of samples. Optuna also achieved similar results as HyperOpt. Optuna generally performed well in terms of computing time due to an adaptive sampling feature that adjusts the search space dynamically and took less computation to find the optimal combinations. Overall, HyperOpt took more computing time compared with others due to the number of iterations for the classifiers during the experiments as shown in fig 5. It explored more search space and evaluated one set of hyperparameters at a time rather than simultaneously which resulted in a longer computing time.

HyperOpt and Optuna had good performance scores in all datasets, however, HyperOpt had a longer runtime. The performance

scores of HyperOpt and Optuna in terms of accuracy were almost the same, however, Optuna had a shorter run time. From the experiments, we ranked Optuna as the best choice for HPO considering the trade-off between the accuracy, F1 score vs. the computing time. HyperOpt can also be a good choice in the case of accurate prediction because it prioritizes accuracy over speed. Furthermore, it has the potential for parallel computation for complex problems with large data. In the end, it is important to note that the performance of the HPO framework relies on many factors such as the selection of the HPO method, the size of the dataset, choosing the search space of hyperparameter with the values, and the computational resources.

4.3 Deep Leaning CNN Model and the Experimental Results

In this section, we built a CNN architecture for the CIFAR-10 benchmark image dataset. The traditional CNN architecture has various layers such as convolution, dense, pooling, and fully connected layers. The selection of such layers and hyperparameters within each layer as well as the number of neurons and padding were made using HPO frameworks. The schematic diagram of the CNN model is presented in fig 6. The convolutional block in the CNN architecture consists of a convolutional layer with the ReLU activation function and the MaxPooling layer. Each convolutional layer is comprised of a 5 X 5 convolution filter, zero padding, and a stride of 1. The pooling layer has a MaxPooling filter with a size of 2 X 2. Both input and output images have the same size. This convolutional block takes the input images, extracts the sharp features, and forwards them to the hidden layers. These are converted into a single feature vector and passed to the fully connected layers of the network. Finally, the resulting output is classified using the Softmax activation function to compute the classification score.

A good choice of hyperparameters (learning rate, momentum, convolution, dense layers, hidden units, batch size, etc.) can lead to higher performance. All the possible hyperparameters to build the CNN model are highlighted in table 6. Each optimization algorithm has its specific number of hyperparameters with a range of values. table 6provides the search space of hyperparameters with values and optional hyperparameters of CNN. The configured values were obtained in the process of using HPO frameworks. For example, using BO resulted in the following configured values: learning_rate 0.000544, conv_layers 2, dense_layers 1, activation relu, and num_nodes 512. The optimal combinations of hyperparameters were validated on the testing set to predict the performance. The performance of each framework was measured and evaluated using the accuracy. Additionally, the computing time during the training of the CNN with a HPO method was measured to evaluate the efficiency.

Experiments were conducted using the CIFAR-10 dataset. The performance results of the CNN with and without HPO frameworks are shown in table 7. First, the training of CNN model was performed with the default hyperparameters for each optimization algorithm. The default values were chosen based on the previous knowledge and trained for 30 epochs. Then, the CNN model was trained with assigned search space hyperparameters for each HPO framework. The performance results of the optimized CNN are analyzed using an accuracy metric and the computing time.

The performance comparison with the four different HPO frameworks is summarized in table 7. With the default values of hyperparameters, the Keras Tuner-Hyperband achieved the best performance with 72.37% training accuracy, while BO, Optuna-TPE, and HyperOpt-TPE achieved 45.87%, 48.65%, and 42.28% training accuracy respectively. With HPO, HyperOpt-TPE improved the training accuracy by 34%, which was higher than all the other HPO frameworks.

Using BO, we achieved 72.88% training accuracy and 71.42% testing accuracy while it took 1 hour and 6 minutes of computing time. BO explored the search space of optimal hyperparameters effectively using a probabilistic model. The efficient exploration, ability to handle both the categorical and continuous values, and adoption of the most promising regions of search space led BO to improve its performance within a short training time.

Keras Tuner improved the CNN performance with a valuable margin. The complexity of the Keras Tuner framework is low due to fewer training parameters and the smaller search space size. Keras Tuner-Hyperband achieved the highest testing accuracy of 84.66% which outperformed other frameworks. However, Keras Tuner showed lower improvements in the training accuracy of 18.39%. With fewer parameters to optimize, it took only 18 minutes to complete the 30 epochs. The multiple features such as the easy-to-use tool interface, the ability to perform parallelization and early stopping, and multiple optimizations runs of optimization processes led to better results for CNN.

For Optuna and HyperOpt, the selected optimization algorithm was TPE. The achieved training accuracies for the HyperOpt-TPE and Optuna-TPE were 74.15% and 76.98% respectively as shown in fig 7. The search space of both the HPO frameworks was larger compared with the Keras Tuner and BO which increased the number of parameters and the complexity, thus resulting in a higher computing time (Optuna took 2 hours and 32 minutes). The reason for the CNN performance improvement using Optuna was due to the regularization and the pruning strategy. By allowing regularization which prevent overfitting, Optuna improved the generalization of the CNN model. The pruning strategy in Optuna eliminated the unpromising trials during the CNN optimization which resulted in better accuracy and the computing time. HyperOpt-TPE detected the best combinations with configured values early and improved the training accuracy by 34%, which was higher than all the other HPO frameworks. It allowed efficient exploration of the larger space to identify the combinations of hyperparameters for CNN model to improve the overall performance.

To summarize, experimental results show that CNN with HPO frameworks achieved significant improvements in training accuracy and detected optimal combinations of hyperparameters. The training accuracies with and without HPO shown in table 7show that all frameworks produced significant improvements. Keras Tuner improved by 18.39%, while BO, Optuna, and HyperOpt improved by 27.01%, 25.5%, and 34%, respectively. It was simple to implement the Keras Tuner and BO to detect the optimal configuration of hyperparameter for a smaller task where the cost matters a lot. Although BO took more computing time due to poor parallelization, there is still a tradeoff between the accuracy vs. the computing time using BO. On the other hand, Optuna and HyperOpt are effective optimization algorithms that worked well for large spaces and guarantee to detect the optimal combinations with the configured values. Larger spaces may include unimportant hyperparameters that increase the complexity of the problems, resulting in more computing time.