Hyperparameter Optimization 2 - Exploring Grid Search and Random Search for Hyperparameter Tuning in Python

In the previous blog post, we introduced the concept of hyperparameter optimization and its importance in machine learning. In this post, we will explore two popular techniques for hyperparameter tuning: Grid Search and Random Search. We will discuss their advantages and disadvantages and provide practical examples using Python and Scikit-learn.

Grid Search

Grid search is a simple and widely used method for hyperparameter tuning. It involves exhaustively searching through a predefined set of hyperparameter values and selecting the combination that results in the best model performance. The main advantage of grid search is that it is guaranteed to find the optimal set of hyperparameters within the search space. However, it can be computationally expensive, especially when dealing with a large number of hyperparameters or large datasets.

Random Search

Random search is an alternative to grid search that involves randomly sampling hyperparameter values from a predefined search space. Instead of exhaustively searching through all possible combinations, random search only evaluates a subset of them. This can significantly reduce the computational cost of hyperparameter tuning. Although random search is not guaranteed to find the optimal set of hyperparameters, it has been shown to be effective in practice, especially when the search space is large or the optimal hyperparameters are not uniformly distributed.

Example: Grid Search vs. Random Search in Python

In this example, we will compare grid search and random search for hyperparameter tuning using the Support Vector Machine (SVM) algorithm on the famous Iris dataset.

1. Import necessary libraries and load the dataset:

import numpy as np
import pandas as pd
from sklearn import datasets
from sklearn.model_selection import train_test_split, GridSearchCV, RandomizedSearchCV
from sklearn.svm import SVC
from sklearn.metrics import classification_report

iris = datasets.load_iris()
X = iris.data
y = iris.target

2. Split the dataset into training and testing sets:

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

3. Define the hyperparameter search space:

param_grid = {
    'C': [0.1, 1, 10, 100],
    'kernel': ['linear', 'rbf'],
    'gamma': [1, 0.1, 0.01, 0.001]
}

4. Perform hyperparameter optimization using GridSearchCV:

grid_search = GridSearchCV(SVC(), param_grid, cv=5, verbose=2)
grid_search.fit(X_train, y_train)

5. Perform hyperparameter optimization using RandomizedSearchCV:

random_search = RandomizedSearchCV(SVC(), param_grid, n_iter=10, cv=5, verbose=2, random_state=42)
random_search.fit(X_train, y_train)

6. Compare the results of Grid Search and Random Search:

best_grid_model = grid_search.best_estimator_
best_random_model = random_search.best_estimator_

y_pred_grid = best_grid_model.predict(X_test)
y_pred_random = best_random_model.predict(X_test)

print("Grid Search Results:")
print(classification_report(y_test, y_pred_grid))

print("Random Search Results:")
print(classification_report(y_test, y_pred_random))

Conclusion

In this blog post, we explored two popular techniques for hyperparameter tuning: Grid Search and Random Search. We discussed their advantages and disadvantages and provided a practical example using Python and Scikit-learn. In the next blog post, we will dive into more advanced techniques for hyperparameter optimization, such as Bayesian optimization and genetic algorithms.
Continue your learning by reading:
Automated Hyperparameter Tuning with Python Libraries, Optuna, Hyperopt, and Scikit-Optimize