Hyperparameter Optimization through Neural Network Partitioning with Christos Louizos - 627

Theory

Christos Louizos, an ML researcher at Qualcomm, explores the innovative use of neural network partitioning for hyperparameter optimization. This approach addresses the inefficiencies of traditional methods like random and grid search, which are less feasible for edge devices due to communication overhead and privacy concerns 1. By partitioning neural networks and datasets, Christos aims to optimize hyperparameters without exhaustive training runs, leveraging marginal likelihood techniques to enhance learning speed and generalization 2.

Every time you communicate something about your data, you reveal something about your data. So eventually, if you do it enough times, you basically spend most of your privacy budget.

--- Christos Louizos

This method not only conserves resources but also maintains data privacy, making it particularly suitable for federated learning environments.

   

Implementation

The practical implementation of this hyperparameter optimization involves defining subnetworks through random parameter selection, ensuring each subnetwork spans from input to output layers 3. This method allows for iterative training, where each subnetwork is optimized on local data, reducing the need for multiple training iterations 4.

We randomly select a subset of parameters and treat that as one specific subnetwork.

--- Christos Louizos

Initialization plays a crucial role, with random initialization proving more effective than zero initialization for non-participating parameters 5. This approach enhances efficiency and performance, crucial for edge device applications.

   

Edge Devices

Hyperparameter optimization through neural network partitioning offers significant advantages for edge devices, particularly in federated learning settings. By optimizing both parameters and hyperparameters jointly, communication costs are reduced, as updates are limited to subnetworks rather than entire networks 6. This method enhances computational efficiency and preserves privacy, crucial for devices with limited resources 7.

The main premise on why you want to do that is that the data are private and therefore cannot leave the device.

--- Christos Louizos

Additionally, this approach allows for efficient learning at scale, aligning with Qualcomm's focus on optimizing communication between servers and devices 8.