Neural Architecture Search
Neural architecture search automates designing network structures, searching over layers and connections for the best design.
Designing networks by search
Neural architecture search (NAS) automates the design of neural network structures, the choice of layers, operations, and how they connect, that is otherwise handcrafted by experts. It is the network-design branch of AutoML, treating architecture as something to optimize rather than to invent by intuition and trial.
Three ingredients
- Search space: the set of allowable architectures, often built from repeatable cells to keep the space manageable
- Search strategy: how candidates are proposed, by reinforcement learning, evolution, or gradient-based relaxation
- Performance estimation: how each candidate is scored without training it fully to convergence
Making it affordable
Early NAS trained thousands of architectures from scratch, an enormous computational expense. Efficient methods slashed this cost. Weight sharing trains one over-parameterized supernet whose sub-networks share weights, so candidates are evaluated by inheriting weights rather than retraining. Differentiable NAS relaxes the discrete choice of operation into a continuous mixture, letting gradient descent optimize architecture and weights together, then discretizing the result.
Realistic objectives and cautions
Practical NAS optimizes more than accuracy: latency, memory, and energy enter the objective so the result fits its deployment target, a hardware-aware search. Cautions abound: results can be sensitive to the search space (a good space does much of the work), reproducibility has been a concern, and strong random or hand-designed baselines sometimes rival elaborate searches. Weight-sharing estimates can also rank architectures inaccurately, so top candidates are usually retrained fully before selection.
NAS shares its optimization machinery with hyperparameter optimization and Bayesian optimization.