Automatic Differentiation: Forward and Reverse
A technique to compute exact derivatives of code by applying the chain rule mechanically, distinct from both symbolic and finite differences.
Neither symbolic nor finite differences
Automatic differentiation (AD) computes derivatives of a function defined by a computer program. It is not symbolic differentiation (which manipulates formulas and can explode in size) and not finite differences (which approximate and suffer roundoff and step-size error). AD decomposes the program into elementary operations, each with a known derivative, and applies the chain rule to accumulate exact derivatives to machine precision.
Forward mode
Forward-mode AD carries, alongside each intermediate value, its derivative with respect to one input. It propagates these derivatives forward through the computation in step with the values. Its cost is proportional to the number of inputs: computing the gradient of a function of many variables would require one forward pass per input. Forward mode is efficient when there are few inputs and many outputs, and it can be implemented elegantly with dual numbers.
Reverse mode
Reverse-mode AD first runs the function forward, recording the computation (the tape), then propagates derivatives backward from the outputs to the inputs. Its cost is proportional to the number of outputs. For a scalar-valued function of many inputs, such as a loss or an objective, a single reverse pass yields the full gradient at a cost of a small multiple of one function evaluation. This is exactly the setting of machine learning, where reverse-mode AD is called backpropagation.
- Forward mode: cheap for few inputs, many outputs; propagates derivatives with values
- Reverse mode: cheap for many inputs, few outputs; needs to store the computation tape
- Both give exact derivatives, no truncation or step-size error
Relation to the adjoint method
Reverse-mode AD is the discrete, mechanical form of the adjoint method used in PDE-constrained optimization. Both compute the sensitivity of a scalar output to many parameters at a cost independent of the number of parameters. AD is central to gradient-based optimization of simulation codes, including tuning design parameters and inverse problems in plasma modeling.