Capsule Networks
Capsule networks represent entities as vectors whose length encodes presence and whose orientation encodes properties, aiming to preserve part-whole spatial relationships.
Beyond scalar neurons
A conventional neuron outputs a single scalar activation. A capsule instead outputs a vector. The vector's length represents the probability that a particular entity, such as an object or a part, is present, and its orientation encodes that entity's instantiation parameters, for example pose, scale, or deformation. The motivation is that pooling in ordinary convolutional networks discards precise spatial relationships, so a network can recognize that features are present without checking that they are arranged correctly.
The squashing nonlinearity
To keep vector lengths interpretable as probabilities, capsules apply a squashing function that shrinks short vectors toward zero and compresses long vectors to just under length one, while preserving direction. This makes the length a bounded confidence measure and lets the orientation carry the properties independently of the confidence.
Part-whole agreement
Lower-level capsules representing parts vote for the pose of higher-level capsules representing wholes, by multiplying their output with a learned transformation matrix. When many parts agree on the same whole, for instance the eyes, nose, and mouth all predicting a consistent face pose, the higher capsule activates strongly. This agreement mechanism, resolved by dynamic routing, is what encourages the network to respect part-whole geometry.
- Vector outputs preserve pose information that pooling would discard
- Aims for viewpoint robustness through explicit transformation matrices
- Reconstruction of the input from active capsules can act as a regularizer
- Routing is iterative and computationally heavier than a forward pass
Promise and difficulty
Capsule networks capture an appealing intuition about how vision might encode structured, hierarchical scenes, and they showed strong results on small, controlled datasets. In practice the routing procedure is expensive, and the approach has been hard to scale to large, complex images relative to convolutional and transformer models. The core routing algorithm is detailed in dynamic routing between capsules.