A2C and A3C

A3C runs many parallel actor-critic workers asynchronously; A2C is its synchronous, simpler variant that often matches it.

Parallel actor-critic

A3C (Asynchronous Advantage Actor-Critic) and A2C (Advantage Actor-Critic) scale actor-critic learning by running many copies of the agent in parallel environments. The parallelism decorrelates the data that on-policy methods depend on, replacing the role that a replay buffer plays for off-policy methods.

A3C: asynchronous

A3C runs multiple worker threads, each with its own environment and a local copy of the network. Workers compute gradients from their own experience and apply them asynchronously to a shared global network, then pull the updated weights. Because workers explore different parts of the environment at different times, their combined updates are diverse and decorrelated, stabilizing training without a replay buffer.

A2C: synchronous

A2C is the synchronous version: all workers step their environments in lockstep, their experiences are gathered into one batch, and a single coordinated update is applied. This removes the noise and complexity of asynchronous updates, uses hardware more efficiently through batching, and in practice performs as well as or better than A3C.

The advantage weight

Both use the advantage as the actor's weight, typically estimated with n-step returns or GAE. The critic learns a state-value function to supply the baseline and bootstrap the returns.

Legacy

A2C and A3C showed that on-policy actor-critic could be both stable and efficient at scale using parallel environments rather than experience replay. They are direct predecessors of PPO, which keeps the parallel-rollout structure but replaces the plain policy-gradient step with a clipped objective for safer, larger updates.