Adaptive Optimizers
Any optimizer that gives each parameter its own effective learning rate from that parameter's own gradient history, rather than one shared rate for all of them.
Think of It Like This
Like giving each runner on a relay team their own personalized pace based on their fatigue, instead of forcing the entire team to match one rigid stopwatch.
Adam is the default nearly everyone starts with, keeping two running-average state vectors per parameter. RMSprop keeps one, dropping momentum. Adafactor compresses Adam's per-matrix second moment into row and column factors instead of the full matrix, cutting memory by orders of magnitude on large layers. Lion keeps one state vector and steps by the sign of it rather than its magnitude.
None of these displaces Adam by default — each answers a specific memory or batch-size constraint Adam runs into only at a particular scale.