Bit depth is the number of bits used per colour channel per pixel. 8-bit video has 2^8 = 256 possible values per channel; 10-bit has 1,024.
That fourfold increase sounds excessive until you look at a gradient. A sky fading from light to dark across 1,000 pixels of screen has more distinct positions than 8-bit has values to assign them, so the transitions have to repeat — which is visible as banding.
Practical points:
- 10-bit helps even for 8-bit output. The encoder's internal calculations round less, so the delivered 8-bit file has fewer visible steps. This is well established practice in animation encoding for exactly this reason.
- It is required for HDR. High dynamic range content in 8-bit is not meaningfully possible.
- It costs compatibility. 10-bit H.264 is poorly supported by hardware decoders and is a reliable way to produce a file that will not play on a TV. 10-bit H.265 is much better supported, because it was part of the standard's mainstream profile from the start.
- The size cost is small, and sometimes negative: the cleaner internal precision can encode more efficiently than the extra data costs.
For everyday compression the answer is usually to leave it alone and match the source. Reach for 10-bit specifically when the content has large smooth gradients and the output codec is H.265 or AV1.