Padding

Padding is a way of filling in blanks so that an input reaches a required length. The structures that run language models often fix in advance the input length they will process, so when the real sentence is short, the remaining space is filled with a special blank value.

This blank holds no meaningful content. It is only a spot that fits the format. So it is often hidden with a mask so that it does not affect the calculation. Still, if the structure also spends compute on the blank positions, they cost time and power too.

A fixed-length graph is a structure that computes every position up to the set length. If you put 50 tokens into a graph built for 1,024 slots, the other 974 slots are filled with blanks and calculated in the same order.

A mask is a mark that tells the attention calculation to leave the blanks out so that they do not mix into the result. But if the structure does not reduce the computation at the blank positions even with a mask, that cost stays.

In a case like the Chimera GPNPU in the article, a single graph handles several lengths up to a set maximum, which reduces the need to build a separate binary for each length range. It is useful on an edge device that handles both short voice commands and long documents with one chip.

A condition is that the runtime environment must support variable lengths. Whether to reduce the burden of building and verifying a binary for each length, or the wasted compute on short inputs, should be decided according to what the product needs.

Padding does not always mean a big waste. The share of blanks depends on the distribution of input lengths and on the maximum length setting, so it is hard to say how much is lost until you measure real inputs.

A common misunderstanding is to think that blanks make the result wrong. Blanks hidden by a mask are left out of the meaning calculation, so they usually do not affect accuracy. Compute time and energy can still be spent, depending on how it is implemented.

It is like a large bus with a set number of seats. Even if only two passengers ride, the empty seats are still included in the running cost.

In services where input length varies a lot, you need to know how much compute padding uses so that you can choose the right model size and runtime environment.