Token
토큰
The small pieces of text an AI reads and writes. Also the unit that measures usage and price.
Overview
An LLM does not handle text as letters or whole words. It turns the text into smaller pieces first. These pieces are tokens. A short word is one token, and a long word is split into several.
AI companies charge by the number of tokens read and written. The length of text an AI can handle at one time is also set in tokens.
An everyday example
A taxi fare is measured by distance. AI usage is measured by tokens. Reading a long document or getting a long answer uses a lot of tokens.
Why it matters
Token counts are used to compare how much different AIs are really used. LastAI's usage ranking is also ordered by how many tokens pass through.