Latency and throughput
응답 속도(지연 시간)
The time from asking an AI until an answer starts to appear, and how fast the answer then pours out.
Overview
AI speed is measured in two ways. One is the wait from sending a request until the first characters appear, called latency. The other is how many tokens come out per second once the answer starts, called throughput.
Speed depends on the model's size, how deeply it thinks, the chips it runs on, and how many users are on it at that moment. Even the same model can differ depending on whose servers run it. Smarter models are usually slower and more expensive, so people choose by the job.
An everyday example
At a restaurant, the wait from ordering until the first plate arrives is latency. How fast the food keeps coming after that is throughput.
Why it matters
AI you talk with by voice, and coding tools, are frustrating to use if they are even a little slow. So news about a new model always mentions how fast it is along with how smart it is.
Also calledlatency, response time