Latency
The time between sending a request and getting a response — for AI chat, split into 'time to first token' (perceived latency) and 'total response time'.
Latency in AI chat has two components that matter for UX. Time to first token (TTFT) is the delay from clicking send to seeing the first word of the response. Total response time is TTFT plus however long the rest of the response takes to stream.
Time to first token is what users perceive as "responsiveness". Under 500ms feels instant; 500ms-2s feels acceptable; over 2s feels like the system is thinking. Total response time is less important because streaming lets you start reading immediately — a slow full response feels fine as long as the first sentence arrived quickly.
Frontier models in 2026 typically deliver TTFT in the 200-500ms range on well-provisioned infrastructure, with total response depending heavily on response length. Cerebras-powered inference (used by Mistral's Le Chat) can hit 1000+ tokens per second, which effectively eliminates total-response-time as a UX concern. Standard GPU inference is meaningfully slower per-token but still usable for chat.