Observability for LLM Inference: Token Metrics, Queues, and Tail Latency

You know that feeling when a chatbot freezes right after you hit send? It’s not just annoying; it’s expensive. As we move deeper into 2026, deploying Large Language Models (LLMs) in production is no longer just about picking the smartest model. It’s about managing the chaotic reality of inference infrastructure. If you are running LLMs at scale, traditional monitoring tools are lying to you. They show you averages, but your users live in the tails.

Real observability for LLM inference means looking past simple uptime checks. You need to understand token throughput, queue dynamics, and the specific latency spikes that kill user experience. This isn't optional overhead anymore. It is the difference between a scalable service and one that collapses under its own weight.

The Problem with "Requests Per Second" in LLM Systems

If you are used to monitoring standard web APIs, you probably rely heavily on requests per second (RPS). That metric is useless for LLMs. Why? Because LLM workloads are wildly variable. One request might ask for a simple "yes or no" answer, consuming ten tokens. Another might demand a 2,000-word essay. Both count as one request in your dashboard, but they consume vastly different amounts of compute power and memory bandwidth.

When you track only RPS, you can see stable traffic while your actual token throughput collapses. This disconnect hides bottlenecks until they become critical failures. Instead of RPS, you need to shift your focus to token metrics. These metrics reveal the true load on your system.

Core token metrics include:

• Prompt Tokens: The input data sent to the model. Tracking this helps you understand context window usage and prefill costs.
• Completion Tokens: The output generated by the model. This is where the heavy computation happens during autoregressive generation.
• Total Token Consumption: Aggregated by model, user, or feature to calculate accurate cost per interaction.

Tools like Text Generation Inference (TGI) and vLLM export these metrics explicitly. Look for counters like `tgi_request_generated_tokens` or histograms tracking `gen_ai.client.token.usage`. By aggregating these, you stop guessing why your GPU utilization spiked and start knowing exactly which prompts caused it.

Decoding Latency: TTFT vs. Inter-Token Latency

Latency in LLM systems isn't a single number. It breaks down into two distinct experiences for the user: waiting for the first word, and watching the rest stream in. Confusing these two leads to poor optimization strategies.

Time-to-First-Token (TTFT)

Time-to-First-Token (TTFT) is the initial delay before any text appears on the screen. This is the "is it working?" moment for users. Research shows that for every additional input token, the P95 TTFT increases by approximately 0.24 milliseconds. While that sounds small, long contexts amplify this delay significantly.

In practice, sub-50ms TTFT feels instantaneous. Anything above 200ms starts to feel sluggish. If your TTFT jumps to several seconds, users assume the application has frozen. Monitoring TTFT requires histogram-type metrics, such as `vllm:time_to_first_token_seconds`. A sudden rise in TTFT is often the first sign that your inference engine is saturated or that your batch size is too large for the current hardware constraints.

Inter-Token Latency

Once the first token arrives, the user cares about the flow. Inter-token latency measures the time between subsequent tokens. If this latency rises, the response feels fragmented or stuttering, even if the total completion time is acceptable. This metric directly impacts perceived responsiveness. Track this using metrics like `tgi_request_mean_time_per_token_duration`. High inter-token latency usually indicates memory bandwidth bottlenecks or inefficient scheduling within the inference engine.

Queueing Dynamics: The Hidden Bottleneck

Most modern LLM deployments use continuous batching to maximize GPU utilization. This creates a queue. Understanding this queue is critical because LLM inference follows an M/G/1 queueing model, where the variability of service times (output length) drastically affects wait times.

Here is the trap: a few requests asking for very long outputs create a "heavy tail" in the distribution. These long requests hold up the entire batch, extending the average queuing delay for everyone else. According to recent queueing theory analysis, this can cause a significant percentage of impatient users to abandon the platform before their request is even processed.

To manage this, you must monitor queue-specific metrics:

• Queue Wait Time: Reveals delays caused by waiting for an available replica or slot in the batch.
• Active Queue Size: Shows how many requests are pending execution.
• Batch Size: Indicates how many requests are being processed simultaneously.

Both TGI and vLLM expose these indicators. If you see queue wait times climbing while GPU utilization remains high, you have a scheduling inefficiency. You aren't lacking compute; you're lacking efficient packing.

Taming Tail Latency with Token Limits

Average latency is a vanity metric. Nobody cares about your average if the 99th percentile (P99) is terrible. Tail latency defines the actual user experience for the most frustrated segment of your audience. In LLM systems, tail latency is driven by the heavy-tailed distribution of output lengths. Some users want a summary; others want a novel chapter. The latter drags down performance for everyone.

Research suggests a counter-intuitive solution: enforcing maximum output token limits. By capping the output length for a small fraction of requests, you can significantly reduce overall queuing delay. The trade-off is quality versus speed. If the limit is too low, you truncate useful responses. If it's too high, you accept longer waits.

Your observability system must provide the data to make this trade-off intelligently. You need to correlate token distribution histograms with queue depth and drop rates. If you notice that requests exceeding 2,000 tokens cause queue depths to spike beyond a certain threshold, you might implement dynamic routing. Short requests go to a low-latency pool; long requests go to a best-effort pool with higher timeout allowances.

Implementing Observability: Tools and Standards

You don't need to build this from scratch. The ecosystem has matured significantly. Standardization is key to avoiding vendor lock-in and ensuring consistent data collection across different models and frameworks.

Adopt OpenTelemetry semantic conventions for GenAI. These conventions provide standardized metric names like `gen_ai.server.time_to_first_token` and `gen_ai.server.time_per_output_token`. Using these standards ensures that whether you switch from vLLM to TGI or integrate a new proprietary model, your dashboards remain relevant.

For aggregation and alerting, pair these exporters with Prometheus and Grafana. Set up alerts based on percentiles, not averages. For example, alert if P95 TTFT exceeds 500ms for more than five minutes. Also, monitor cost anomalies. Different models have vastly different cost profiles. An unexpected spike in token consumption could indicate a prompt injection attack or a runaway recursive loop in your application logic.

Finally, integrate business-level metrics. Track success rates (`vllm:request_success_total`) alongside technical metrics. A high error rate combined with low latency might mean your model is rejecting inputs rather than processing them slowly. Context matters.

Next Steps for Your Infrastructure

Start by auditing your current metrics. If you are only tracking HTTP status codes and request counts, you are flying blind. Instrument your inference servers to export token-level and latency-percentile data. Map out your queue behavior under load. Identify the heavy tails in your output distribution. Then, decide where to draw the line between speed and completeness. Observability isn't just about watching; it's about giving you the leverage to optimize.