vLLM MTP: Faster Inference That Quietly Broke Tool Calls

The verdict. If you self-host a local LLM lane for always-on agents, ship the reliability lane first. On our vLLM 0.24.0, Qwen3.8-27B BF16, 262K-context lane, MTP=3 delivered a real throughput win on 18 August 2026. Two days later, concurrent long-context tool work turned into repeated ! reasoning, empty assistant messages, and finish_reason=length before a single tool call. We removed MTP from the production launcher. A speed feature that can make agents burn their own retry budget is not capacity. It is operational debt.

This is not a claim that speculative decoding is broken, that MTP is always unsafe, or that local inference is a bad buy. It is one dated failure class on one Blackwell workstation: an already-paid-for 96GB RTX PRO 6000 serving private-context Hermes, GBrain, and Kanban workers. The buying decision is narrower: when does faster decode stop being worth the reliability work you inherit?

1. The operator and the leak: agents that must call tools

The operator here is the founder running a small agent stack off one local lane. The model is not writing a disposable chat reply. It is reading a long worker prompt, deciding which tool to call, emitting valid arguments, then continuing from the result. If the first tool call does not happen, the workflow did not merely get slower. It stopped.

That distinction matters because a token benchmark can look excellent while the agent is already broken. On 18 August, eight non-thinking clients completed 6,180 tokens in 19.553 seconds, or 316.056 aggregate completion tokens per second. All eight stopped normally. The peak scheduler sample showed eight running and zero waiting; MTP accepted 3,595 of 7,758 draft tokens, 46.34%; peak temperature was 72C. The report calls this all_ok: true. Source: /home/kit/projects/vllm-qwen38-27b-bf16/reports/heavy-load-8-20260818-185540.json (total_completion_tokens, wall_elapsed_s, aggregate_completion_tok_s, peak_running, peak_waiting, metrics_delta, peak_temperature_c).

The trap is assuming that result describes every agent workload. It does not. The same day, eight thinking-mode clients each hit max_tokens=900, returned no visible content, and finished with length. The aggregate rate was still 212.126 tokens per second, MTP acceptance was 52.32%, and the run peaked at 90C. That is a client output-budget failure, not proof of model corruption: hidden reasoning spent the whole 900-token allowance. Source: /home/kit/projects/vllm-qwen38-27b-bf16/reports/heavy-load-8-20260818-185758.json (results[*].completion_tokens, content_chars, finish_reason, aggregate_completion_tok_s, metrics_delta.mtp_acceptance_pct, peak_temperature_c).

A local lane gives this operator two things the buy side may not: private context stays on-box, and the failure is inspectable. It also hands them the pager. If the lane is always on, the cost is not only hardware or API tokens. It is prompt caps, retry policy, queue policy, thermal policy, rollback time, and the discipline to say a fast benchmark is not a production pass.

2. How vLLM MTP changes the decision

Multi-token prediction (MTP) is speculative decoding in this lane. A draft path proposes several future tokens; vLLM verifies them against the target model. Accepted drafts can move decode forward faster. The useful number is therefore workload-specific acceptance, not the presence of the flag.

The 18 August direct API baseline showed why MTP looked worth trying. With MTP enabled, the raw report recorded 2,304 draft tokens and 1,152 accepted, exactly 50.0% acceptance. It also recorded a 493-token non-thinking response in 8.962 seconds (55.012 end-to-end tokens/s), a 339-token medium-reasoning response in 4.661 seconds (72.732 tokens/s), and correct exact-needle retrieval from a 60,040-token prompt in 16.161 seconds. An identical cached repeat took 0.564 seconds. Those are separate observations: short decode, cold prefill, and warmed prefix reuse are not one speed number. Source: /home/kit/projects/vllm-qwen38-27b-bf16/reports/raw-vllm-benchmark-20260818-184336.json (metrics_delta, cases named short_nonthinking, reasoning_medium, long_prefill_first, and long_prefill_prefix_cache_repeat).

The same raw report also caught a capacity boundary before the later failure: with max_num_seqs=1, two nominally simultaneous 536-token callers serialized at 10.404 and 20.822 seconds. After the 18 August change to max_num_seqs=8, telemetry still put KV-cache maximum concurrency at 1.69 at 262K context. Eight admits short or normal batching; it does not make eight full-context sessions physically real. Source: raw report notes and cases concurrent_2/concurrent_3; /Users/kit/.hermes/kit-log.md, 18 August 2026 entries at lines 71–75.

For the broader engine, model, and capacity trade-offs, read the Blackwell field guide. It covers the cutover and the separate benchmark table. This piece is about the reversal: a lane can win the meter and still lose the job.

3. Pick a lane based on the work that can fail

These are operating approaches, not vendor rankings. The rows describe what was measured on this lane and what still needs your own receipt.

Approach	Good for	Cost shape	Catch
MTP=3 throughput lane	Short, non-thinking batch work	GPU time; 316.056 aggregate tok/s observed at c8	This lane later degenerated on concurrent long-context tools
MTP-off reliability lane	Always-on agent tool loops	Same owned GPU; lower speed unmeasured after removal	This is the lane we ship after 20 Aug
No speculative decoding	Baseline and fault isolation	GPU time; measure your own throughput	Do not call it slower until you test it
Hosted API	Teams buying operations away	Requests/tokens, context, retries, and uptime	Get dated provider pricing for your traffic before comparing
Do nothing first	Agents failing at output caps	$0 config and policy work	Fix the token-budget trap before changing engines

The table has an uncomfortable answer for founders who already bought the GPU: do nothing first can be the right performance fix. The eight thinking-client run was not a reason to swap models or pay an API bill. The artifact says all eight used their 900-token cap in hidden reasoning. Raise or bound the client output policy, choose thinking deliberately, and log the finish reason before you call a model unhealthy.

The hosted comparison needs a different receipt. Price a month as: (input tokens × input rate) + (output tokens × output rate) + retry/continuation tokens + any fixed service charges. Add the prompt sizes and tool-call count your agents actually use. This article has no dated equivalent API-price receipt, so it makes no dollar break-even claim. For the site’s decision framing, see what the Lab costs: the work is the comparison, not a fake monthly total.

4. Failure modes: the ! was the bill

The later incident was not the earlier token-budget trap. On 20 August, six Kanban workers made concurrent long-context tool requests. All six reached finish_reason=length before their first tool call. Stored assistant text was empty; the recorded reasoning consisted only of repeated !, token ID 0. One task made four attempts and produced 229,376 reasoning characters. Five peers made three attempts and produced 163,840 reasoning characters each. Source: /Users/kit/.hermes/kit-log.md, 20 August 2026, 12:40 IST entry (lines 46–50).

That signature changes the incident response. A normal hard task can reason for a long time. A response that is almost entirely one punctuation token cannot make tool progress, and a continuation prompt cannot repair it. The right control is a degeneration guard: stop the request, retain the request record, and quarantine the lane or workload. Do not turn a broken stream into a more expensive broken stream.

Hermes made the blast radius worse. Its finish_reason=length path appended a continuation prompt and increased the requested output cap from 16,384 to 32,768, eventually up to 114,688 requested output tokens per failed turn. Separately, a stale Discord gateway session auto-continued for about 38 minutes at 100% GPU, roughly 300W, and 89C. Those are incident observations, not energy-cost math. Source: /Users/kit/.hermes/kit-log.md, 20 August 2026, 12:18 and 12:25 IST entries (lines 36–39 and 52–56).

There was a useful control experiment. After restart, the exact 44,334-character worker system prompt plus task produced correct kanban_context tool calls with thinking enabled and disabled. That rules out prompt wording as the sufficient explanation. It does not isolate a CUDA kernel or prove a universal root cause. The defensible label is an inference-degeneration class under this vLLM 0.24.0 / Qwen3.8-27B / MTP=3 / concurrent long-context-tool workload combination. Source: /Users/kit/.hermes/kit-log.md, 20 August 2026, 12:40 IST entry (lines 46–49).

There was outside corroboration, not a diagnosis. vLLM issue #35800 reports MTP plus Qwen tool-call degradation on Blackwell across fresh sessions until restart. An upstream issue can justify suspicion and a safe rollback; it cannot prove that every local ! loop has the same path.

The actual remediation was narrow. With explicit owner approval at 13:32 IST, we removed --speculative-config from the launcher, ran bash -n, kept a remote backup named serve-qwen38-27b.sh.bak-20260820-132128-mtp-enabled, verified the deployed SHA-256 matched, and restarted only the qwen38-vllm supervisor program. The verified runtime command had no speculative configuration. A bounded tool smoke then returned finish_reason=tool_calls, 71 reasoning characters, and a valid get_status {} result; the settled lane was zero running/queued, 16W, and 62C. Source: /Users/kit/.hermes/kit-log.md, 20 August 2026, 13:32 IST entry (lines 27–30).

One operational footnote matters because it is easy to misread a recovery: supervisorctl clear qwen38-vllm recreated an unlinked supervisor output log file after the restart. It did not restart the service. Keep repair actions separated in the incident record or the next person will credit the wrong change. Source: /Users/kit/.hermes/kit-log.md, 20 August 2026, 13:32 IST entry (line 30).

5. When not to use speculative decoding, or local at all

Skip MTP for the tool lane until it survives the same ugly work your agents do: long system prompts, concurrent tool requests, retries, cancellations, and a real client parsing the result. A clean curl response is not enough.

Skip a local always-on lane when the operational controls are missing. If nobody owns logs, output limits, a kill switch, and a rollback lane, the apparent savings are not measured. A hosted API may be the better purchase when you want an availability contract more than you want private context and diagnostic control. That is a cost-shape judgment, not a claim about any provider’s price.

Fix the $0 failures before you touch the serving engine. Set a per-attempt output ceiling. Do not auto-continue a one-token repetition signature. Bound agent dispatch to the lane’s proven concurrency rather than its configured admission count. Record tool-call success and first-tool-call time; aggregate tokens/s cannot substitute for either.

6. The 30-day test: make the speed feature earn its place

Use a seven-day baseline on the MTP-off reliability lane, then a 30-day production observation period. Do not create a synthetic benchmark that hides retries, cached prompts, or tool parsing.

Metric	Seven-day baseline	Day-30 target	Receipt
Degeneration events	Count exact repeated-token incidents	0	Request transcript and incident log
Tool-call success	Successful first calls / eligible turns	At least baseline	Agent run records
Time to first tool call	p50 over real eligible turns	No worse than baseline	Timestamped agent traces
Capacity and idle draw	Aggregate tok/s; settled idle W	Throughput within N%; idle W recorded	Load report and GPU sample

Treat N as your declared tolerance before day 1, not a number you choose after a good week. Also set X, the maximum auto-continuation duration, before the test. This lane already demonstrated that an unattended continuation can run for about 38 minutes; a production policy should stop well before your own X-minute boundary, preserve the request, and alert the owner.

Keep MTP only if the 30-day run has zero degeneration events, first-tool-call success is at least the baseline, throughput stays within your declared tolerance, and no continuation loop exceeds X minutes. Otherwise drop MTP from the tool lane and test it only in a bounded throughput lane. That is not anti-performance. It is refusing to pay for speed with invisible failed work.

7. Bottom line: tool success is the meter

MTP=3 earned its throughput claim on this vLLM lane: 316.056 aggregate completion tokens per second at eight non-thinking clients, with 46.34% MTP acceptance. It did not earn a place in the always-on agent tool lane after the 20 August repeated-! degeneration and retry amplification incident.

Run local when private context and operational control are worth owning. Start with the reliability lane, fix output-budget and retry-policy traps for $0, then let speculative decoding back in only after it survives the 30-day test against real tool work. A model that cannot reach its first tool call is not fast.

FAQs

Does speculative decoding hurt tool-call reliability?

It can in a specific lane. On 20 August 2026, vLLM 0.24.0 with Qwen3.8-27B BF16, MTP=3, and concurrent long-context tool work degenerated into repeated ! reasoning before any tool call. The same prompt worked after a restart with MTP removed. That is evidence about this configuration, not a general verdict on speculative decoding.

How do I benchmark a vLLM lane honestly?

Separate short decode, cold long-context prefill, warm-prefix reuse, real-concurrency traffic, and tool-call success. Record output caps, finish reasons, acceptance, queue state, and thermals. A throughput number without those controls cannot tell you whether the lane is safe for agents.

When is a local 96GB lane cheaper than API calls?

Only after you price the actual alternative: requests or tokens, prompt and output context, retries, and always-on agent volume. This workstation was already paid for and kept private context on-box; this article does not claim a universal API-versus-GPU break-even price.

What should an agent do after finish_reason=length?

Do not blindly auto-continue a response whose reasoning is almost entirely one repeated punctuation token. Stop it, preserve the request record, and treat it as an incident. In this case, length retries enlarged the requested output cap from 16,384 to 32,768 and eventually as high as 114,688 tokens per failed turn.

---

More on this decision, three ways to look at it:

Sources

• /home/kit/projects/vllm-qwen38-27b-bf16/reports/heavy-load-8-20260818-185540.json, observed eight-client non-thinking load result, 18 August 2026.
• /home/kit/projects/vllm-qwen38-27b-bf16/reports/heavy-load-8-20260818-185758.json, observed eight-client thinking output-budget result, 18 August 2026.
• /home/kit/projects/vllm-qwen38-27b-bf16/reports/raw-vllm-benchmark-20260818-184336.json, observed direct-API baseline, prefill, cache, and serialization results, 18 August 2026.
• /Users/kit/.hermes/kit-log.md, dated incident, remediation, and verification entries cited inline (18 and 20 August 2026).
• vLLM issue #35800, upstream Blackwell/Qwen/MTP corroboration; not proof of this incident’s exact root cause.