A new paper on tiered large language model inference sets out a condition for when no confidence threshold can make a local-to-datacenter model cascade worthwhile compared with sending every query to the datacenter backstop. The work is published as a position paper on arXiv.
Tiered inference runs models across devices such as user hardware and edge systems, as well as datacenters. In systems that decide whether to answer a query locally or escalate it, that decision often relies on a confidence signal from the forward model. The paper treats routing as a binary test and considers both how well the signal detects local errors and how often it escalates queries the local model would have answered correctly.
The analysis introduces an architecture cost ratio, κ, defined as the marginal cost of a local model pass relative to an escalation. The authors derive a feasibility condition bounded by two limits: a signal-independent floor below which a cascade is feasible regardless of signal quality, and a perfect-signal ceiling above which no cascade is feasible. The paper also reports that the condition reverses across deployment settings.
When local computation is billed, as in a cloud API, its cost can constrain whether a cascade is worthwhile. When the hardware is already owned and the marginal cost of another local pass approaches zero, feasibility becomes unconditional in the model; signal quality then determines the results the cascade can achieve. The authors say the cost conclusion therefore applies to paid-compute settings rather than all decentralized deployments.
The paper also identifies three requirements for an architecture intended to overcome the feasibility limit: a calibrated escalation signal available at near-zero marginal cost, a cross-tier payload that can be compressed to fit available bandwidth, and placement that respects locality. It says existing techniques meet these requirements separately, but no current model meets all three together.
The proposed co-designed architecture and a protocol for testing it are presented as future work. The paper does not implement or evaluate the architecture, and describes its contribution as the feasibility condition and the resulting analysis of deployment regimes.
Sources: arXiv cs.AI.