Design and Analysis of a Local Model Engineering runtime Gateway Based on vLLM

When using Ollama, several small models are started locally, and the business CoT (Chain of Thought) directly accesses these small models. Due to constraints, models under Ollama are difficult to run concurrently, and a fixed request is highly homogenized. Models can be selected based on requirements, and the risks of local small models lacking a toolchain are relatively limited.

However, after setting up local models using vLLM, we completely disconnect the ability to directly access the vLLM port. All functions are routed through the API, which completely changes these circumstances.

Simply inheriting the security protection of a single service for the vLLM service is no longer sufficient. It requires upstream intent recognition for small models, access control over the main model's toolchain invocation, admission control, and log recording with statistics.
Additionally, the primary responsibility is to host multiple services simultaneously. Driven by this requirement, features like queues, priority control, output filtering and judgment, failure retry mechanisms, and the crucial thinking budget control are further expanded.
These features might seem redundant, but they are absolutely not.

First, we need to identify users and provide different features for different users. Let's design three scenarios:

Scenario 1

A simple automated text summarization business often does not require a thinking process.
A VSCode programming task needs to display the complete thinking process.
For these two types of requirements, one can wait and then respond, while the other requires streaming transmission so that the model's thinking process can penetrate to the user.

Scenario 2

To briefly explain, an excessively long thinking budget will lead to low model utilization efficiency; many "think" models suffer from overthinking. There is also a severe issue: if the allocated token budget is entirely consumed by the thinking process, no result will be returned.

A simple query task A requires a thinking budget of 3000, and it might not even use it all.
A complex character classification and recognition task B in a book requires a thinking budget of 20,000 or more.
Therefore, in this scenario, the two users need different thinking budgets, and a soft budget might be more appropriate.

Scenario 3

User A needs to call the network to query some information.
User B does not need to call the network.
External information is dangerous. Ideally, no external information should be input at all, but we all know this is impossible. Therefore, the toolchain should serve as a double safeguard. If the current user has permission to access network tools, they can only access them when explicitly requested; this capability cannot be opened to users without protection. Thus, reorganizing the tool declaration after the request comes in, and appending tool_choice=none by default, can improve security to a certain extent.
Second, how to improve the availability of the service. Similarly, three scenarios:

Scenario 1: Multi-turn thinking and retry.

In VSCode, a single conversation might involve multiple turns of thinking. This brings up an issue: every time a multi-turn thought occurs, a new request is initiated, and the context of the previous turn disappears. If it cannot be passed to the next round's request, it means the next round starts from scratch. We need to pass the previous round's context to the next round based on the session ID. Another similar problem is that the model is extremely prone to falling into a loop of repetitive thinking; without the content of the previous thought, the model starts anew every single time (lol).
If a user's request fails due to a model issue, such as OOM, how should we retry? When retrying, we face the same problem: the thoughts generated in the previous round cannot be discarded.

Scenario 2: Shape learning of the thinking budget

As mentioned in Scenario 2 of the first point, giving too few tokens in a single request while encountering too many thinking tokens will result in thinking without an answer. Giving too many tokens will cause waste.
My method here is to dynamically adjust the user's thinking budget based on their request logs. This is the "shape learning" (pattern learning) of the gateway.
Allocate a thinking budget to the user in advance based on their historical requests. When returning the result, inspect it: if the thinking is not completed, we must intercept the return, allocate a larger budget to the current request, overlay the partial results into the body of the new request, and send the request again.

Scenario 3: Priority queue and interruption

After the model processes a request, among multiple services in the queue, which service can start first? This requires scanning.
After the user terminates the request, does the model still need to finish its thinking? My personal judgment is that it is no longer needed, and the model should interrupt its ongoing thinking.
Based on the above issues, we summarized the following architecture under the coexistence of multiple tasks.

Therefore, the model's capability is not unlocked simply by setting up vLLM and flipping a few switches and dial parameters on a black box.
It requires a massive amount of practical engineering experience and accumulation. These practices are our current mastery of the process, or rather, the "harness".
Of course, this feature is still very rudimentary and has many loopholes right now, but regardless, it's up and running, isn't it?