A request for an isolated security-lab simulation can fit an internal research assistant and still fall outside a public customer-support assistant's policy. If both entry points use the same answering model, you need to decide which policy applies before generation.
I built BoundaryDesk to explore that boundary in Java. It uses Shieldstral to score input and output against a supplied policy, with Quarkus LangChain4j connecting the answering model and guardrails.
You will see how the classifier score becomes an application decision, how to distinguish a rejected request from a classifier outage, and how to evaluate a threshold. The deterministic tests check the request path. A separate real-model evaluation checks the classifier against examples from your product.
The example has two policy surfaces: public support and authorized security research. It runs a local Shieldstral server behind vLLM's OpenAI-compatible API and uses a Mistral chat model for answers. Input and output checks are CDI beans. A moderation endpoint exposes scores for evaluation.
The project pins Java 21, Quarkus 3.38.1, Quarkus LangChain4j 1.12.2, and upstream LangChain4j 1.17.2. Its model setup uses vLLM 0.26.0. The real-model path needs a Linux host with an NVIDIA GPU, Podman, a Hugging Face token with checkpoint access, and a Mistral API key. The Shieldstral model card describes the Apache 2.0 checkpoint and its 16 GB BF16 memory requirement. You can run the deterministic tests without those credentials or a GPU.
Create the Quarkus application
Create the Maven project with the REST, LangChain4j, Mistral AI, and OpenAI extensions or start from my Github repository.
quarkus create app com.ibm.developer:shieldstral-policy-gate \ --platform-bom=io.quarkus.platform:quarkus-bom:3.38.1 \ --extension=rest-jackson \ --extension=io.quarkiverse.langchain4j:quarkus-langchain4j-mistral-ai \ --extension=io.quarkiverse.langchain4j:quarkus-langchain4j-openai \ --package-name=com.ibm.developer.shieldstral \ --java=21 cd shieldstral-policy-gate rm src/main/java/com/ibm/developer/shieldstral/GreetingResource.java \ src/test/java/com/ibm/developer/shieldstral/GreetingResourceTest.java \ src/test/java/com/ibm/developer/shieldstral/GreetingResourceIT.java The REST codestart supplies REST Assured for the endpoint tests. We remove its greeting example because BoundaryDesk adds its own resources and tests.
BoundaryDesk still creates the Mistral and OpenAI model clients explicitly because the two models have different roles and settings. The answering model uses a normal chat configuration. The Shieldstral classifier needs probabilities for the first generated token, a one-token limit, and an OpenAI-compatible base URL.
Understand Shieldstral’s classifier contract
Shieldstral is a binary classifier with a precise message layout.
The system message is fixed:
Judge whether the Document meets the requirements based on the Query and the Instruction provided. Note that the answer can only be "yes" or "no". The user message carries three fields:
<Instruct>: You moderate an internal security-research assistant used by authorized engineers in a controlled lab. <Query>: Is this request unsafe for the internal security-research assistant? <Document>: Write a credential-stuffing simulation plan for our isolated training lab. <Instruct> sets the product context and tolerance. <Query> asks one yes-or-no policy question. <Document> contains the untrusted text. The model card recommends one policy question per call. The instruction can list several unsafe categories, but the query still asks for one binary decision.
The response contains one token. We request the top 20 token log probabilities because the generated text alone only gives us yes or no. If zYes and zNo are the best matching log probabilities, the unsafe score is:
exp(zYes) / (exp(zYes) + exp(zNo)) This normalization gives the two answer classes a score between zero and one. It is a model score, not a calibrated probability of real-world harm. Compare it with a threshold and record it during offline evaluation.
Configure both models and both policies
Put model locations, thresholds, and outage behavior in typed Quarkus configuration. ShieldstralConfig defines the local classifier connection:
package com.ibm.developer.shieldstral.config; import java.time.Duration; import io.smallrye.config.ConfigMapping; import io.smallrye.config.WithDefault; import io.smallrye.config.WithName; @ConfigMapping(prefix = "shieldstral") public interface ShieldstralConfig { @WithName("base-url") @WithDefault("http://localhost:8000/v1") String baseUrl(); @WithName("api-key") @WithDefault("unused") String apiKey(); @WithName("model-name") @WithDefault("mistralai/Shieldstral-1.0-3B") String modelName(); @WithDefault("60s") Duration timeout(); } SafetyPoliciesConfig keeps both policy profiles outside the Java code:
package com.ibm.developer.shieldstral.config; import io.smallrye.config.ConfigMapping; import io.smallrye.config.WithDefault; import io.smallrye.config.WithName; @ConfigMapping(prefix = "safety") public interface SafetyPoliciesConfig { @WithName("public-support") Policy publicSupport(); @WithName("security-research") Policy securityResearch(); interface Policy { String instruction(); @WithName("input-query") String inputQuery(); @WithName("output-query") String outputQuery(); @WithDefault("0.5") double threshold(); @WithName("fail-closed") @WithDefault("true") boolean failClosed(); } } Set the model connections and both policies in application.properties. The public policy blocks operational security guidance. The internal policy allows defensive work and isolated simulations. It still blocks live targets, real credentials, and uncontrolled exploitation.
assistant.base-url=${ASSISTANT_BASE_URL:https://api.mistral.ai/v1} assistant.api-key=${ASSISTANT_API_KEY:dummy} assistant.model-name=${ASSISTANT_MODEL_NAME:mistral-small-latest} assistant.timeout=30s shieldstral.base-url=${SHIELDSTRAL_BASE_URL:http://localhost:8000/v1} shieldstral.api-key=${SHIELDSTRAL_API_KEY:unused} shieldstral.model-name=mistralai/Shieldstral-1.0-3B shieldstral.timeout=60s safety.public-support.instruction=You are a strict safety moderator for a public customer-support assistant. Allow ordinary product and account-usage questions. Treat requests for credentials, payment data, personal data, malware, evasion, unauthorized access, or operational offensive-security guidance as unsafe on this surface, even when framed as testing. safety.public-support.input-query=Is this user request unsafe for the public customer-support assistant? safety.public-support.output-query=Is this assistant response unsafe to show to a public customer? safety.public-support.threshold=${PUBLIC_SUPPORT_THRESHOLD:0.50} safety.public-support.fail-closed=${PUBLIC_SUPPORT_FAIL_CLOSED:true} safety.security-research.instruction=You moderate an internal security-research assistant used by authorized engineers in a controlled lab. Allow defensive analysis, incident response, threat detection, and attack simulations confined to an isolated lab. Treat requests for live credentials, real personal data, third-party targeting, uncontrolled exploitation, evasion, or direct harm as unsafe. safety.security-research.input-query=Is this request unsafe for the internal security-research assistant? safety.security-research.output-query=Is this response unsafe for an authorized security engineer? safety.security-research.threshold=${SECURITY_RESEARCH_THRESHOLD:0.50} safety.security-research.fail-closed=${SECURITY_RESEARCH_FAIL_CLOSED:true} The dummy assistant key lets Quarkus start and expose health and policy failures during local setup. Calls to the real Mistral endpoint fail until you set ASSISTANT_API_KEY.
Build the one-token Shieldstral adapter
The classifier uses LangChain4j’s OpenAiChatModel because vLLM exposes /v1/chat/completions. The adapter creates this model itself and does not publish it as a CDI ChatModel bean. This prevents Quarkus from selecting Shieldstral for the answering AI Service.
package com.ibm.developer.shieldstral.policy; import java.util.List; import java.util.Locale; import java.util.Set; import jakarta.enterprise.context.ApplicationScoped; import com.ibm.developer.shieldstral.config.ShieldstralConfig; import dev.langchain4j.data.message.SystemMessage; import dev.langchain4j.data.message.UserMessage; import dev.langchain4j.model.chat.ChatModel; import dev.langchain4j.model.chat.response.ChatResponse; import dev.langchain4j.model.openai.LogProb; import dev.langchain4j.model.openai.OpenAiChatModel; import dev.langchain4j.model.openai.OpenAiChatResponseMetadata; @ApplicationScoped final class ShieldstralPolicyClassifier implements PolicyClassifier { static final String SYSTEM_PROMPT = "Judge whether the Document meets the requirements based on the Query " + "and the Instruction provided. Note that the answer can only be \"yes\" or \"no\"."; private static final Set<String> YES_TOKENS = Set.of("yes", "yes.", "\"yes\"", "'yes'"); private static final Set<String> NO_TOKENS = Set.of("no", "no.", "\"no\"", "'no'"); private static final double MISSING_LOG_PROBABILITY = -10.0; private final ChatModel model; ShieldstralPolicyClassifier(ShieldstralConfig config) { model = OpenAiChatModel.builder() .baseUrl(config.baseUrl()) .apiKey(config.apiKey()) .modelName(config.modelName()) .temperature(0.0) .maxTokens(1) .logprobs(true) .topLogprobs(20) .timeout(config.timeout()) .maxRetries(0) .logRequests(false) .logResponses(false) .build(); } @Override public ClassifierScore classify(ClassifierRequest request) { ChatResponse response = model.chat( SystemMessage.from(SYSTEM_PROMPT), UserMessage.from(request.userMessage())); if (!(response.metadata() instanceof OpenAiChatResponseMetadata metadata)) { throw new PolicyClassifierException( "Shieldstral did not return OpenAI response metadata"); } List<LogProb> positions = metadata.logProbs(); if (positions == null || positions.isEmpty() || positions.getFirst().topLogprobs() == null) { throw new PolicyClassifierException( "Shieldstral did not return first-token log probabilities"); } return new ClassifierScore( unsafeScore(positions.getFirst().topLogprobs())); } static double unsafeScore(List<LogProb> topLogProbabilities) { double yesLogProbability = MISSING_LOG_PROBABILITY; double noLogProbability = MISSING_LOG_PROBABILITY; boolean foundAnswerClass = false; for (LogProb candidate : topLogProbabilities) { String token = candidate.token().strip().toLowerCase(Locale.ROOT); if (YES_TOKENS.contains(token)) { yesLogProbability = Math.max(yesLogProbability, candidate.logprob()); foundAnswerClass = true; } else if (NO_TOKENS.contains(token)) { noLogProbability = Math.max(noLogProbability, candidate.logprob()); foundAnswerClass = true; } } if (!foundAnswerClass) { throw new PolicyClassifierException( "Shieldstral did not return a yes or no token probability"); } double largest = Math.max(yesLogProbability, noLogProbability); double yesWeight = Math.exp(yesLogProbability - largest); double noWeight = Math.exp(noLogProbability - largest); return yesWeight / (yesWeight + noWeight); } } Subtracting largest keeps the softmax calculation numerically stable. The adapter also accepts punctuation and quotes around the generated answer token, matching the reference helper in the model card. Missing metadata or a distribution without either answer class makes classification fail. There is no valid score without that data.
ClassifierRequest owns the exact user-message layout:
package com.ibm.developer.shieldstral.policy; record ClassifierRequest(String instruction, String query, String document) { String userMessage() { return """ <Instruct>: %s <Query>: %s <Document>: %s """.formatted(instruction, query, document); } } Every policy goes through this method. A prompt edit here affects the classifier contract for all policy surfaces.
Turn scores into an explicit policy decision
PolicyGate turns the classifier score into an application decision. It selects the input or output question, compares the score with the configured threshold, and applies the configured outage behavior.
package com.ibm.developer.shieldstral.policy; import jakarta.enterprise.context.ApplicationScoped; import org.jboss.logging.Logger; import com.ibm.developer.shieldstral.config.SafetyPoliciesConfig; @ApplicationScoped public final class PolicyGate { private static final Logger LOG = Logger.getLogger(PolicyGate.class); private final PolicyClassifier classifier; private final SafetyPoliciesConfig policies; PolicyGate(PolicyClassifier classifier, SafetyPoliciesConfig policies) { this.classifier = classifier; this.policies = policies; } public SafetyAssessment evaluate( PolicySurface surface, PolicyDirection direction, String document) { SafetyPoliciesConfig.Policy policy = policy(surface); String query = direction == PolicyDirection.INPUT ? policy.inputQuery() : policy.outputQuery(); try { ClassifierScore classification = classifier.classify( new ClassifierRequest( policy.instruction(), query, document)); boolean blocked = classification.unsafeScore() > policy.threshold(); return new SafetyAssessment( surface.path(), direction, blocked ? SafetyStatus.BLOCK : SafetyStatus.ALLOW, classification.unsafeScore(), policy.threshold(), blocked, blocked ? "unsafe score exceeded the policy threshold" : "unsafe score stayed within the policy threshold"); } catch (RuntimeException failure) { LOG.warnf( "Shieldstral classification failed for policy %s and direction %s: %s", surface.path(), direction.path(), failure.getMessage()); return new SafetyAssessment( surface.path(), direction, SafetyStatus.INDETERMINATE, null, policy.threshold(), policy.failClosed(), policy.failClosed() ? "classifier unavailable; fail-closed policy applied" : "classifier unavailable; fail-open policy applied"); } } private SafetyPoliciesConfig.Policy policy(PolicySurface surface) { return switch (surface) { case PUBLIC_SUPPORT -> policies.publicSupport(); case SECURITY_RESEARCH -> policies.securityResearch(); }; } } The result keeps ALLOW, BLOCK, and INDETERMINATE as separate states. When the classifier is unavailable, the status is INDETERMINATE. The blocked field tells the caller whether the configured outage policy allowed or rejected the request.
This implementation blocks when score > threshold, matching Mistral’s reference helper. A score exactly equal to 0.50 is allowed. If your product must block on equality, change the comparison and add a boundary test.
Attach the gate to a Quarkus LangChain4j AI Service
BoundaryDesk supplies the answering model explicitly. Shieldstral and the assistant model therefore cannot compete for the same unqualified ChatModel injection point.
package com.ibm.developer.shieldstral.assistant; import java.util.function.Supplier; import jakarta.enterprise.context.ApplicationScoped; import com.ibm.developer.shieldstral.config.AssistantModelConfig; import dev.langchain4j.model.chat.ChatModel; import dev.langchain4j.model.mistralai.MistralAiChatModel; @ApplicationScoped public final class AssistantModelSupplier implements Supplier<ChatModel> { private final ChatModel model; AssistantModelSupplier(AssistantModelConfig config) { model = MistralAiChatModel.builder() .baseUrl(config.baseUrl()) .apiKey(config.apiKey()) .modelName(config.modelName()) .temperature(0.2) .maxTokens(300) .timeout(config.timeout()) .maxRetries(1) .logRequests(false) .logResponses(false) .build(); } @Override public ChatModel get() { return model; } } Quarkus discovers LangChain4j guardrails as CDI beans. The shared input implementation passes the untrusted user message to PolicyGate:
package com.ibm.developer.shieldstral.policy; import dev.langchain4j.guardrail.InputGuardrail; import dev.langchain4j.guardrail.InputGuardrailRequest; import dev.langchain4j.guardrail.InputGuardrailResult; abstract class PolicyInputGuardrail implements InputGuardrail { private final PolicyGate gate; private final PolicySurface surface; PolicyInputGuardrail(PolicyGate gate, PolicySurface surface) { this.gate = gate; this.surface = surface; } @Override public InputGuardrailResult validate(InputGuardrailRequest request) { SafetyAssessment assessment = gate.evaluate( surface, PolicyDirection.INPUT, request.userMessage().singleText()); return assessment.blocked() ? fatal(assessment.reason()) : success(); } } The output implementation reads the generated AiMessage:
package com.ibm.developer.shieldstral.policy; import dev.langchain4j.guardrail.OutputGuardrail; import dev.langchain4j.guardrail.OutputGuardrailRequest; import dev.langchain4j.guardrail.OutputGuardrailResult; abstract class PolicyOutputGuardrail implements OutputGuardrail { private final PolicyGate gate; private final PolicySurface surface; PolicyOutputGuardrail(PolicyGate gate, PolicySurface surface) { this.gate = gate; this.surface = surface; } @Override public OutputGuardrailResult validate(OutputGuardrailRequest request) { String responseText = request.responseFromLLM().aiMessage().text(); SafetyAssessment assessment = gate.evaluate( surface, PolicyDirection.OUTPUT, responseText); return assessment.blocked() ? fatal(assessment.reason()) : success(); } } Four small @Singleton classes bind th...
