The Tug of War Between Words: How One Word Can Change an AI's Prediction

A practical look at prompt sensitivity, next-token probabilities, negation, and vision-language models.
Change one word in a prompt and you may change the answer.

Compare:

"Describe a bright room."
"Describe a dark room."

The difference is only one word, but the likely descriptions diverge:
sunlight, open windows, and vivid colours in one direction; shadows, dim lighting, and muted colours in the other.

Now try a less obvious change:

"Describe a room with a window."
"Describe a room without a window."

For a person, the distinction seems straightforward. For an AI model, that small word changes the constraint the model must interpret and satisfy. The result can be surprisingly inconsistent, especially when a prompt contains several instructions or when the model must reason about an image.

Why can one word have so much influence?

The answer is not that every word carries a fixed positive or negative score. It is that a prompt establishes context. That context shapes the model's internal representations and changes the relative scores assigned to possible continuations.

This is the hidden tug-of-war inside a prompt.

1. Words do not work alone

A language model does not process a sentence in quite the same way a human reader does. Text is first divided into tokens. A token may represent a whole word, part of a word, punctuation, or another text unit, depending on the tokenizer.

The tokens are mapped to numerical vectors. As they pass through
Transformer layers, their representations are updated using contextual information. Self-attention allows each position to incorporate information from other positions; feed-forward layers and other operations further transform those representations.

That means a word's contribution depends on its context.

Consider:

"The dog is inside the house."
"The dog is outside the house."

The model does not simply retrieve a permanent score for inside or outside. Their representations participate in a sequence of
computations conditioned on the surrounding tokens. Those computations affect what the model is likely to generate next.

The foundational paper Attention Is All You Need introduced the Transformer architecture and its self-attention mechanism.

2. Where the tug-of-war happens: logits and probabilities

For an autoregressive language model, the process can be simplified:

The prompt is tokenized.
The model computes contextual representations of the input.
The final representation is projected into a logit for each token in the vocabulary.
A softmax function can convert the logits into a probability distribution.
A decoding strategy selects the next token. The model then repeats the process with the expanded context.

For candidate token i, softmax converts logits into probabilities:

P(token i | context) = exp(z_i) / Σ exp(z_j)

Where:

z_i is the logit (raw score) assigned to candidate token i.
The denominator sums the exponentiated logits across all candidate tokens.
Softmax converts these scores into a probability distribution that sums to 1.

A higher logit generally results in a higher probability relative to competing candidates.

A simplified example

Imagine a model predicting the next token after two different contexts.
The following logits are hypothetical, created only to illustrate the mechanism; they are not measurements from a real model.

Prompt A: "A bright ..."

sunny: logit 3.0
clear: logit 1.0
night: logit 0.0

Prompt B: "A dark ..."

sunny: logit 0.5
clear: logit 2.5
night: logit 1.0

After applying softmax, the approximate probabilities are:

Prompt A: "A bright ..."

sunny: 84.4%
clear: 11.4%
night: 4.2%

Prompt B: "A dark ..."

sunny: 9.9%
clear: 73.6%
night: 16.5%

The numbers are illustrative, but the principle is real: changing the context changes the model's internal computation, which can change the logits and therefore the probability distribution.

This is a more accurate way to understand positive and negative
influence:

Context can make some continuations more compatible with the prompt, increasing their relative logits.
Other continuations can become less compatible, decreasing their relative logits.
Softmax normalizes the scores into probabilities.

There is no universal positive value attached to bright or negative value attached to dark. The effect emerges from learned representations, attention, nonlinear transformations, and the output projection. Attention weights alone do not explain the model's complete behavior.

3. The smallest word can carry a major constraint: negation

Negation is one of the clearest ways to expose the gap between fluent language generation and reliable instruction following.

Compare:

"Find an image containing a dog."
"Find an image containing no dog."
"Find an image containing a dog, but not on grass."

The words no and not are short, but they change the logical
constraints of the task. In the third example, the model must preserve
one condition (the dog is present) while rejecting another (the dog is on grass).

Recognizing the token not is not enough. The model must correctly determine what the negation applies to and maintain that relationship while interpreting the rest of the prompt.

This is an active research problem. In Vision-Language Models Do Not Understand Negation,
Alhamoud and colleagues introduce NegBench, a benchmark with 79,000
examples across 18 task variations. They report substantial difficulties with negation in the evaluated vision-language models. Their experiments
also investigate whether targeted synthetic training data can improve performance.

The practical lesson: adding not to a prompt does not guarantee that a model will reliably enforce the intended exclusion.

4. When language meets vision, the competition gets harder

A text-only LLM primarily conditions on linguistic context. A
vision-language model (VLM) must also use visual information.

The exact architecture varies. For example,
LLaVA connects a vision encoder to a language model through a learned projection, allowing visual features to be used alongside textual instructions. Other VLMs use different encoders, fusion mechanisms, tokenization schemes, and processing
pipelines.

Imagine providing a photograph of a dog standing on grass and asking:

"Describe the image."
"Describe the image, but do not mention the dog."
"Describe the dog without mentioning the grass."

The image provides visual evidence. The prompt specifies which aspects of that evidence matter to the requested response. The word not does not remove the dog or grass from the image representation; it changes the linguistic constraint the model is being asked to follow.

Whether the model obeys that constraint depends on how its architecture and training have learned to connect language with visual content.

Research on multimodal prompt sensitivity reinforces the point.
Promptception: How Sensitive Are Large Multimodal Models to
Prompts? studies how prompt variations affect multimodal question answering. The authors report accuracy deviations of up to 15 percentage points for certain prompt variations and model/task combinations. This is not evidence that every wording change has the same effect on every VLM; it shows that phrasing can materially affect performance under some conditions.

5. Try the experiment: change one word at a time

You do not need access to a model's internal logits to observe prompt sensitivity. Start with a fixed task and vary one phrase at a time.

Experiment A: a text model

Keep the input document fixed and change the instruction:

Original: "Summarize this incident report in three bullet points."
More concise: "Summarize this incident report in three concise bullet points."
Add a requirement: "Summarize this incident report in three bullet points, including the root cause."
Add an exclusion: "Summarize this incident report in three bullet points, excluding personal details."

Experiment B: a vision-language model

Keep the image fixed and vary only the instruction:

"Describe the image."
"Describe the image without mentioning people."
"Describe the image, focusing only on the background."

Compare the outputs across several examples, not just one. Record
whether the model followed each constraint, omitted required
information, or introduced details that were not supported by the input.

For a more rigorous experiment, keep the model version, decoding
settings, input data, and evaluation criteria fixed. If you have access to token log-probabilities, compare them too---but remember that a change in one next-token probability does not, by itself, explain the entire response.

6. What this means for prompt design

The underlying mechanics suggest a few practical habits:

Avoid contradictory instructions. "Be extremely brief" and "explain every detail" compete. State which requirement takes priority.
Make exclusions specific. Replace "avoid irrelevant details" with a clear description of what to omit.
Separate constraints. Use distinct instructions for the task, output format, and exclusions.
Test wording instead of assuming equivalence. Two prompts that mean nearly the same thing to a person may not produce equivalent model behavior.
Evaluate with representative inputs. For VLMs, test the actual kinds of images the system will encounter. A text-only test cannot establish visual grounding.

Conclusion: a prompt reshapes the probability landscape

A prompt is more than a list of instructions. It supplies context that shapes a model's internal representations and changes the relative likelihood of possible continuations.

A single word can matter because it changes the relationships the model computes; not because the word carries a universal positive or negative score.

When language meets vision, the challenge becomes even more interesting: the model must reconcile what it sees with what the prompt asks it to describe, identify, or ignore.

Words do not simply tell AI what to do. They reshape the competition between possible interpretations and possible predictions.