Understanding AI: The Fundamentals Finally Make Sense When You See How They Connect

Artificial Intelligence can feel complicated when you encounter terms like tokens, embeddings, attention, transformers, fine-tuning, RAG, and AI agents.

But these concepts are not isolated technologies. They are building blocks that solve specific problems, one after another.

The easiest way to understand modern AI is to follow the journey of how a computer goes from processing ordinary text to understanding context, generating responses, retrieving information, and eventually taking action.

Let's break it down.

1. Tokens: Teaching Computers to Read Text

You want a computer to understand text. The problem is that computers fundamentally process numerical representations, not words the way humans do.

So, how do you make text readable to a machine?

You use tokens.

A tokenizer breaks text into smaller pieces called tokens. These can be whole words, parts of words, or even individual characters.

For example, "Unbelievable" might be split into:

• Un
• believ
• able

Each token is assigned a numerical ID.

Now the computer has a numerical representation of the text it needs to process.

But there is a problem. Those numbers are simply identifiers. They don't inherently communicate meaning.

The token for "dog" could be 4521, while "puppy" could be 9873. Nothing about those numbers tells the computer that the two words are related.

2. Embeddings: Giving Numbers Meaning

This is where embeddings come in.

Instead of representing each token with a single numerical ID, embeddings transform tokens into vectors: lists of numbers that capture aspects of their meaning and relationships.

Words with similar meanings tend to have similar vector representations.

For example:

• Dog and puppy are semantically related.
• King and queen have a meaningful relationship.
• Dog and Paris are generally unrelated.

These relationships can be represented geometrically in a high-dimensional space.

Now the computer has more than numerical labels. It has numerical representations that carry semantic information.

But another problem emerges.

A word can have different meanings depending on the sentence in which it appears.

Consider the word "bank":

• I deposited money at the bank.
• We sat beside the river bank.

The word is identical, but the meaning is different.

Embeddings alone do not fully resolve this contextual ambiguity.

3. Attention: Understanding Context

This is where attention mechanisms become important.

Attention allows a model to examine relationships between tokens and determine which other tokens are relevant when interpreting a particular token.

Consider:

"The man deposited money at the bank."

The model can use the surrounding words, particularly "deposited" and "money", to understand the intended meaning of "bank".

In the sentence:

"The fisherman sat beside the bank."

The surrounding context points towards a riverbank.

The model is no longer treating every word as an isolated piece of information. It can consider how words relate to one another.

This mechanism is central to the Transformer architecture, which powers most modern large language models.

But having the architecture does not automatically mean the model knows anything useful.

Where does its knowledge come from?

4. Training: Learning Patterns from Data

This is where training comes in.

A language model is exposed to enormous amounts of text and learns through an objective that, in its simplest form, involves predicting the next token.

For example:

"The capital of France is..."

The model learns to assign probabilities to possible next tokens, with "Paris" being a highly likely continuation.

Whenever its predictions differ from the training target, the model's parameters are adjusted through an optimization process called gradient descent, using backpropagation to calculate how those parameters should change.

These parameters are commonly called weights.

After processing vast amounts of training data, the model learns patterns involving language, grammar, information, relationships, and problem-solving.

However, a model trained primarily to predict the next token is not automatically a helpful conversational assistant.

Ask it a question, and it might simply continue generating text rather than answering in the way you expect.

How do we change that?

5. Fine-Tuning: Turning a Language Model into an Assistant

This is where fine-tuning comes in.

The model undergoes additional training using carefully prepared examples that demonstrate desirable behaviours.

For example:

• User asks a question.
• Assistant provides a useful answer.
• User requests an explanation.
• Assistant explains step by step.
• User provides an instruction.
• Assistant follows the instruction.

Instruction fine-tuning helps the model learn how to respond to requests rather than simply continue arbitrary text.

Other alignment techniques, including preference optimization and reinforcement learning, can further improve how the model responds.

The result is a model that behaves much more like an assistant.

But there is still another consideration.

How predictable or creative should its responses be?

6. Temperature: Controlling Randomness in Responses

This is where temperature comes in.

Temperature influences how the model samples from the probability distribution of possible next tokens.

A lower temperature generally makes the model more predictable, favouring higher-probability choices.

A higher temperature makes less likely alternatives more competitive, increasing variation and randomness.

Think about the practical applications:

• Code generation: Lower temperature can help produce more consistent outputs.
• Factual explanations: Lower temperature can improve predictability, although it does not guarantee accuracy.
• Brainstorming: Higher temperature can encourage more varied ideas.
• Creative writing: Higher temperature can introduce more stylistic diversity.

Temperature does not make a model more intelligent. It changes how it selects its responses.

Now imagine giving the model a 500-page company policy document and asking it a question about page 400.

Or imagine having a conversation that runs for hours.

Why might the model fail to remember information from earlier in the conversation?

7. Context Window: The Model's Working Space

This is where the context window comes in.

A context window defines how much information a model can process at once.

It can include:

• Your current message.
• Previous conversation messages.
• System instructions.
• Retrieved documents.
• Tool outputs.

If the information exceeds the available context, some content may need to be removed, truncated, or summarized.

This is why even powerful models can struggle with very long conversations or large documents.

The context window is not the same as permanent memory. It is the information available to the model during a particular processing operation.

Now consider another problem.

You ask an AI assistant about your company's internal HR policies, but those documents were never included in its training data.

How can it answer accurately?

Sometimes it cannot. Worse, it may generate an answer that sounds convincing but is completely incorrect.

8. Hallucinations: When AI Sounds Confident but Is Wrong

This behaviour is commonly called hallucination.

A language model generates responses based on learned patterns and the information available in its context.

It does not automatically verify every statement against an authoritative source.

For example, ask it about an internal company policy it has never seen. It might generate a plausible answer based on general knowledge, even though that answer has nothing to do with your actual company policy.

It is not necessarily deliberately misleading you.

It is generating a likely response without sufficient grounding.

So, how do we connect language models to information they were not trained on?

9. RAG: Giving AI Access to External Knowledge

This is where Retrieval-Augmented Generation (RAG) comes in.

Instead of expecting the model to remember everything, you build a system that retrieves relevant information before the model generates its response.

A simplified RAG workflow looks like this:

1. Documents are collected from sources such as SharePoint, databases, PDFs, and internal systems.
2. Documents are divided into smaller chunks.
3. The chunks are converted into embeddings.
4. The embeddings are stored in a vector database or another retrieval system.
5. When a user asks a question, the system searches for relevant content.
6. The retrieved content is added to the model's context.
7. The model generates an answer using that information.

Think of it as an open-book examination rather than a memory test.

For example, an employee asks:

"What is our company's maternity leave policy?"

Instead of guessing, the system retrieves the relevant section from the actual HR policy and provides an answer based on it.

RAG improves grounding, but it does not guarantee correctness. Retrieval quality, document freshness, permissions, and answer verification still matter.

But there is another limitation.

The model can retrieve information and generate responses, but can it actually do something?

Can it query a database, check a server, send an email, or create a support ticket?

Not without additional capabilities.

10. Tools and Function Calling: Giving AI the Ability to Act

This is where tool calling comes in.

You give the model access to predefined tools that allow it to interact with external systems.

For example:

• A database query tool.
• An email-sending tool.
• A ticket creation API.
• A weather API.
• A payment status verification API.
• A SharePoint document search tool.

The model can determine when a tool is needed, generate the required arguments, and request its execution.

The application executes the tool, returns the result, and the model uses that result to decide what to do next.

For example:

User: "Check whether payment transaction TXN123 was successful."

The AI could:

1. Identify that it needs payment information.
2. Call the payment verification API.
3. Receive the transaction status.
4. Interpret the response.
5. Tell the user whether the payment succeeded.

Notice the distinction: the model does not magically access your payment system. Your application provides the integration and controls what the model is allowed to do.

And this brings us to one of the most talked-about concepts in AI today.

11. AI Agents: Bringing Everything Together

An AI agent combines a language model with tools, instructions, context, and a process for taking actions towards a goal.

Unlike a traditional chatbot that primarily responds to messages, an agent can work through a sequence of steps.

Imagine an AI agent responsible for handling customer support requests.

A customer reports a failed payment.

The agent could:

1. Understand the customer's request.
2. Retrieve the customer's relevant information.
3. Check the payment transaction through an API.
4. Investigate the transaction status.
5. Determine whether escalation is necessary.
6. Create a support ticket.
7. Notify the customer.
8. Record the outcome.

It can use the results of one action to determine its next action.

That is the difference between simply generating a response and completing a task.

Of course, agents still need guardrails, permissions, monitoring, and human approval for sensitive operations. More autonomy does not automatically mean better outcomes.

The Bigger Picture: How Everything Connects

Look at the progression:

The important thing to understand is that these concepts are not competing technologies. They solve different problems and often work together.

You do not need to understand every mathematical detail before building useful AI solutions.

But you should understand what each component does, why it exists, and when to use it.

Because building AI applications is not simply about connecting an API to ChatGPT or prompting a model to generate something.

It is about understanding the entire system around the model: data, retrieval, context, integrations, permissions, orchestration, and execution.

The real opportunity in AI engineering is not just building systems that can talk. It is building systems that can understand business problems, access the right information, make informed decisions, and safely get work done.