You've probably heard about ChatGPT, Gemini, Claude, Copilot, and a dozen other AI tools.
You type something into a chat window, and within seconds, you get an answer. It can write emails, explain complicated topics, generate code, summarize documents, and even help you plan your next vacation.
Pretty impressive, right?
But have you ever wondered what actually happens behind that chat window?
How does a computer understand the sentence you just typed? How does it know which words are related? How does it generate a meaningful response instead of randomly throwing words at you?
Well, the technology behind much of today's Generative AI revolution is something called Transformers.
Let's start from scratch and understand what they are, how they work, and why they changed the world of Artificial Intelligence.
Don't worry. We won't be getting into complicated mathematics or making you regret opening this article!
1. First Things First: What Is NLP?
Before we understand Transformers, let's understand Natural Language Processing, commonly known as NLP.
NLP is a branch of Artificial Intelligence that helps computers work with human language.
Think about it. We communicate through emails, WhatsApp messages, documents, social media posts, reviews, and conversations.
But computers don't naturally understand these things the way humans do.
They need a way to process language and identify patterns.
That's where NLP comes in.
There are three broad types of tasks that NLP systems can perform.
Classification
Classification means assigning something to a category.
For example, imagine your inbox receives an email:
"Congratulations! You've won a free iPhone. Click here to claim your prize!"
Most of us would immediately suspect that it's spam.
An NLP model can analyse the email and classify it as either Spam or Not Spam.
Another example is sentiment analysis.
If someone writes:
"The food was amazing, and the service was excellent!"
The model can classify this as Positive.
If someone writes:
"The food was terrible. I wish I had stayed home!"
It can classify it as Negative.
Multiple Labels and Classification
Now, imagine you have the following sentence:
"Rahul joined Microsoft in London in 2024."
An NLP model can identify:
Rahul – Person
Microsoft – Organization
London – Location
2024 – Date
This is called Named Entity Recognition (NER).
It can also identify grammatical categories such as nouns, verbs, adjectives, and prepositions. This is known as Part-of-Speech Tagging.
Generation
And this is where things get interesting.
Generation means creating new content.
For example, you ask ChatGPT:
"Write a funny birthday message for my manager."
And it generates an entirely new message.
Other examples include translation, summarization, question answering, and code generation.
So, while traditional NLP systems often focused on identifying or classifying information, Generative AI can actually create new content.
2. How Do We Know Whether an AI Model Is Good?
Imagine we've built an AI model that can identify spam emails.
But how do we know whether it's actually working?
We need to evaluate its performance.
Let's understand some commonly used evaluation metrics.
Accuracy
This is the simplest one.
Suppose our model analyses 100 emails and correctly classifies 90 of them.
Its accuracy is 90%.
Simple!
But accuracy can sometimes be misleading.
Imagine a hospital has a dataset of 1,000 people, and only one person has a particular disease.
A model predicts that everyone is healthy.
It gets 999 predictions correct!
That's 99.9% accuracy.
But it missed the one person who actually needed attention.
Not exactly a great result!
Precision and Recall
Precision answers:
When the model predicts something is positive, how often is it correct?
Recall answers:
Out of all the actual positive cases, how many did the model successfully identify?
For example, in spam detection:
Precision tells us how many emails marked as spam were actually spam.
Recall tells us how many of all the actual spam emails the model successfully caught.
F1-Score
F1-Score combines Precision and Recall into a single metric.
It is particularly useful when we want to balance the two.
BLEU and ROUGE
These metrics are commonly used in language-related tasks.
BLEU stands for Bilingual Evaluation Understudy. It is used to evaluate machine-generated translations by comparing them with reference translations.
ROUGE is commonly used to evaluate summaries by measuring overlap between generated and reference text.
Perplexity
Now, this one sounds complicated, but the intuition is quite simple.
Imagine I say:
"The cat sat on the..."
What word comes to your mind?
Probably mat, sofa, or floor.
Now imagine I say:
"The cat sat on the refrigerator-banana-spaceship."
You'd probably wonder what happened to the sentence!
Perplexity measures how surprised a language model is by a sequence of words.
Generally, lower perplexity means the model finds the sequence more predictable.
These metrics help us understand how well different NLP models perform.
3. What Are Tokens? Understanding Tokenization
Now, here's an important question.
When you type a sentence into ChatGPT, does the model process the entire sentence exactly as you see it?
Not quite.
Before processing the text, the model breaks it into smaller units called Tokens.
This process is called Tokenization.
For example:
"I love Artificial Intelligence."
Could be broken into:
"I" + "love" + "Artificial" + "Intelligence".
But tokenization doesn't always happen at complete word boundaries.
A word like "Unbelievable" might be broken into smaller pieces such as:
"Un" + "believ" + "able".
The exact split depends on the tokenizer.
There are three common approaches.
Word-Level Tokenization
Each complete word becomes a token.
For example:
"I love AI."
Becomes:
"I", "love", "AI".
The problem? What happens when the model encounters a word it has never seen before?
This is called an Out-Of-Vocabulary (OOV) problem.
Character-Level Tokenization
Here, every character becomes a token.
For example:
CAT becomes C + A + T.
This handles unfamiliar words better, but sequences become much longer.
Imagine processing an entire paragraph character by character. That's a lot of work!
Subword-Level Tokenization
This is widely used in modern language models.
Instead of splitting every word into individual characters or keeping every word whole, we break words into reusable smaller pieces.
Popular approaches include Byte Pair Encoding (BPE) and WordPiece.
This gives us a practical balance between vocabulary size and sequence length.
Once the text is broken into tokens, these tokens are mapped to numerical IDs.
The model then processes their numerical representations.
And that brings us to our next question.
How do we represent the meaning of words using numbers?
4. Word2Vec and Embeddings: Giving Words a Numerical Representation
Computers work with numbers.
So, how do we represent words such as King, Queen, Dog, or Cat?
One basic approach is One-Hot Encoding.
Imagine we have five words:
Cat, Dog, King, Queen, Apple.
We assign each word a unique vector:
Cat = [1, 0, 0, 0, 0]
Dog = [0, 1, 0, 0, 0]
King = [0, 0, 1, 0, 0]
Queen = [0, 0, 0, 1, 0]
Apple = [0, 0, 0, 0, 1]
Every word gets a unique numerical identity.
But there's a problem.
The representation doesn't tell us that King and Queen are related.
It's similar to employee ID numbers.
Employee 101 and Employee 102 have different IDs, but those IDs don't tell us whether they work in the same department.
Enter Word2Vec
Word2Vec was an important development in learning meaningful word representations.
Instead of manually defining the relationships between words, the model learns them by analysing how words appear in sentences.
It has two important approaches.
CBOW – Continuous Bag of Words
CBOW uses surrounding words to predict a target word.
For example:
"The quick brown ___ jumps over the lazy dog."
The model uses the surrounding context to predict "fox".
Skip-Gram
Skip-Gram does the opposite.
It takes a target word and tries to predict its surrounding context.
For example, if the target word is "King", the model tries to predict words that commonly appear around it.
What Are Dense Embeddings?
An embedding is a numerical representation of a word.
Unlike One-Hot Encoding, dense embeddings contain values that can capture useful relationships between words.
For example:
King = [0.12, -0.03, 0.87, ...]
Queen = [0.11, -0.02, 0.85, ...]
These are illustrative values.
Words with similar meanings or contexts often have similar vector representations.
This leads to the famous example:
King - Man + Woman ≈ Queen
The model has learned relationships between words and represented some of those relationships mathematically.
One important thing to remember:
Word2Vec generally creates a relatively fixed representation for a word.
Modern Transformers can create contextual representations, where the representation of a word changes depending on the sentence.
For example, the word "Bank" in a financial sentence and "Bank" in a river-related sentence can have different contextual representations.
And that's a major improvement.
5. RNNs and LSTMs: How Did Machines Remember Previous Words?
Let's say I give you these two sentences:
"Dog bites man."
"Man bites dog."
Same words. Completely different meaning.
So, understanding language isn't just about knowing individual words. The order matters too.
Researchers developed sequential models to handle this.
Recurrent Neural Networks (RNNs)
Imagine I'm telling you a story:
"Yesterday, I went to the supermarket..."
"...to buy some milk..."
"...but I came back with a television!"
You remember the earlier parts of the story while listening to the later parts.
An RNN tries to do something similar.
It processes a sentence one token at a time.
At each step, it receives the current input and information from the previous step.
This information is stored in something called a Hidden State, represented by h_t.
Think of it as the model's working memory.
The Problem with RNNs
RNNs struggled to retain information from much earlier in long sequences.
This is related to a problem called the Vanishing Gradient Problem.
During training, the learning signal can become extremely small as it travels backwards through many time steps.
As a result, the model may struggle to learn relationships between words that are far apart.
LSTMs – Long Short-Term Memory
To address this, researchers introduced LSTMs.
Think of an LSTM as an RNN with a more sophisticated memory system.
It uses three important gates:
Forget Gate: Decides what information should be removed.
Input Gate: Decides what new information should be added.
Output Gate: Decides what information should be passed forward.
LSTMs also maintain a Cell State, represented by c_t, which helps carry information across longer sequences.
LSTMs were a major improvement.
But they still had one limitation.
They processed information sequentially.
One step had to follow another.
This made parallel processing difficult.
Imagine having 100 employees available, but forcing them to complete a task one after another even though several parts could be done simultaneously.
That's not the best use of resources!
And this brings us to one of the biggest breakthroughs in modern AI.
6. Attention: The Real Game Changer
Let's take a simple example.
Imagine you're reading a long email from your manager.
There are ten paragraphs, but you're trying to find one thing:
"What is the project deadline?"
You don't give equal importance to every word in that email.
You focus on the information related to the deadline.
That's the basic intuition behind Attention.
What Is Self-Attention?
Consider this sentence:
"The animal didn't cross the road because it was tired."
What does "it" refer to?
The animal.
To understand this, we need to connect the word "it" with another word in the sentence.
Self-Attention allows each token to look at other tokens in the same sequence and calculate how relevant they are.
This helps the model understand relationships between different parts of the sentence.
Queries, Keys and Values – Q, K and V
Now, let's understand three important terms.
Imagine you're visiting a library.
You want a book about Artificial Intelligence.
You ask the librarian:
"Do you have a book about AI?"
That question is your Query.
The librarian checks the titles and descriptions of available books. These help identify which books match your requirement.
Think of these as Keys.
Once the librarian identifies the relevant books, you take them and read their contents.
That's the Value.
So:
Query: What am I looking for?
Key: What information is available, and how well does it match?
Value: What actual information should I collect?
In a Transformer, every token is converted into three learned numerical representations: Query, Key and Value.
The model compares Queries with Keys to calculate relevance scores.
These scores are converted into attention weights, which determine how much information to take from each Value.
Understanding the Attention Formula
Here is the scaled dot-product attention formula:
Attention(Q, K, V) = softmax((Q × Kᵀ) / √dₖ) × V
Let's break it down.
Q × Kᵀ: Compares Queries with Keys to calculate relevance scores.
√dₖ: Scales the scores to keep them manageable.
Softmax: Converts the scores into weights that add up to 1.
× V: Combines the Value vectors using those weights.
Imagine three words receive attention weights of 70%, 20% and 10%.
The model uses these weights to give more importance to some information than others.
Why Is Attention So Important?
RNNs process tokens sequentially.
Self-attention allows tokens to establish direct computational relationships with other tokens in the sequence.
This makes it easier to learn long-range relationships and enables parallel computation during training.
And this is one of the major reasons Transformers became so powerful.
7. The Original Transformer Architecture
In 2017, researchers published a famous paper titled:
Attention Is All You Need.
This paper introduced the original Transformer architecture.
The architecture had two main components:
Encoder
Decoder
Encoder
Think of the Encoder as the part that reads and processes the input.
For example:
"The boy kicked the ball."
The encoder processes the sentence and creates contextual representations.
It doesn't just look at individual words. It also learns how they relate to each other.
Decoder
The Decoder generates the output.
Imagine translating:
"I love artificial intelligence."
Into French.
The encoder processes the original sentence, and the decoder generates the translated sentence.
The decoder produces the output one token at a time.
Causal Masking
Imagine writing an exam.
You answer one question at a time. You cannot look at the answers to future questions.
Causal masking works with a similar idea.
When generating text, the decoder must not look at future tokens.
It can use previous tokens and the current position, but not the tokens it hasn't generated yet.
Multi-Head Attention
Imagine a team reviewing the same document.
One person focuses on grammar.
Another focuses on dates.
Another looks at technical details.
Another focuses on the overall meaning.
Everyone is examining the same document, but from different perspectives.
Multi-Head Attention works with a similar concept.
Multiple attention heads process information in parallel using different learned Query, Key and Value projections.
Their outputs are combined to capture different types of relationships.
8. Position Embeddings: Why Does Word Order Matter?
Here's a small problem.
If attention can process multiple tokens in parallel, how does the model know their original order?
Let's compare:
"Dog bites man."
"Man bites dog."
Same words, different meanings.
So, the model needs positional information.
Learned Positional Embeddings
One approach is to learn a separate numerical representation for each position.
Position 1 gets one representation.
Position 2 gets another.
Position 3 gets another.
And so on.
Sinusoidal Positional Encoding
Another approach uses sine and cosine functions to create positional patterns.
You don't need to understand the mathematics right now.
Just remember that these patterns help the model identify where tokens appear in a sequence.
Rotary Position Embeddings (RoPE)
RoPE is a more modern approach.
Instead of simply adding positional information, it rotates Query and Key vectors according to their positions.
This allows attention calculations to incorporate positional relationships.
The important takeaway:
Attention helps understand relationships between tokens, while positional information helps understand their order.
9. Layer Normalization and Other Training Techniques
Now, let's talk about training stability.
Imagine learning to drive a car.
Initially, you accelerate too much, brake too late, and turn too sharply.
You gradually adjust your actions.
Neural networks also need stable training.
Layer Normalization
Layer Normalization helps keep the values flowing through a neural network on a manageable scale.
This can make training more stable.
Pre-Norm and Post-Norm
These describe where normalization is applied in relation to a sublayer.
Pre-Norm applies normalization before the sublayer.
Post-Norm applies it after the sublayer.
RMSNorm
RMSNorm stands for Root Mean Square Normalization.
It is a streamlined normalization technique that scales values using their root mean square, without subtracting the mean.
Label Smoothing
Imagine you're teaching someone to identify a dog.
The correct answer is Dog.
Label Smoothing slightly softens the target probabilities during training instead of assigning all the probability to just one correct answer.
It can help discourage excessive confidence.
These techniques help improve the training process, even though they aren't the main reason Transformers became successful.
10. Making Attention Faster and More Efficient
Transformers are powerful, but they can be computationally expensive.
In full self-attention, every token can interact with every other token.
The computational cost grows approximately with the square of the sequence length.
In other words, longer inputs can become expensive to process.
Researchers introduced several techniques to improve efficiency.
Sliding Window Attention
Imagine you're in a classroom.
Instead of speaking to every student, you interact mainly with students sitting nearby.
Sliding Window Attention follows a similar idea.
Each token attends mainly to a local group of tokens.
This reduces the number of attention connections.
Multi-Query Attention (MQA)
In standard Multi-Head Attention, each head has its own Key and Value projections.
MQA allows multiple attention heads to share a single Key and Value representation.
This reduces the memory needed to store them.
Grouped-Query Attention (GQA)
GQA takes a middle approach.
Instead of sharing one Key and Value representation across all heads, it groups the attention heads.
Each group shares its own Key and Value representations.
KV Cache
When an LLM generates a response, it produces tokens one after another.
The KV Cache stores previously calculated Key and Value representations so they don't need to be recalculated unnecessarily at every generation step.
This helps improve generation efficiency.
11. Different Types of Transformer Architectures
Not every Transformer is built in exactly the same way.
There are three major architectural categories.
Encoder-Decoder
The encoder processes the input, and the decoder generates the output.
Think of translation.
You read a sentence in English and produce the corresponding sentence in French.
Examples include T5 and ByT5.
Encoder-Only
Encoder-only models are designed primarily for understanding text.
BERT and RoBERTa are popular examples.
They can use context from both the left and right sides of a token.
For example:
"The bank is beside the river."
"I deposited money in the bank."
The surrounding words help the model understand the meaning of "bank".
These models are useful for classification and text understanding.
Decoder-Only
This is where the GPT family comes in.
Decoder-only models generate text one token at a time.
You provide:
"Once upon a time..."
The model predicts the next token, then uses the generated text to predict the following token.
This is called Autoregressive Generation.
It is particularly useful for conversational AI and open-ended text generation.
12. BERT: Understanding Text from Both Directions
Let's take a closer look at BERT.
BERT stands for Bidirectional Encoder Representations from Transformers.
Imagine the sentence:
"The man went to the bank to deposit money."
The model can use words from both sides of "bank" to understand that we're talking about a financial institution.
Now compare:
"The man sat near the bank of the river."
The context changes the meaning.
BERT uses information from both directions to create contextual representations.
Special Tokens
BERT uses special tokens such as:
[CLS]
This is placed at the beginning of the input and can be used for classification tasks.
[SEP]
This separates sentences or text segments.
Masked Language Modeling
One of BERT's important pre-training tasks is Masked Language Modeling (MLM).
For example:
"The cat is sitting on the [MASK]."
The model tries to predict the missing word using the surrounding context.
Next Sentence Prediction
The original BERT training approach also included Next Sentence Prediction (NSP).
The model was trained to identify whether one sentence naturally followed another in the original text.
For example:
Sentence A: "I went to the supermarket."
Sentence B: "I bought some milk."
These sentences could naturally follow each other.
BERT was trained to distinguish actual consecutive sentence pairs from unrelated pairs.
BERT is particularly useful for text classification, entity recognition and other language understanding tasks.
13. Making Models Smaller: DistilBERT and RoBERTa
Now, building a powerful model is one thing.
Running it efficiently is another.
Imagine you've developed a large application that works perfectly on your powerful development machine.
But when you deploy it on a smaller server, it consumes too much memory.
What do you do?
You optimize it.
The same thing happens with language models.
Knowledge Distillation
Imagine an experienced teacher and a student.
The teacher has years of knowledge.
The student learns by observing the teacher's explanations and examples.
Knowledge Distillation follows a similar approach.
We have:
A large Teacher Model.
A smaller Student Model.
The student learns from the teacher's outputs, including the probability distributions it produces.
The objective is to transfer useful knowledge into a smaller model.
DistilBERT
DistilBERT is a smaller version of BERT.
It was designed to retain much of BERT's capabilities while reducing model size and improving efficiency.
The trade-off is that a smaller model may lose some performance, depending on the task.
RoBERTa
RoBERTa is an improved training approach based on BERT.
Some of its important changes include:
Removing Next Sentence Prediction.
Using dynamic masking.
Training on more data.
Dynamic masking means that different tokens can be masked across different training examples.
Think of it as practising vocabulary with different missing words each time instead of memorizing the same answer.
And that's an important lesson.
Sometimes, improving how we train an existing architecture can be just as important as developing a completely new architecture.
Bringing Everything Together
Let's quickly recap.
We started with NLP and understood how computers classify, identify and generate text.
We learned about evaluation metrics, tokenization and embeddings.
We explored RNNs and LSTMs, which helped computers process sequences and maintain information.
Then came Attention, which allowed models to calculate relationships between different tokens.
The Transformer architecture brought these ideas together using encoders, decoders, positional information and multi-head attention.
We then explored ways to make Transformers faster, different architectural variants, BERT, DistilBERT and RoBERTa.
If there's one thing I want you to remember from this article, it's this:
Transformers changed the way computers process language by allowing them to identify and use relationships between different parts of a sequence through attention.
The next time you ask ChatGPT to write an email, explain a technical concept, or summarize a 50-page document, remember that there's a lot happening behind that simple chat window.
Your text is broken into tokens.
Those tokens are converted into numerical representations.
The model processes their relationships using attention.
And a decoder-based language model generates its response one token at a time.
What looks like a simple conversation is actually the result of years of research, mathematics and engineering.
And we're still only getting started!
If you enjoyed this introduction, keep exploring. You don't need to understand every mathematical detail on day one. Start with the intuition, understand the building blocks, and gradually move towards the implementation.
Happy learning!
