Logistic Regression

When you start your journey in Machine Learning, the first algorithm you usually learn is Linear Regression. Linear regression draws a straight line using the equation:

$y = m x + c$

This straight line is amazing for predicting continuous numbers, like the price of a house or a student's score. But what happens if you want to predict a category? For example, predicting whether a student is Placed (1) or Not Placed (0) based on their exam scores?

If you try to fit a straight line to binary data (0 and 1), the line keeps going forever. It will eventually predict impossible values like 1.5 or -0.4. Because a straight line fails to categorize discrete choices, Logistic Regression was introduced.

Historical Note: The mathematical foundation of logistic regression was popularized by statistician David Cox in 1958 to analyze binary datasets, adapting the "logistic curve" first formulated by Pierre François Verhulst in 1838 to model population growth.

Instead of drawing a straight line, Logistic Regression squishes the output into an S-shaped curve that strictly stays between 0 and 1. This allows the algorithm to output a clean probability (like an 85% chance of being placed).

Steps of Logistic Regression (The Training Loop)

To see how the computer learns, let's track a tiny dataset with two inputs (CGPA and IQ) to predict a binary output (Placed: 1, Not Placed: 0).

Student	CGPA ( $X_{1}$ )	IQ ( $X_{2}$ )	Actual Placed ( $Y$ )
Student 1	8.0	110	1
Student 2	5.0	90	0

Step 1: Initialize Weights and Biases

If we have $n$ features, we need $n$ weights plus 1 extra intercept term (called the bias or $β_{0}$ ). For our 2 features, we need $2 + 1 = 3$ weights. The computer starts by setting them all to zero:

import numpy as np # n = number of features (CGPA, IQ) n = 2 weights = np.zeros(n) # Array: [0.0, 0.0] bias = 0.0 # Scalar: 0.0

Step 2: Make Predictions & Calculate the Loss

The model calculates a raw linear score ( $z$ ) for each student using its current weights:

z = bias + (w_{1} \times CGPA) + (w_{2} \times IQ)

Since all our weights are currently 0, Student 1 gets a score of $z = 0$ . We pass this $z$ into the Sigmoid function to turn it into a probability ( $p$ ):

p = \frac{1}{1 + e ^{- 0}} = 0.5 (50%)

Now, we check how bad our guess was using Binary Cross-Entropy (BCE) Loss. The formula for a single row is:

Loss = - [Y lo g (p) + (1 - Y) lo g (1 - p)]

For Student 1 (  $Y = 1, p = 0.5$  ):  $Loss = - [1 \cdot lo g (0.5) + 0] = - (- 0.693) = 0.693$  For Student 2 (  $Y = 0, p = 0.5$  ):  $Loss = - [0 + 1 \cdot lo g (1 - 0.5)] = - (- 0.693) = 0.693$  Average Total Loss:  $\frac{0.693 + 0.693}{2} = 0.693$

Step 3: Update Weights Using Gradient Descent

To lower this error, we compute the derivative (gradient) of the loss with respect to each parameter. The math simplifies beautifully to:

Gradient = Predicted Probability - Actual Outcome

Let's look at Student 1 (Predicted $p = 0.5$ , Actual $Y = 1$ ):

$Error = 0.5 - 1 = - 0.5$

We calculate the weight update by multiplying this error by the student's feature value and our Learning Rate ( $l r = 0.1$ ). Since we want to decrease loss, we subtract the gradient:

New Parameter = Old Parameter - (l r \times Gradient)

Step 4: Run the Training Loop

The computer repeats Steps 2 and 3 thousands of times. Over time, the loss drops from 0.693 down toward 0. Let's assume after training, our optimized parameters become:

  $bias = - 15$   $w_{1} (CGPA) = 1.5$   $w_{2} (IQ) = 0.05$

Let's verify the calculation for Student 1 (CGPA=8.0, IQ=110):

Find  $z$  :  $z = - 15 + (1.5 \times 8.0) + (0.05 \times 110) = - 15 + 12 + 5.5 = + 2.5$  Find  $p$  (Sigmoid):  $p = \frac{1}{1 + e ^{- 2.5}} = \frac{1}{1 + 0.082} = \frac{1}{1.082} \approx 0.924 (92.4$  Threshold Check: Since  $0.924 \geq 0.5$  , the model confidently predicts 1 (Placed). The math works!

Important Concepts Explained Simply

The Sigmoid Function

The Sigmoid function is the mathematical wizard that squishes any real number (from negative infinity to positive infinity) into a neat window between 0 and 1.

Mathematical Formula:

σ (z) = \frac{1}{1 + e ^{- z}}

How to read the curve graph:

If  $z$  is a large positive number (like +5),  $e^{- 5}$  becomes almost 0, so  $\frac{1}{1 + 0} = 1$  . If  $z$  is exactly 0,  $e^{0} = 1$  , so  $\frac{1}{1 + 1} = 0.5$  . If  $z$  is a large negative number (like -5),  $e^{- (- 5)} = e^{5}$  , making the denominator massive, so the output drops close to 0.

The Logit Function (Log of Odds)

You might hear people talk about the "Logit" function. It is simply the exact mathematical inverse of the Sigmoid function. Instead of turning a score into a probability, it turns a probability back into a straight-line score.

Odds = \frac{p}{1 - p}

Logit (p) = lo g (\frac{p}{1 - p}) = β_{0} + β_{1} X_{1} + \dots

Binary Cross-Entropy (BCE) vs. Categorical Cross-Entropy (CCE)

These are formulas used to measure error.

BCE (Binary): Used when you only have two choices (0 or 1). It only checks the probability of the single correct class. CCE (Categorical): Used when you have multiple choices (e.g., predicting if an image is a Cat, Dog, or Bird). It measures the error across a distribution of multiple category classes.

Softmax Function

While Sigmoid handles binary items, Softmax is the multi-class version of Sigmoid. If you are choosing between Cat, Dog, and Bird, Softmax handles all output channels simultaneously and ensures that all their independent probabilities add up to exactly 1.0 (100%).

Softmax (z_{i}) = \frac{e ^{z_{i}}}{\sum e ^{z_{j}}}

Complete Working Code From Scratch

Here is how you write this entire mathematical loop in pure Python using numpy:

import numpy as np class CleanLogisticRegression: def __init__(self, lr=0.1, num_iterations=1000): self.lr = lr self.num_iterations = num_iterations import numpy as np self.weights = None self.bias = None def _sigmoid(self, z): return 1 / (1 + np.exp(-z)) def fit(self, XSource：https://dev.to/jubito27/logistic-regression-4oijMy favorites

PreviousHow to Build an Airdrop Monitor with AI — 2026-10-10 #6NextSpot vs on-demand GPUs: what breaks, and how do you make spot safe?

`Related posts`


The Autonomous Edge — Issue #13: Cheaper Agents, Harder Governance (Week of November 2, 2026)
Cross-posted from The Autonomous Edge, a free weekly briefing on AI agents in the enterprise. Get it by email every Mond
0dev.to AI
Plano Físico vs Plano Funcional en AWS Networking: cómo los arquitectos senior resuelven lo que otros no ven
Por Oscar A. Gaviria González — Cloud Architect Lead, NTT DATA | AWS Golden Jacket Introducción El túnel VPN está UP. El
0dev.to AWS
We built Origami Tutor: free paper-craft lessons you can turn and replay in 3D
Paper diagrams are hard to follow. One arrow means "fold behind", another means "reverse fold", and after step 12 you're
0dev.to WebDev
From two authorized photos to a short AI rap video (consent-first workflow)
I built a small web tool for a narrow job: take 1–2 photos the user owns rights to , generate original AI audio (not scr
0dev.to WebDev
Check a File Hash in the Browser, Without Uploading the File
Full disclosure: I work on OnePassApps. OnePassApps is one membership for a set of web apps. The File Studio runs inside
0dev.to WebDev
Your AI keys were leaked at a platform. A local guard you can paste catches the abuse before the invoice.
If an AI API key you store on a platform was just leaked, rotation is the headline. It is not the whole fix. Rotation st
0dev.to Security