Most "ML basics" articles teach you to call
model.fit(X, y)and pretend that's the job. The actual job — where 80% of your time goes and 100% of the mistakes happen — is everything aroundfit: figuring out what your dataset really is, fixing what's broken, splitting it without leaking, picking the right metric, and noticing when your "great" model is just memorising garbage. This is the missing manual. (Full version on the original — link at the end.)
TL;DR
- Data > model. A mediocre model on clean, leak-free, well-split data beats a SOTA model on bad data every time.
- The pipeline: raw data → clean → split → preprocess (fit on train, transform everywhere) → train → evaluate → iterate. Every step has a "wrong way" that silently inflates your score.
- Pick the right metric. Accuracy is wrong for imbalanced classification. RMSE is wrong for skewed regression. ROC-AUC is wrong for ranking.
- Use
sklearn.pipeline.Pipeline+ColumnTransformer. They prevent leakage by design. Manual preprocessing is how production ML breaks. - Validate the way you'll deploy. Time-series prod = time-series CV. Tenant-scoped prod = group-aware CV. Random CV on time-series gives a fantasy score.
1. What ML actually is
ML is a function-approximation game: you have inputs X, outputs y, and a dataset of (X, y) pairs you trust to be representative. You search for f such that f(X) ≈ y and generalises to new data you haven't seen. That "generalises" is the whole reason ML is hard — memorising the training set is easy; predicting next month's data is not. Three things must be true: the signal exists (y depends on X in a learnable way), the data is representative (train looks like prod), and you can measure success honestly.
2. Recognise your dataset type
| If your data is... | Treat as | Reach for |
|---|---|---|
| CSV / SQL table | Tabular | sklearn, LightGBM/XGBoost/CatBoost (trees beat DL here) |
| Natural-language strings | Text | transformers, sentence-transformers, TF-IDF + linear |
| JPEGs / PNGs | Image | PyTorch + torchvision, pretrained ResNet/ViT (transfer learning) |
| WAV / MP3 | Audio | librosa, Whisper, Wav2Vec2 |
| Indexed by timestamp, order matters | Time series | Prophet, statsmodels, XGBoost on lag features |
| Relationships matter as much as nodes | Graph | PyTorch Geometric, DGL |
Most real projects are mixed — tabular features + a text column + a timestamp. Identify the dominant structure and build around it.
3. The pipeline (five steps, in order)
Raw data -> Clean -> Split -> Preprocess -> Train -> Evaluate ^ | +--- fit on train ------+ transform val/test The most important arrow is the feedback loop: preprocessing is fit on training data and applied to validation/test, never the other way round. Get that wrong and your score is a fantasy.
4. Cleaning — the unglamorous 60%
Handle missing values honestly (drop rows, drop columns, or impute using training-set stats only — and add a was_missing indicator, because missingness is often informative). Drop duplicates (df.sort_values('updated_at').drop_duplicates(subset=['id'], keep='last')). Treat outliers carefully — they can be the actual signal (fraud, rare events), so don't blindly cap. Add cheap sanity asserts (assert (df['age'] >= 0).all()). And check class balance now — if positives are <10%, accuracy and naive resampling will mislead you.
5. Splitting — where leakage starts
You need three sets: train (fit params), validation (tune / pick the model), test (touch once, at the very end). Pick the split that matches your data:
# Random (IID) — stratify to keep class ratios X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, stratify=y, random_state=42) # Time-aware — a random split here lets the model peek at the future cutoff = df['date'].quantile(0.80) train, test = df[df['date'] < cutoff], df[df['date'] >= cutoff] # Group-aware — every user ends up entirely in train OR test from sklearn.model_selection import GroupShuffleSplit train_idx, test_idx = next(GroupShuffleSplit(test_size=0.2, random_state=42).split(X, y, groups=df['user_id'])) The four leakage failure modes, each of which inflates your score and tanks production: (1) info from val/test leaking into preprocessing (scaling on the full dataset before splitting); (2) time leakage (training on data from after the prediction time); (3) group leakage (same user in both splits); (4) target leakage (a feature only available after the target is known).
6. Preprocessing — fit on train, transform everywhere
Scale numerics (StandardScaler / RobustScaler when outliers are present), encode categoricals (OneHotEncoder for low cardinality, TargetEncoder for high), vectorise text (TF-IDF or sentence embeddings), and engineer features (date parts, ratios like debt_to_income, aggregations, lag features — domain knowledge wins here). The key: never scaler.fit(X_full) before splitting. Wrap it all in a Pipeline + ColumnTransformer:
preprocessor = ColumnTransformer([ ('num', Pipeline([('imputer', SimpleImputer(strategy='median')), ('scaler', StandardScaler())]), num_cols), ('cat', Pipeline([('imputer', SimpleImputer(strategy='constant', fill_value='UNKNOWN')), ('ohe', OneHotEncoder(handle_unknown='ignore', sparse_output=False))]), cat_cols), ]) model = Pipeline([('preprocess', preprocessor), ('clf', LogisticRegression(max_iter=1000, class_weight='balanced'))]) model.fit(X_train, y_train) The pipeline object is the model — joblib.dump it, reload it for serving, and you cannot accidentally apply different preprocessing in production.
7. Choosing a model — start simple, always
| Task | Start with | If you need more |
|---|---|---|
| Tabular classification/regression | Logistic/Linear baseline | LightGBM / XGBoost / CatBoost |
| Image classification | Pretrained ResNet-50 (transfer) | Vision Transformer |
| Text classification | TF-IDF + Logistic | Pretrained BERT/DistilBERT |
| Time-series forecasting | Naive baseline, ARIMA | LightGBM w/ lag features, Prophet |
Always ship the baseline first. A 5-line logistic regression at 0.78 AUC tells you the problem is learnable. A 200-line transformer at 0.79 tells you you wasted a week. And watch bias vs variance: big train/val gap = variance (regularise, more data, simpler model); bad on both = bias (bigger model, better features, less regularisation).
8. Evaluation — the part that decides everything
Pick the metric the task requires, not the one your library prints by default.
The accuracy trap: if 97% of your data is "not default" and the model predicts "not default" for everyone, accuracy is 97% — useless model, great-looking number. For imbalanced classification, lead with PR-AUC and recall. Use precision when false positives are costly, recall when false negatives are, ROC-AUC when you'll pick the threshold later, log loss / calibration when probabilities matter. Always look at the confusion matrix.
y_proba = model.predict_proba(X_val)[:, 1] print('ROC AUC :', roc_auc_score(y_val, y_proba)) print('PR AUC :', average_precision_score(y_val, y_proba)) # lead with this when positives are rare print(classification_report(y_val, model.predict(X_val))) For regression: MAE (robust), RMSE (penalises big errors), R² — and np.log1p(y) before training for skewed targets. Cross-validate the way you'll deploy: StratifiedKFold (classification), GroupKFold (grouped), TimeSeriesSplit (train always before validate). Tune with grid (small), random (medium), or Optuna (large/expensive).
9. The full end-to-end example
Loan-default prediction — tabular, imbalanced, mixed numeric + categorical, time-aware split, LightGBM in a leak-proof Pipeline:
# Time-aware split by application_date quantiles train = df[df['application_date'] < df['application_date'].quantile(0.70)] val = df[(df['application_date'] >= q70) & (df['application_date'] < q85)] test = df[df['application_date'] >= q85] clf = lgb.LGBMClassifier(n_estimators=500, learning_rate=0.05, num_leaves=31, is_unbalance=True, random_state=42) # ... preprocessor.fit(X_train); transform val/test; early stopping on val AUC ... # Evaluate with ROC-AUC + PR-AUC + classification_report + confusion_matrix. # Touch the TEST set exactly once, at the very end. Every concept is in there: time-aware split, imputation inside the Pipeline (no leakage), class-weighted training (imbalance), early stopping (free regularisation), PR-AUC alongside ROC-AUC, test set touched once.
10. From notebook to production
The model artifact IS the pipeline (joblib.dump(model, 'model.pkl')) — preprocessing travels with it, so no train/prod skew. Monitor feature drift (inputs shift) and concept drift (the X→y relationship changes). Retrain on a cadence that matches the data (quarterly for stable sensors, daily/weekly for consumer behaviour, continuous for fraud). And A/B test new models — route 5% of traffic to v2 and measure business metrics, because a val score is necessary but not sufficient.
The mental model
Machine learning is engineering with statistics inside it. The model is the last 10% of the work; everything before it — understanding your data, splitting it honestly, engineering features, picking the metric that matches the business — is what separates a model that ships from a notebook that doesn't. Three habits: ship a baseline first; suspect leakage by default (every great-looking score has leakage until proven otherwise); build the pipeline once, retrain forever.
Go build the pipeline.
This is a condensed version. The full guide — with every section in depth, the dataset-type code, all the cleaning/splitting/preprocessing snippets, the complete metrics treatment, calibration, and the full runnable LightGBM loan-default pipeline — is on PrepStack:
→ Core AI/ML Concepts — Datasets, Preprocessing, and the Full Pipeline from Data to Evaluation

