You exported OHLCV from a free source, pointed your backtest at it, and got an equity curve. The problem: free and exported data ships with duplicate timestamps, gaps, mixed timezones and adjustment artifacts — and backtesting engines don't complain, they just eat it. Here's the cleaning pass I'd run on any OHLCV file before trusting a single trade signal from it.

(Part 1 covered getting the data for free; this covers making it honest.)

1. Normalize timestamps to UTC before anything else

df["timestamp"] = pd.to_datetime(df["timestamp"], utc=True)

Mixed timezones are the #1 silent killer: exchange data in UTC joined against a local-time indicator produces columns that align "mostly" — which is worse than not aligning at all. Set utc=True even when the source claims it's already UTC; a naive datetime with hidden local semantics will parse without error and be wrong.

2. Deduplicate on timestamp — and decide which row wins deliberately

dupes = df[df.duplicated("timestamp", keep=False)] df = df.drop_duplicates("timestamp", keep="last").sort_values("timestamp")

Duplicates usually come from re-running a download with overlapping ranges. keep="last" wins because later pulls contain the exchange's revisions — but only after you've looked at dupes: if the duplicated rows disagree in OHLC values, the source revised history, and you want the newest values everywhere, not just on the duplicated timestamps.

3. Find gaps with the frequency you actually expect

full = pd.date_range(df["timestamp"].min(), df["timestamp"].max(), freq="1D") missing = full.difference(df["timestamp"])

Not every gap is bad data: crypto trades 24/7 (any missing day is a real hole), equities skip weekends and holidays (compare against the trading calendar, not the calendar). The distinction matters because filling an equity weekend with forward-fill fabricates prices, while leaving a real crypto hole unfabricated means your rolling windows silently skip days.

4. Run the four OHLC sanity checks in one vectorized pass

bad = df[ (df["high"] < df[["open", "close"]].max(axis=1)) | (df["low"] > df[["open", "close"]].min(axis=1)) | (df["volume"] < 0) | ((df["open"] <= 0) | (df["close"] <= 0)) ]

High below the open, low above the close, negative volume, zero or negative prices — each one means the row is corrupt and any indicator computed through it is corrupt downstream. Free sources ship all four regularly. Decide per-row: drop it, or re-pull that date range from the source.

5. Catch adjustment artifacts: impossible returns

rets = df["close"].pct_change() df.loc[rets.abs() > 0.5, "timestamp"]

A single-day move beyond ±50% on a major pair is almost never the market — it's a split/dividend adjustment applied to part of the history but not all of it, or a raw/adjusted seam where two exports were concatenated. Print those rows and look: if the open and the previous close disagree by exactly the split ratio, you found the seam. Fix by re-pulling the whole range adjusted (auto_adjust=True on yfinance, or adjusted endpoints on your exchange) — never by patching the seam rows by hand.

6. Final form: sorted, typed, Parquet

df = df.sort_values("timestamp").reset_index(drop=True) df.to_parquet("btc_1d_clean.parquet")

Sorting is not cosmetic — a duplicated-download artefact that leaves one out-of-order block will make pct_change() compute returns backwards across the seam. Parquet preserves dtypes (datetimes stay datetimes), shrinks the file ~5x, and makes every future load skip the re-parsing step entirely.

7. Cache the clean file, re-pull the raw

Free endpoints revise history (splits, error corrections). Keep your cleaned Parquet for iteration speed, but re-download and re-run the cleaning pass weekly — a clean cache of stale raw data is still stale data, just tidier.

Every check in steps 1–6 is exactly what the data cleaner in my Python Backtest Toolkit automates: point it at a messy CSV/Excel export and it runs the duplicate/gap/OHLC/volume checks, fixes timezones, flags impossible returns, and writes clean Parquet ready for a backtest. The toolkit also ships a one-command backtrader template (Sharpe, max drawdown, win rate), 3 sample strategies and a 500-day sample OHLCV set. $1 launch price while collecting the first honest reviews — if it misses an error type you've hit in the wild, tell me and I'll add it: https://datafixtools.sell.app/product/python-backtest-toolkit

Not investment advice. Educational tooling only.