My submission for the Hacktoberfest Open-Source AI Challenge: Week 3 — "Slow and Steady." 🦥

Repo: https://github.com/garanews/slothctf

The lazy payoff

Writing reverse-engineering CTF challenges by hand is tedious: you invent a scheme, write the C, compile it, write a solver, double-check the flag actually comes out, make sure strings doesn't give it away… for every single task.

I wanted to type one sentence and get a finished, verified challenge:

python main.py generate --name xor_demo \ --prompt "Create an easy reverse engineering challenge with XOR encryption" \ --model qwen2.5-coder:32b

SlothCTF does exactly that. A local open-weight model writes the whole challenge in C; the tool compiles it, runs it, and proves it works before I ever look at it. I fed it the 20 challenge ideas I'd been meaning to build and got 20/20 valid challenges out — source, compiled binary, player README, and an organizer walkthrough each. The slow, boring part now happens while I get a coffee. Very on-brand for International Sloth Day.

The twist nobody warns you about: LLMs can't do byte math

My first design asked the model to do everything, including computing the hidden bytes — "store the flag XORed with 0x42 as an array." Every model I tried got the arithmetic wrong, from Gemma 3 4B up to Qwen2.5-Coder 14B. It would declare the flag sloth{easy_reverse} and then emit an array that decoded to LCTE…. An LLM simply cannot reliably compute flag[i] ^ key across 20 bytes in its head.

This is the same "LLMs confidently hallucinate in reverse engineering" problem others have hit — see Stopping LLM Hallucinations in Reverse Engineering: Meet Reverify by qisanxi and, for the CTF-authoring angle, NiceTryGPT: Less pattern matching. More actual hacking. by aleff.

The fix: let the machine do the arithmetic

So I stopped asking the model to compute anything. Instead it writes two C programs, and the machine runs them:

model → encoder.c (organizer tool) + challenge.c (has a placeholder for the bytes) → compile & run encoder on the secret → EXACT bytes (machine math) → inject bytes into challenge.c, compile, run → behaviour check → decode: stdout == flag | check: right input accepted, wrong rejected → strings(binary) must NOT contain the plaintext secret ↺ any failure is fed back to the model for another attempt

The model invents the scheme and writes the code; the compiler does the exact byte math it's bad at. No templates, no per-scheme if ladders. Here's a real generated pair (XOR):

/* organizer/encoder.c — run on the flag to produce the bytes */ int main(int argc, char **argv) { for (char *p = argv[1]; *p; ++p) printf("0x%02x,", (unsigned char)(*p ^ 0x42)); } 
/* src/challenge.c — the player's binary; the machine filled in the array */ static unsigned char data[] = { 0x31, 0x2e, 0x2d, 0x36, 0x2a, 0x39, /* … */ }; int main(void) { for (size_t i = 0; i < sizeof data; ++i) putchar(data[i] ^ 0x42); }

Two kinds of challenge

  • decode — the binary decodes the embedded array and prints the flag.
  • check — the binary reads stdin and accepts only the right secret (password checks, hashes, ciphers, state machines, anti-debug gates…).

That second mode covers the classic "crack the binary" tasks. The verifier feeds the real secret (must be accepted) and a wrong one (must be rejected), and confirms the plaintext never shows up under strings.

See it run

All 20 of my prompts, re-validated independently — each one is compiled, run, its flag checked, and strings-checked:

$ for n in generated/*; do python main.py validate --dir "$n"; done xor_demo OK state_machine_demo OK obfuscation_demo OK func_pointers_demo OK anti_debug_demo OK constructor_demo OK custom_b64_demo OK fnv1a_hash_demo OK tea_crypto_demo OK sbox_crypto_demo OK rc4_cipher_demo OK dynamic_imports_demo OK junk_code_demo OK math_equations_demo OK multithread_check_demo OK custom_vm_demo OK timing_attack_demo OK anti_vm_demo OK self_modifying_demo OK seh_exception_demo OK VALID 20/20

A decode challenge, end to end:

$ cat generated/xor_demo/challenge.json { "title": "XORed Flag", "difficulty": "easy", "flag": "sloth{easy_xor}", "challenge_kind": "decode" } $ ./generated/xor_demo/bin/challenge sloth{easy_xor}

A check challenge (anti-debug) — correct input wins, wrong input loses, and the secret is never in the binary:

$ echo 'sloth{DebugMeNot}' | ./generated/anti_debug_demo/bin/challenge Access Granted $ echo 'wrongpass' | ./generated/anti_debug_demo/bin/challenge Denied $ strings ./generated/anti_debug_demo/bin/challenge | grep DebugMeNot (nothing — the plaintext secret never appears in the binary)

A couple of lessons that cost me attempts

  • Tell the model where it compiles. ptrace (Linux) and MSVC __try/__except don't exist on Windows/MinGW. Adding a one-line "BUILD TARGET: Windows/MinGW, no ptrace/SEH" to the prompt flipped the anti-debug and exception challenges from fails to valid on the first try (the model reached for IsDebuggerPresent instead).
  • Hashes and block ciphers are the hard case. FNV-1a and TEA need two byte-identical implementations (encoder + challenge) to agree. The fixes: emit the target one byte per token (don't print a 32-bit word as a single token — a pitfall also worth reading Clean JSON Extraction with Ollama and Python by nearshi about), and tell the model to copy the transform function verbatim into both programs. That got me the last 2 of 20.

Why open innovation matters here

This only works because it's open and local:

  • It runs offline on open-weight models via Ollama. No cloud API, no per-call cost, no challenge ideas leaving my machine — which matters when the "challenges" are security tasks.
  • I can swap models freely. I watched the exact same pipeline fail on a 4B and succeed on a 32B; that experiment is only possible when you control the model. (For picking a local coder model by VRAM, Best Local LLM for Coding: 8GB to 24GB VRAM Picks by mrsaynothing is a good map.)
  • "Slow and Steady" is the whole point. The model is allowed to take minutes and several tries, because a deterministic verifier catches every mistake. I don't need it fast; I need it correct, and I'm not the one waiting.

Results

With Qwen2.5-Coder 32B on a borrowed A100: 20/20 of my prompts produced valid, verified challenges — XOR, custom Base64, S-box, RC4, TEA, FNV-1a, a bytecode VM, state machines, function-pointer checks, anti-debug, timing side-channels, and more. Small models follow the two-program protocol poorly; 32B is the sweet spot.

Which brings me to my one serious piece of engineering advice: the cheapest way to make a local 32B model pleasant to iterate on is, regrettably, to just buy yourself a local A100. 🦥💸 (I'm told a workstation with 512 GB of RAM also works, slowly. The sloth approves.)

Try it 

git clone https://github.com/garanews/slothctf cd slothctf python main.py generate --name demo --prompt "a reverse challenge with a custom cipher" \ --model qwen2.5-coder:32b --ollama-url http://localhost:11434 python main.py validate --dir generated/demo

Everything — the generator, the self-verifying loop, the two challenge kinds, remote-Ollama support, and tests — is in the repo: https://github.com/garanews/slothctf

Next up: a GUI with a small database to browse generated challenges by type and difficulty, and support for the other classic CTF genres (pcap, memory dumps, disk forensics, logs, OSINT).

hf26challenge