Local AI testing - Running an 8-billion-parameter model on a four-core laptop CPU, no GPU: about 7 tokens per second, 80 °C, under 22 watts, 6.2 of 11.2 GB of memory. All figures were measured.

OS = Linux Debian 12 / Crunchbang ++

Test hardware:
HP ENVY Laptop 17m-ch0xxx, BIOS F.12. The CPU is an 11th Gen Intel Core i7-1165G7 (x86_64, one socket) with 4 physical cores and 8 logical, a 2.80 GHz base clock and 4.70 GHz maximum. Caches are 192 KiB L1d and 128 KiB L1i (4 instances each), 5 MiB L2 (4 instances) and 12 MiB L3 (shared). Supported instruction sets include AVX-512, AVX512-VNNI, AES-NI, SHA-NI, VAES, FMA and SSE4.2, plus VT-x virtualization.

Graphics are the integrated Intel Iris Xe (TigerLake-LP GT2, rev 01); there is no discrete GPU. Memory is 12 GB (about 11 GiB usable) with 975 MiB of swap. Storage is a 476.9 GB Intel NVMe SSD, which holds the root and /home filesystem (467 GB, 72 GB used), and a 27.3 GB Intel Optane NVMe module.

The model: llama3-8b-darkidol-2.0-uncensored-q4_k_m-imat.gguf is an 8.03-billion-parameter Llama-3 derivative quantized to 4 bits. File is 4.58 GB. It is an unaligned model (no refusal training), used here so that the measurements reflect inference performance without a safety layer. The SHA-256 hash matches the publisher's.

CPU versus integrated GPU:
The laptop has an Intel Iris Xe integrated GPU. On prompt processing, the Vulkan backend ran at 104 tokens/sec against about 28 for the CPU (AVX-512, 4 threads). On generation, the CPU ran at 6.6 tokens/sec against 2.6 for the GPU. Prompt processing is one large parallel calculation, which suits a GPU. Generation is a chain of sequential steps, each requiring the 4.58 GB of weights to be read through memory, and an integrated GPU shares the CPU's memory with no bandwidth advantage.

The GPU-enabled build of the inference engine also reserved about 1.6 GB of system memory with zero GPU layers requested. A CPU-only build recovered 1.58 GB at a cost of roughly 0.3 tokens/sec.

Uncapped, the CPU spiked to 93–98 °C in the first 15–20 seconds of a session, then settled near 80 °C. The fix was the processor's own power limits: long-term reduced from 28 W to 20 W, short-term from 51 W to 22 W. The long-term limit averages over a 28-second window, which covers the spike.

Results: worst-case temperature fell from 93–98 °C to 89 °C, and the average over the first 25 seconds from about 95 °C to 72 °C. Steady state went from 80.8 °C to 79.7 °C, leaving 11 °C of margin below the 100 °C throttle point. Generation speed went from 6.6 to 6.3 tokens/sec. That is 29 % less power for 4.5 % less speed.

Speed in practice: at about 7 tokens per second and roughly 0.75 words per token, output runs near 315 words per minute, slightly above typical reading speed. The capped benchmark figure is 6.3 tokens/sec; 7 is what the model sustains in ordinary interactive use. This is roughly the ceiling for this class of hardware.

How to reproduce

The engine is llama.cpp. These are the settings and commands behind the numbers above.

Build (CPU-only):

sudo apt install build-essential git cmake git clone https://github.com/ggml-org/llama.cpp cd llama.cpp cmake -B build -DGGML_NATIVE=ON cmake --build build --config Release -j$(nproc)

Settings used:

Threads: 4, one per physical core. Generation is limited by memory bandwidth, and hyperthreads sharing a core's cache did not help.
Context window: 8,192 tokens. Doubling it raised peak memory by 3 MB.
KV cache precision: 8-bit, which halves working memory.
Response cap: 512 tokens, about 73 seconds of output.
GPU layers: 0.

Run:

./build/bin/llama-cli -m MODEL.gguf -ngl 0 -c 8192 -t 4 -tb 4 -ctk q8_0 -ctv q8_0 -fa on -n 512 --temp 0.7

Power limits: the CPU caps are not a llama.cpp setting. They are the Linux RAPL interface, which takes values in microwatts. Check your current values first, since defaults differ by CPU:

cat /sys/class/powercap/intel-rapl/intel-rapl:0/constraint_0_power_limit_uw cat /sys/class/powercap/intel-rapl/intel-rapl:0/constraint_1_power_limit_uw

Constraint 0 is the long-term limit and constraint 1 is the short-term limit. To set 20 W and 22 W:

echo 20000000 | sudo tee /sys/class/powercap/intel-rapl/intel-rapl:0/constraint_0_power_limit_uw echo 22000000 | sudo tee /sys/class/powercap/intel-rapl/intel-rapl:0/constraint_1_power_limit_uw

These reset at reboot, and some laptop firmware locks or overrides them. To apply them at every boot, save the script below as /usr/local/sbin/darkidol-powercap (make it executable with chmod +x):

#!/bin/bash set -euo pipefail RAPL=/sys/class/powercap/intel-rapl/intel-rapl:0 echo 20000000 > "$RAPL/constraint_0_power_limit_uw" echo 22000000 > "$RAPL/constraint_1_power_limit_uw"

Then create /etc/systemd/system/darkidol-powercap.service:

[Unit] Description=CPU package power cap (RAPL PL1/PL2) After=multi-user.target [Service] Type=oneshot ExecStart=/usr/local/sbin/darkidol-powercap RemainAfterExit=yes [Install] WantedBy=multi-user.target

And enable it:

sudo systemctl enable --now darkidol-powercap.service

(2026) Ben Santora - Montserrat MA