Making Local LLMs Usable on 8GB GPUs

A local model can manage a brief question without any issues, but it may have difficulty when dealing with a complete document. The model's internal settings stay the same, but the effort needed and the memory required to process the request increase a lot.

In Part 1, I explained how to run a quantized model on a smaller GPU.

The next step is to choose a task, measure how long it takes, and find a setup that works within the GPU's limits.

It includes memory calculations, a fixed starting configuration, and a Python-based benchmark that you can run on your own computer. The numerical memory estimates are calculated values, and this article does not claim to have measured improvements in GPU performance.

To understand the actual delay you experience, there are two main waiting periods in a conversation. The first occurs when the model processes the initial prompt, often called the prefill phase. The second happens when the model generates the answer one word at a time, known as the decode phase.

If a document takes several seconds to process, fast decoding won't eliminate that initial delay. However, if the first words appear quickly but the rest of the response is slow, then the decoding phase is the area that needs optimization.

Keep these measurements separate:

A target such as "start a short answer within two seconds" can guide your own experiments. It is a product requirement you choose, not a promised result for every 8GB card.

Budget for the KV cache as well as the weights

The model file is only part of the memory requirement. During generation, the runtime also stores attention keys and values for tokens already processed. This KV cache avoids recomputing those states, but it needs memory of its own. Working buffers and other GPU applications need space too. [1]

For a conventional full-attention transformer, an approximate KV-cache payload is:

KV bytes = 2 x layers x stored tokens x KV heads x head dimension

x bytes per cache element

The initial 2 accounts for keys and values. Use the number of KV heads, which can differ from the number of query heads.

Qwen2.5-7B-Instruct has 28 layers, 4 KV heads, and a head dimension of 128, obtained from its hidden size of 3,584 divided by 28 query heads. With FP16 cache elements, the calculation becomes 2 x 28 x tokens x 4 x 128 x 2. [2]

These figures describe one sequence and exclude allocation padding, runtime buffers, and model weights. They are not measurements of total VRAM usage. Other architectures, cache layouts, and attention types need different calculations.

This is why a large advertised context window needs its own memory budget. On this model, moving from 4K to 32K tokens adds about 1.53 GiB of calculated FP16 cache payload. Whether that increase fits depends on everything else already occupying the GPU.

Weight quantization and cache quantization are separate choices. A Q4_K_M model can still use an FP16 KV cache. Reducing cache precision may free up space, but it needs a quality check and a latency measurement. Hugging Face's cache documentation also cautions that cache quantization can hurt latency in some short-context situations. [1]

Establish one repeatable baseline

The shell commands below target Linux or WSL2 with an NVIDIA GPU. Install a working NVIDIA driver, a compatible CUDA Toolkit, Git, CMake, and a C++ compiler first. The benchmark itself requires Python 3.10 or newer and no third-party Python packages.

Use the current CMake build route in the llama.cpp build documentation: [3]

git clone https://github.com/ggml-org/llama.cpp cd llama.cpp cmake -B build -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release cmake --build build --config Release -j 4 git rev-parse HEAD > llama-commit.txt ./build/bin/llama-server --version > llama-version.txt ./build/bin/llama-server --help > llama-help.txt nvidia-smi > gpu-environment.txt 

Keep that checkout unchanged throughout the experiment. Save the model filenames and their SHA-256 hashes too, including every shard if the download is split. A model tag alone is not enough to reproduce an experiment after its files change.

I use the official Qwen GGUF repository because its model configuration is available to inspect. It is a concrete baseline, not a claim that this is the best available model. [4]

Start the server from the llama.cpp directory:

./build/bin/llama-server \ -hf Qwen/Qwen2.5-7B-Instruct-GGUF:Q4_K_M \ --n-gpu-layers all \ --ctx-size 4096 --parallel 1 \ --batch-size 512 --ubatch-size 128 \ --flash-attn off \ --cache-type-k f16 --cache-type-v f16 \ --fit off --no-context-shift --perf \ --host 127.0.0.1 --port 8080 \ 2> baseline-server.log 

The first launch downloads the selected model. Exclude that download and startup from request timings. Once the server is ready, inspect its log for the CUDA device, layer placement, cache allocation, and effective context settings. If an installed version rejects an option, consult its saved help output before continuing. The current server options are documented upstream. [5]

This configuration requests full GPU offload; it does not guarantee that every 8GB system has enough free memory. If startup fails, first close another GPU workload or test a smaller context. If partial offload is necessary, record the actual GPU layer count and use it consistently across the comparison.

The baseline has one request slot. Keep other inference clients stopped while measuring. With automatic fitting disabled, a failed configuration is visible instead of becoming a different experiment through an adjusted setting.

Run a benchmark that preserves the evidence

Save this small input as prompt.txt:

Answer using only these project notes.

Project Cedar: Mira owns the CSV import validator.

The release target is 15 November 2026.

Duplicate IDs are blocking the import test.

No project budget has been approved.

In at most 80 words, identify the owner, target date, and blocker. Then say whether the notes contain an approved budget amount. Do not invent missing details. Save the following script as bench_local.py. It formats the prompt using the server's chat template, checks a declared token budget, performs a warm-up, and records five measured requests by default.

#!/usr/bin/env python3 """Measure a local llama-server with Python 3.10+ standard libraries only.""" import argparse import hashlib import json from pathlib import Path import statistics import time from urllib.error import HTTPError, URLError from urllib.request import Request, urlopen def request(base, route, data): body = json.dumps(data).encode("utf-8") return Request(base + route, data=body, headers={"Content-Type": "application/json"}) def post(base, route, data): with urlopen(request(base, route, data), timeout=120) as response: value = json.load(response) if "error" in value: raise RuntimeError(str(value["error"])) return value def events(response): """Parse complete SSE events, including multiple data lines.""" data = [] for raw in response: line = raw.decode("utf-8").rstrip("\r\n") if not line: if data: payload = "\n".join(data) if payload == "[DONE]": return yield json.loads(payload) data = [] elif line.startswith("data:"): data.append(line[5:].removeprefix(" ")) if data: raise RuntimeError("Incomplete SSE event at end of stream") def measure(base, prompt, max_tokens): req = request(base, "/completion", { "prompt": prompt, "stream": True, "n_predict": max_tokens, "temperature": 0, "seed": 42, "cache_prompt": False, }) first = final = None parts = [] started = time.perf_counter() with urlopen(req, timeout=300) as response: for event in events(response): if "error" in event: raise RuntimeError(str(event["error"])) content = event.get("content", "") if content: if first is None: first = time.perf_counter() - started parts.append(content) if event.get("stop"): final = event break elapsed = time.perf_counter() - started if final is None: raise RuntimeError("Stream ended without final completion statistics") if final.get("truncated"): raise RuntimeError("Context overflow invalidated this run") timing = final.get("timings", {}) needed = ("prompt_n", "prompt_ms", "predicted_n", "predicted_per_second") if any(key not in timing for key in needed): raise RuntimeError("Missing timings; check --perf and server version") if timing.get("cache_n", 0): raise RuntimeError("Unexpected prefix-cache reuse invalidated this run") if first is None or timing["predicted_n"] <= 0: raise RuntimeError("No generated text to measure") return { "ttft_s": first, "elapsed_s": elapsed, "decode_tok_s": timing["predicted_per_second"], "prompt_tokens": timing["prompt_n"], "prompt_eval_ms": timing["prompt_ms"], "output_tokens": timing["predicted_n"], "cached_prompt_tokens": timing.get("cache_n"), "stop_type": final.get("stop_type"), "text": "".join(parts), } def main(): parser = argparse.ArgumentParser(description=__doc__) parser.add_argument("--prompt", type=Path, required=True) parser.add_argument("--label", required=True) parser.add_argument("--base", default="http://127.0.0.1:8080") parser.add_argument("--context", type=int, default=4096) parser.add_argument("--max-tokens", type=int, default=256) parser.add_argument("--runs", type=int, default=5) args = parser.parse_args() if min(args.context, args.max_tokens, args.runs) <= 0: parser.error("context, max-tokens and runs must be positive") base = args.base.rstrip("/") user_text = args.prompt.read_text(encoding="utf-8") messages = [{"role": "user", "content": user_text}] prompt = post(base, "/apply-template", {"messages": messages})["prompt"] tokens = post(base, "/tokenize", { "content": prompt, "add_special": True, "parse_special": True, })["tokens"] if len(tokens) + args.max_tokens + 32 > args.context: parser.error("prompt + output + 32-token reserve exceeds --context") warmup = measure(base, prompt, args.max_tokens) runs = [measure(base, prompt, args.max_tokens) for _ in range(args.runs)] metrics = ("ttft_s", "elapsed_s", "decode_tok_s") report = { "label": args.label, "base": base, "prompt_sha256": hashlib.sha256(user_text.encode()).hexdigest(), "formatted_prompt_tokens": len(tokens), "declared_context": args.context, "max_tokens": args.max_tokens, "cache_prompt": False, "warmup": warmup, "runs": runs, "median": {key: statistics.median(r[key] for r in runs) for key in metrics}, "range": {key: [min(r[key] for r in runs), max(r[key] for r in runs)] for key in metrics}, } print(json.dumps(report, ensure_ascii=False, indent=2)) if __name__ == "__main__": try: main() except HTTPError as exc: detail = exc.read().decode("utf-8", errors="replace") raise SystemExit(f"HTTP {exc.code}: {detail}") from exc except (URLError, OSError, ValueError, KeyError, RuntimeError) as exc: raise SystemExit(f"Benchmark failed: {exc}") from exc 

In a second terminal, run it from the folder containing the script and input:

python3 bench_local.py --prompt prompt.txt \ --label A-4096-F16 --context 4096 > A-short.json 

Its ttft_s is a client-side approximation of time to first token: timing starts before the generation HTTP request and stops at the first nonempty content chunk. That chunk may contain more than one token. Template formatting, token counting, model loading, and any document retrieval happen outside this timer.

The script uses the server's token statistics for decoding speed. Counting streaming events would measure chunks, whose boundaries are not reliable token boundaries. The native completion endpoint and its timing fields are documented in llama.cpp. [5]

Prefix reuse is disabled for these comparisons. The model still uses a KV cache within each generation; the setting prevents an identical prompt from being accelerated by a previous request's prefix. The separate warm-up is excluded from the median. Failed or truncated requests stop the experiment instead of producing a successful-looking score.

The --context value is a declared budget, not a command that reconfigures the server. Match it to the server's effective context. The script allows an additional 32-token reserve and records the formatted input token count.

Repeat the test with a realistic document saved in another input file. Keep that file unchanged across configurations. A short fixture helps check the pipeline, but it cannot establish long-document performance.

Change one setting at a time

Restart the server for each configuration, adjusting only the settings listed here. Everything else, including weights, prompt files, output cap, and GPU placement, stays fixed.

A to B tests Flash Attention. B to C tests cache precision. C to D tests the larger context allocation. D to E tests a larger physical batch. Q8_0 cache support depends on the model, backend, and build; inspect startup output and skip unsupported combinations rather than comparing a fallback unknowingly. [5]

For C, replace the baseline attention and cache options with:

--flash-attn on --cache-type-k q8_0 --cache-type-v q8_0 

Flash Attention is designed to reduce memory traffic during attention computation. It does not remove the persistent KV-cache requirement. [6] A larger physical batch may help prompt processing but can require more temporary memory, so treat E as an experiment rather than a recommended default.

There are two different comparisons to make. Use the same input across A through E to compare configuration overhead. Separately, increase the actual input length to study longer requests. Changing an 8K capacity setting while keeping a 200-token prompt does not simulate an 8K prompt.

Monitor GPU usage in a third terminal: [7]

nvidia-smi \ --query-gpu=timestamp,name,memory.total,memory.used,utilization.gpu \ --format=csv -l 1 > gpu-samples.csv 

Stop sampling with Ctrl+C when the experiment ends. These are device-wide samples, including other processes; one-second sampling can miss brief peaks. Use them alongside the server's allocation logs, not as an exact per-request peak measurement.

Compare median first-text delay, decoding rate, total time, and the observed range. Also inspect output lengths and stop_type: an answer that hit the output limit may be incomplete. Five repetitions are an initial check, not enough evidence for a dependable p95 latency claim.

Keep answer quality in the experiment

For the Cedar fixture, check that the answer preserves Mira, the date, and duplicate IDs. It should also distinguish an unapproved budget from a known budget amount. A faster configuration that invents an amount fails this task.

Build a small fixed test set from the work you intend to do. Include a question answered by a detail near the end of a document, an extraction task with a required format, and a question whose answer is absent. Save the generated text before changing cache precision or switching to a smaller model.

A useful score is the number of tasks that meet your written acceptance rules. Keep those rules unchanged while comparing settings. Temperature zero reduces sampling variation, but numerical differences across backends and builds can still affect output.

Reduce the prompt before expanding the hardware budget

Once the runtime is measured, inspect what the application sends it. A chat assistant can accumulate old turns, repeated instructions, and irrelevant document chunks until most of its context carries little useful evidence.

For a document assistant with a 4,096-token context, this is one possible allocation:

These are application design choices, not model limits or measured token counts. Count the fully formatted prompt with the model's tokenizer. The input and reserved output must fit together.

Remove duplicate excerpts and retain passages that support the answer, including qualifications and conflicting evidence. Evaluate retrieval changes against the fixed question set. A shorter prompt is only useful when it still contains what the model needs. Once a configuration meets your latency target and quality checks, save its launch command, engine revision, model hashes, prompts, raw responses, and GPU logs together. That gives you a working profile you can revisit when a model update or a new workload changes the results.

References

[1] Hugging Face Transformers, Cache strategies.

[2] Qwen, Qwen2.5-7B-Instruct configuration. The KV-cache table is calculated from these architecture values.

[3] llama.cpp, Build documentation.

[4] Qwen, Official Qwen2.5-7B-Instruct GGUF repository.

[5] llama.cpp, HTTP server documentation.

[6] Dao and colleagues, FlashAttention.

[7] NVIDIA, NVIDIA System Management Interface documentation.