Make a local LLM test reproducible.

Record the configuration

Keep the exact model revision, file names, SHA-256 hashes, quantization, combined bytes, software commit, backend, drivers, OS, hardware, memory, power mode and full command. Include context length, cache precision, GPU offload and concurrent sessions. Leave unknown fields blank.

Download the blank test worksheet (CSV). The worksheet is original Local AI Stats material, available under CC BY 4.0. It contains no benchmark results or third-party datasets.

Measure different workloads separately

1. Close unrelated workloads. Record idle memory and power before the test.
2. Warm up the model. Repeat each fixed workload at least five times, then keep the individual results, median and spread. Five runs do not by themselves prove statistical significance.
3. Separate prompt processing from token generation. Do not compare aggregate multi-user throughput with single-user chat.
4. Test short and longer contexts separately. Record time to first token, peak RAM/VRAM, swapping and failed runs.
5. Use the same task prompts for quality checks. Faster generation does not establish better answers.

The official llama-bench documentation describes repetitions, prompt/generation lengths and machine-readable outputs. Pin a version and check its help output before using these examples.

llama-bench -m /path/to/model.gguf -p 512 -n 128 -r 5 -o json > result.json

This example measures prompt processing and generation under the tool’s defaults. It does not specify a long conversation context or fix GPU offload. Record the effective settings; do not label its output as an end-to-end application benchmark.

Useful original datasets to build

• A context-versus-memory sweep at 2K, 8K and 32K tokens, including out-of-memory outcomes for the same quantization.
• A matched CPU, Metal, CUDA and Vulkan comparison with identical model files and workload settings where each backend supports them.
• A quantization comparison that reports task quality, memory and generation speed on the same hardware.
• A dated price/configuration history from permitted sources, including unavailable variants rather than dropping them.

Publish the raw runs, hashes and collection method alongside summary tables. These records let others check and cite the work. No independent results have been collected for these proposed datasets.