llama.cpp: The Fastest Way to Run LLMs on CPU

llama.cpp is an open-source C++ library for running large language models on consumer hardware — including CPU-only machines. Created by Georgi Gerganov in 2023, it enabled the first practical local LLM inference without a GPU and remains the most hardware-flexible inference engine available.

What makes llama.cpp different

Most LLM inference tools require NVIDIA GPUs and CUDA. llama.cpp uses highly optimized CPU inference with optional GPU acceleration — making it the only option for:

• Machines with no GPU or non-NVIDIA GPUs (AMD, Intel)
• Apple Silicon Macs (uses Metal acceleration)
• Edge devices and embedded hardware
• Situations where you need absolute maximum control over inference

GGUF: the format that made it possible

llama.cpp introduced GGUF (GPT-Generated Unified Format) — a quantization format that dramatically reduces model size without proportional quality loss. A 70B model that normally requires 140GB at FP16 can run in 40GB at Q4 quantization, or 24GB at Q3.

Getting started with llama.cpp

git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
make -j4

# Download a GGUF model from Hugging Face
# Then run inference
./llama-cli -m models/llama-3.2-8b-q4_k_m.gguf -p "Hello, world" -n 128

# Run as an OpenAI-compatible server
./llama-server -m models/llama-3.2-8b-q4_k_m.gguf --port 8080

The server mode exposes an OpenAI-compatible API, so you can point Tokoscope or any other tool at it directly.

llama.cpp vs Ollama vs vLLM

When to use llama.cpp

• You have a CPU-only machine or non-NVIDIA GPU
• You're running on Apple Silicon and want Metal acceleration
• You need the smallest possible binary with no dependencies
• You're building on edge hardware or embedded systems
• You need maximum control over inference parameters

For most local development use cases, Ollama (which uses llama.cpp under the hood) is easier. For production serving on NVIDIA hardware, vLLM is faster. llama.cpp shines when neither of those apply.