teamai-test/.teamai/skills/common/golang-benchmark/references/benchstat.md

benchstat Reference

benchstat computes statistical summaries and A/B comparisons of Go benchmark results. A single benchmark run tells you nothing about variance — benchstat tells you whether the difference between two runs is real or noise.

Installation

go install golang.org/x/perf/cmd/benchstat@latest

Usage

benchstat [flags] inputs...

Each input is a file containing go test -bench output. Optionally label inputs with label=path syntax.

Basic Workflow

Step 0: Write benchmarks

Use the standard Go benchmark function signature in *_test.go:

Step 1: Measure baseline

Run benchmarks with -count=10 or more. Each run produces one data point — you need at least 10 to compute a meaningful confidence interval:

go test -run='^$' -bench=BenchmarkParse -benchmem -count=10 ./pkg/parser | tee old.txt

-run='^$' skips unit tests so only benchmarks run — avoids wasting time on tests during measurement sessions.

Step 2: Make your change

Edit the code you want to optimize.

Step 3: Measure again

Same command, same flags, same machine, same load conditions:

go test -run='^$' -bench=BenchmarkParse -benchmem -count=10 ./pkg/parser | tee new.txt

Step 4: Compare

benchstat old.txt new.txt

Output:

goos: linux
goarch: amd64
pkg: myapp/pkg/parser
cpu: AMD Ryzen 9 5950X 16-Core Processor
          │   old.txt   │              new.txt               │
          │   sec/op    │   sec/op     vs base               │
Parse-32    4.592µ ± 2%   3.041µ ± 1%  -33.78% (p=0.000 n=10)

          │  old.txt   │             new.txt              │
          │    B/op    │    B/op     vs base              │
Parse-32    1.024Ki ± 0%   0.512Ki ± 0%  -50.00% (p=0.000 n=10)

          │  old.txt  │            new.txt             │
          │ allocs/op │ allocs/op   vs base            │
Parse-32    12.00 ± 0%   6.000 ± 0%  -50.00% (p=0.000 n=10)

Reading the Output

Element	Meaning	What to look for
median (e.g., 4.592µ)	Central value across runs — more robust than mean because outliers don't skew it	The reference number for this benchmark
± N% (e.g., ± 2%)	Half-width of the 95% confidence interval as a percentage of the median	Low (≤2%) = stable measurement. High (>5%) = noisy — investigate noise sources before trusting results
vs base (e.g., -33.78%)	Percentage change from the first input (base) to subsequent inputs	Negative = faster/smaller. Positive = slower/larger
p=N (e.g., p=0.000)	p-value from Mann-Whitney U-test (non-parametric)	<0.05 = statistically significant. ≥0.05 = difference could be noise
n=N (e.g., n=10)	Number of samples used in the comparison	Should usually match your -count; if it does not, check that each input file contains the same benchmark rows and units
~	No statistically significant difference detected	Do NOT claim improvement — the change might be zero
geomean row	Geometric mean of changes across all benchmarks in the table	Overall proportional change; useful when comparing many benchmarks at once

Unit normalization

benchstat automatically normalizes units for display:

• ns/op → displayed as sec/op (with µ, m prefixes) to avoid nonsensical µns/op
• MB/s → displayed as B/s (with K, M, G prefixes)

When the ~ symbol appears

Parse-32    4.592µ ± 8%   4.481µ ± 7%  ~ (p=0.089 n=10)

This means benchstat cannot distinguish the difference from random noise. The wide confidence intervals (±8%, ±7%) overlap. Do not claim improvement. Options:

• Increase -count to 20+ (narrower CI may reveal a real difference)
• Reduce noise sources (close applications, plug in power, use dedicated machine)
• Accept that the change has no measurable effect on this benchmark

Flags Reference

Projection flags

These flags control how benchmark results are grouped into tables, rows, and columns.

Flag	Default	Purpose
-table KEYS	.config	Group results into separate tables by these keys
-row KEYS	.fullname	Group results into table rows by these keys
-col KEYS	.file	Compare across columns with different values of these keys
-ignore KEYS	(none)	Omit keys from grouping — suppresses "benchmarks vary" warnings

Available keys:

Key	Meaning	Example value
.name	Base benchmark name (without sub-benchmark config)	Parse from BenchmarkParse/size=4k-16
.fullname	Full name including sub-benchmark configuration	Parse/size=4k-16
.file	Input file name or custom label	old.txt or baseline
.config	All file-level configuration keys combined	goos/goarch/pkg/cpu
.unit	Metric unit name	sec/op, B/op, allocs/op
/{name-key}	Per-benchmark sub-name key	/size extracts 4k from Parse/size=4k
/gomaxprocs	GOMAXPROCS value — recognizes both /gomaxprocs=N and the -N suffix convention	16 from Parse-16
goos	Operating system (from benchmark output header)	linux, darwin
goarch	Architecture (from benchmark output header)	amd64, arm64
pkg	Package path (from benchmark output header)	myapp/pkg/parser
cpu	CPU model (from benchmark output header)	AMD Ryzen 9 5950X

Sort order modifiers — append to any key:

Modifier	Meaning	Example
@alpha	Alphabetic sort	/format@alpha
@num	Numeric sort (understands prefixes: 2k, 1Mi)	/size@num
@(val1 val2 ...)	Fixed order + filter (only listed values, in this order)	/format@(gob json)

Filter flag

Flag	Purpose
-filter EXPR	Filter which benchmarks are processed before grouping and comparison

See Filter Expression Syntax below for full details.

Input labeling

Not a flag but a syntax feature — label input files for clearer column headers:

# Default: file names become column headers
benchstat old.txt new.txt

# Custom labels
benchstat baseline=old.txt optimized=new.txt

# Multiple versions
benchstat v1=v1.txt v2=v2.txt v3=v3.txt

The first input is always the base for comparison. All subsequent inputs are compared against it.

Filter Expression Syntax

Filters select which benchmarks to include before grouping and comparison. The syntax is:

Matching operators

Pattern	Meaning	Example
key:value	Exact match	goos:linux
key:"value"	Exact match with quoted value (allows spaces, special chars)	pkg:"github.com/user/repo"
key:/regexp/	Regular expression match (Go regexp syntax)	.name:/Parse|Encode/
key:(val1 OR val2)	Match any of the listed values	goos:(linux OR darwin)
*	Match everything (all benchmarks)	*

Logical operators

Operator	Meaning	Example
x y	AND — both must match (implicit)	goos:linux goarch:amd64
x AND y	AND — explicit form	goos:linux AND goarch:amd64
x OR y	OR — either must match	goos:linux OR goos:darwin
-x	NOT — must not match	-goos:windows
(...)	Grouping / subexpression	(goos:linux OR goos:darwin) -pkg:/internal/

Filter key types

Key	What it matches	Example
.name	Base benchmark name	.name:Parse
.fullname	Full name with sub-benchmark config	.fullname:/Parse\/size=4k/
/{name-key}	Sub-benchmark parameter	/size:4k
/gomaxprocs	GOMAXPROCS value	/gomaxprocs:16
.file	Input file label	.file:old.txt
.unit	Metric unit	.unit:sec/op
goos	OS from header	goos:linux
goarch	Architecture from header	goarch:amd64
pkg	Package from header	pkg:/parser/

Filter examples

# Only Parse benchmarks
benchstat -filter '.name:Parse' old.txt new.txt

# Only benchmarks with size=4096 sub-parameter
benchstat -filter '/size:4096' old.txt new.txt

# Exclude Parallel benchmarks
benchstat -filter '-.name:/Parallel/' old.txt new.txt

# Linux amd64 only
benchstat -filter 'goos:linux goarch:amd64' old.txt new.txt

# Multiple benchmark names
benchstat -filter '.name:(Parse OR Encode OR Decode)' old.txt new.txt

# Complex: Linux or Darwin, not internal packages, only sec/op metric
benchstat -filter '(goos:linux OR goos:darwin) -pkg:/internal/ .unit:sec/op' old.txt new.txt

# Regex: all benchmarks starting with Bench
benchstat -filter '.name:/^Bench/' old.txt new.txt

Projection Examples

Default: before/after file comparison

benchstat old.txt new.txt
# Equivalent to:
benchstat -table .config -row .fullname -col .file old.txt new.txt

Creates one row per benchmark, one column per file.

Compare sub-benchmark parameters within a single file

When a single benchmark file contains multiple sub-benchmarks (e.g., BenchmarkEncode/format=json and BenchmarkEncode/format=gob):

benchstat -col /format bench.txt

Creates columns for each value of /format, comparing them against each other.

Simplify rows to base name only

benchstat -col /format -row .name bench.txt

Strips sub-benchmark configuration from row names, making the table more compact.

Control column order

# Force gob first, then json (instead of alphabetical)
benchstat -col '/format@(gob json)' bench.txt

Group by GOMAXPROCS

benchstat -col /gomaxprocs bench.txt

Compares performance across different GOMAXPROCS values within the same file.

Separate tables per package

benchstat -table pkg old.txt new.txt

Creates one table per package — useful when comparing benchmarks across multiple packages.

Ignore a dimension

# Suppress "benchmarks vary in /gomaxprocs" warning
benchstat -row .name -ignore /gomaxprocs bench.txt

Compare three versions

benchstat v1=v1.txt v2=v2.txt v3=v3.txt

Shows v2 vs v1 and v3 vs v1 (first input is always the base).

Cross-dimensional comparison

# Rows = benchmark name, columns = OS, separate tables per architecture
benchstat -row .name -col goos -table goarch results.txt

Unit Metadata

assume=exact

For metrics that should not vary between runs (e.g., binary size, generated code size):

BenchmarkSize 1 42 custom-bytes/op
Unit custom-bytes/op assume=exact

With assume=exact:

• Non-parametric statistics are disabled
• benchstat warns if measured values vary
• Shows comparisons even with a single before/after measurement (no -count needed)

assume=nothing (default)

Standard behavior — uses non-parametric statistics (median + Mann-Whitney U-test). Requires multiple samples.

Interleaving Runs

Sequential runs (all old, then all new) are vulnerable to systematic bias — thermal throttling builds up over time, background processes come and go, CPU frequency scaling adapts. Interleaving reduces this:

# Pre-compile both versions to avoid measuring compilation time
go test -c -o old.test ./pkg/parser
# ... make your change ...
go test -c -o new.test ./pkg/parser

# Interleave runs — alternating reduces systematic bias
for i in $(seq 1 10); do
    ./old.test -test.bench=BenchmarkParse -test.benchmem >> old.txt
    ./new.test -test.bench=BenchmarkParse -test.benchmem >> new.txt
done

benchstat old.txt new.txt

Pre-compiling with go test -c is critical — without it, each go test -bench invocation includes compilation time, which varies and contaminates results.

How Many Runs?

Scenario	Minimum -count	Why
Quick local check	6	Enough for a rough confidence interval; fast feedback loop
Pre-merge comparison	10	Standard for detecting moderate (>5%) changes with confidence
Detecting small changes (<5%)	20-30	More samples narrow the CI; needed when signal is small relative to noise
Noisy CI environment	20+	Shared CI runners have higher variance; more runs compensate

Never "retry until significant" — rerunning benchmarks until ~ goes away introduces selection bias (p-hacking). If 10 runs show ~, the change is probably not meaningful. Increase run count once and accept the result.

At α=0.05, expect ~5% of benchmarks to randomly report significance with no real change (false positives). This is normal — don't chase them.

Single-File Summary

Analyze variance of a single run without comparison:

benchstat bench.txt

Shows median and confidence interval for each benchmark. Use to:

• Check measurement stability before making code changes
• Identify noisy benchmarks that need more runs or better isolation
• Get a quick summary of current performance

Common Pitfalls

Pitfall	Why it's wrong	Fix
-count=1	Single run has no variance information; benchstat can't compute confidence	Always use -count=6 minimum, prefer -count=10
Running on a laptop on battery	CPU throttles to save power; variance explodes	Plug in, disable power saving, or use a desktop/server
Running with browser/IDE open	Background processes steal CPU cycles; adds noise	Close unnecessary applications, or accept wider CIs
Rerunning until ~ disappears	Selection bias (p-hacking) — you're cherry-picking runs that showed improvement	Run once with high -count, accept the result
Comparing across machines	Different CPUs, memory, OS = incomparable baselines	Same machine, same conditions, both runs
Not interleaving	Systematic bias from thermal throttling, background load drift	Pre-compile both versions with go test -c, alternate runs
Measuring compilation time	go test -bench compiles first; startup overhead varies	Pre-compile with go test -c, run the binary directly
Ignoring wide CI (± >5%)	Results look significant but variance is too high to be trustworthy	Fix the noise first, then compare; or increase -count
Comparing different -count values	Unequal sample sizes bias the comparison	Use the same -count for all inputs

benchstat in CI

See CI Regression Detection for integrating benchstat comparisons into CI pipelines with benchdiff, cob, and gobenchdata.