XingfenD/teamai-test
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
65c04def51fa33e310360b85e79c53b460c81f7c
teamai-test/.teamai/skills/common/golang-benchmark/references/prometheus-go-metrics.md
T

11 KiB
Raw Blame History

Prometheus Go Runtime Metrics Reference

Complete listing of Go runtime metrics actually exposed as Prometheus metrics by prometheus/client_golang library.

Important Clarification

runtime/metrics are NOT Prometheus metrics. They're Go runtime data structures.

The Prometheus Go client library (prometheus/client_golang) selectively converts some runtime/metrics into Prometheus format. By default, it exposes only the traditional go_memstats_* and go_gc_* metrics to keep cardinality low.

This document lists only Prometheus metrics (the ones you actually scrape from /metrics endpoint).

Quick Reference

Metrics with Labels

Metric Label Values
go_gc_duration_seconds quantile 0, 0.25, 0.5, 0.75, 1
go_info version e.g., "go1.21.3"

All Other Metrics

All other metrics have no labels.

Default Go Metrics (Always Exposed)

These are exposed by default by prometheus/client_golang.

Memory Allocation

Metric Type Description
go_memstats_alloc_bytes gauge Current bytes allocated on heap
go_memstats_alloc_bytes_total counter Cumulative bytes allocated
go_memstats_sys_bytes gauge Total bytes requested from OS

Heap State

Metric Type Description
go_memstats_heap_alloc_bytes gauge Allocated heap bytes
go_memstats_heap_idle_bytes gauge Idle heap bytes
go_memstats_heap_inuse_bytes gauge Heap bytes in use
go_memstats_heap_objects gauge Count of heap objects
go_memstats_heap_released_bytes gauge Heap bytes released to OS
go_memstats_heap_sys_bytes gauge Heap bytes reserved from OS

Stack and Metadata

Metric Type Description
go_memstats_stack_inuse_bytes gauge Stack in-use bytes
go_memstats_stack_sys_bytes gauge Stack reserved bytes
go_memstats_mspan_inuse_bytes gauge Mspan in-use bytes
go_memstats_mspan_sys_bytes gauge Mspan reserved bytes
go_memstats_mcache_inuse_bytes gauge Mcache in-use bytes
go_memstats_mcache_sys_bytes gauge Mcache reserved bytes
go_memstats_other_sys_bytes gauge Other runtime bytes
go_memstats_gc_sys_bytes gauge GC internal bytes
go_memstats_buck_hash_sys_bytes gauge Profiling bucket hash table bytes

Allocation and Free Counters

Metric Type Description
go_memstats_mallocs_total counter Total malloc calls
go_memstats_frees_total counter Total free calls

GC Configuration and Timing

Metric Type Description
go_gc_gogc_percent gauge GOGC target percentage
go_gc_gomemlimit_bytes gauge GOMEMLIMIT soft memory limit
go_memstats_last_gc_time_seconds gauge Last GC end time (Unix timestamp)
go_memstats_next_gc_bytes gauge Heap size target for next GC

GC Pause Duration (with labels)

Metric Type Labels Description
go_gc_duration_seconds summary quantile (0, 0.25, 0.5, 0.75, 1) GC pause durations with quantiles
go_gc_duration_seconds_count counter — GC pause count
go_gc_duration_seconds_sum counter — GC pause total time

Runtime State

Metric Type Description
go_goroutines gauge Current goroutine count
go_threads gauge Current OS thread count
go_sched_gomaxprocs_threads gauge Current GOMAXPROCS value

Version Information (with labels)

Metric Type Labels Description
go_info gauge version Go version string

Optional Go Metrics (Opt-in, Go 1.17+)

Enable via:

prometheus.NewRegistry().MustRegister(
    collectors.NewGoCollector(
        collectors.WithGoCollectorRuntimeMetrics(
            collectors.MetricsAll,
        ),
    ),
)

GC Cycles

Metric Type Description
go_gc_cycles_automatic_gc_cycles_total counter Automatic GC cycles (heap growth)
go_gc_cycles_forced_gc_cycles_total counter Forced GC cycles (runtime.GC())

Additional Heap Metrics

Metric Type Description
go_gc_heap_allocs_bytes_total counter Cumulative heap allocations (bytes)
go_gc_heap_allocs_objects_total counter Cumulative heap allocations (count)
go_gc_heap_frees_bytes_total counter Cumulative heap frees (bytes)
go_gc_heap_frees_objects_total counter Cumulative heap frees (count)
go_gc_heap_goal_bytes gauge Heap size target for next GC
go_gc_heap_live_bytes gauge Live heap bytes
go_gc_heap_objects_objects gauge Total heap objects count

GC Pauses Distribution

Metric Type Description
go_gc_pauses_seconds distribution GC pause durations

CPU Classes

Metric Type Description
go_cpu_classes_gc_mark_assist_cpu_seconds_total counter GC mark assist CPU time
go_cpu_classes_gc_mark_dedicated_cpu_seconds_total counter GC dedicated workers CPU time
go_cpu_classes_gc_mark_idle_cpu_seconds_total counter GC idle workers CPU time
go_cpu_classes_gc_pause_cpu_seconds_total counter GC pause CPU time
go_cpu_classes_gc_total_cpu_seconds_total counter Total GC CPU time
go_cpu_classes_idle_cpu_seconds_total counter Idle CPU time
go_cpu_classes_scavenge_assist_cpu_seconds_total counter Scavenger assist CPU time
go_cpu_classes_scavenge_background_cpu_seconds_total counter Background scavenger CPU time
go_cpu_classes_scavenge_total_cpu_seconds_total counter Total scavenger CPU time
go_cpu_classes_total_cpu_seconds_total counter Total CPU time (all classes)
go_cpu_classes_user_cpu_seconds_total counter User-mode CPU time

Memory Classes

Metric Type Description
go_memory_classes_heap_free_bytes gauge Free heap memory
go_memory_classes_heap_objects_bytes gauge Allocated heap objects
go_memory_classes_heap_released_bytes gauge Heap released memory
go_memory_classes_heap_stacks_bytes gauge Stack memory
go_memory_classes_heap_unused_bytes gauge Unused heap
go_memory_classes_metadata_mcache_free_bytes gauge Free mcache memory
go_memory_classes_metadata_mcache_inuse_bytes gauge In-use mcache memory
go_memory_classes_metadata_mspan_free_bytes gauge Free mspan memory
go_memory_classes_metadata_mspan_inuse_bytes gauge In-use mspan memory
go_memory_classes_other_bytes gauge Other memory
go_memory_classes_total_bytes gauge Total memory

Scheduler Metrics

Metric Type Description
go_sched_goroutines_running_goroutines gauge Running goroutines
go_sched_goroutines_runnable_goroutines gauge Runnable goroutines waiting
go_sched_goroutines_goroutines gauge Current total goroutines
go_sched_goroutines_created_goroutines_total counter Total goroutines ever created
go_sched_goroutines_waiting_goroutines gauge Goroutines waiting (not runnable)
go_sched_latencies_seconds distribution Goroutine scheduling latency
go_sched_pauses_stopping_gc_seconds distribution STW pause time (GC stop)
go_sched_pauses_stopping_other_seconds distribution STW pause time (other stop)
go_sched_pauses_total_gc_seconds distribution Total GC pause duration
go_sched_pauses_total_other_seconds distribution Total other pause duration
go_sched_threads_total_threads counter Total OS threads ever created
go_sync_mutex_wait_total_seconds_total counter Total time goroutines waited on mutex

CGO Metrics

Metric Type Description
go_cgo_go_to_c_calls_total counter Total calls from Go to C

Process Metrics

Exposed by Prometheus process collector (not Go-specific):

CPU and Memory

Metric Type Description
process_cpu_seconds_total counter Total CPU time (user + system)
process_resident_memory_bytes gauge RSS (physical memory used)
process_virtual_memory_bytes gauge Virtual memory allocated
process_virtual_memory_max_bytes gauge Maximum virtual memory allowed

File Descriptors

Metric Type Description
process_open_fds gauge Open file descriptors
process_max_fds gauge Maximum file descriptors allowed

Process Information

Metric Type Description
process_start_time_seconds gauge Process start time (Unix timestamp)

Page Faults

Metric Type Description
process_page_faults_total counter Total page faults
process_page_faults_minor_total counter Minor page faults
process_page_faults_major_total counter Major page faults

Common PromQL Queries

Memory Leak Detection

# Current heap allocation (should be stable under constant load)
go_memstats_alloc_bytes

# Live heap bytes (optional metric)
go_gc_heap_live_bytes

# Heap growth rate
rate(go_memstats_alloc_bytes_total[5m])

GC Pressure

# Worst-case GC pause (quantile 1 = max)
go_gc_duration_seconds{quantile="1"}

# Average GC pause
rate(go_gc_duration_seconds_sum[5m]) / rate(go_gc_duration_seconds_count[5m])

# GC frequency (cycles per second)
rate(go_gc_duration_seconds_count[5m])

Goroutine Leaks

# Current goroutine count
go_goroutines

# Goroutine growth (leak indicator)
delta(go_goroutines[1h])

CPU Usage

# Total CPU time consumed
rate(process_cpu_seconds_total[5m])

# CPU utilization ratio (0-1)
rate(process_cpu_seconds_total[5m]) / <GOMAXPROCS>

File Descriptor Leaks

# FD growth
delta(process_open_fds[1h])

# FD saturation ratio
process_open_fds / process_max_fds

→ See samber/cc-skills@promql-cli skill for executing these queries directly against your Prometheus instance from the CLI.

References

Reference in New Issue View Git Blame Copy Permalink