199 lines
7.3 KiB
Markdown
199 lines
7.3 KiB
Markdown
# Distributed Tracing with OpenTelemetry
|
|
|
|
→ See `samber/cc-skills-golang@golang-context` skill for propagating context across service boundaries. → See `samber/cc-skills-golang@golang-samber-oops` skill for structured errors with stack traces in spans.
|
|
|
|
When using the OpenTelemetry Go SDK, refer to the library's official documentation for up-to-date API signatures and examples.
|
|
|
|
## Why Tracing
|
|
|
|
When a request crosses multiple services, logs from each service are isolated. Tracing connects them: a single trace shows the full request path with timing for every operation. This is how you answer "why was this request slow?" in a microservices architecture.
|
|
|
|
## OTel SDK Setup
|
|
|
|
Set up the TracerProvider early in your application. On new projects, do this first — then add spans everywhere incrementally.
|
|
|
|
```go
|
|
import (
|
|
"go.opentelemetry.io/otel"
|
|
"go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc"
|
|
"go.opentelemetry.io/otel/sdk/resource"
|
|
sdktrace "go.opentelemetry.io/otel/sdk/trace"
|
|
semconv "go.opentelemetry.io/otel/semconv/v1.26.0"
|
|
)
|
|
|
|
func initTracer(ctx context.Context) (func(), error) {
|
|
exporter, err := otlptracegrpc.New(ctx)
|
|
if err != nil {
|
|
return nil, fmt.Errorf("creating OTLP exporter: %w", err)
|
|
}
|
|
|
|
res, err := resource.New(ctx,
|
|
resource.WithAttributes(
|
|
semconv.ServiceNameKey.String("my-service"),
|
|
semconv.ServiceVersionKey.String("1.0.0"),
|
|
),
|
|
)
|
|
if err != nil {
|
|
return nil, fmt.Errorf("creating resource: %w", err)
|
|
}
|
|
|
|
tp := sdktrace.NewTracerProvider(
|
|
sdktrace.WithBatcher(exporter),
|
|
sdktrace.WithResource(res),
|
|
)
|
|
otel.SetTracerProvider(tp)
|
|
|
|
shutdown := func() {
|
|
_ = tp.Shutdown(context.Background())
|
|
}
|
|
return shutdown, nil
|
|
}
|
|
```
|
|
|
|
## Creating Spans
|
|
|
|
Every meaningful operation should have a span. Think of spans as the building blocks of a trace — they show where time was spent.
|
|
|
|
```go
|
|
import "go.opentelemetry.io/otel"
|
|
|
|
var tracer = otel.Tracer("myapp/order-service")
|
|
|
|
func (s *OrderService) Create(ctx context.Context, req CreateOrderRequest) (*Order, error) {
|
|
ctx, span := tracer.Start(ctx, "OrderService.Create")
|
|
defer span.End()
|
|
|
|
// Add attributes that help with debugging
|
|
span.SetAttributes(
|
|
attribute.String("order.payment_method", req.PaymentMethod),
|
|
attribute.Float64("order.amount", req.Amount),
|
|
)
|
|
|
|
order, err := s.repo.Insert(ctx, req.ToOrder())
|
|
if err != nil {
|
|
span.RecordError(err)
|
|
span.SetStatus(codes.Error, err.Error())
|
|
return nil, fmt.Errorf("inserting order: %w", err)
|
|
}
|
|
|
|
return order, nil
|
|
}
|
|
|
|
func (r *OrderRepo) Insert(ctx context.Context, order Order) (*Order, error) {
|
|
ctx, span := tracer.Start(ctx, "OrderRepo.Insert")
|
|
defer span.End()
|
|
|
|
_, err := r.db.ExecContext(ctx, "INSERT INTO orders ...", order.ID)
|
|
if err != nil {
|
|
span.RecordError(err)
|
|
span.SetStatus(codes.Error, err.Error())
|
|
return nil, fmt.Errorf("exec insert: %w", err)
|
|
}
|
|
return &order, nil
|
|
}
|
|
```
|
|
|
|
**Where to add spans** — spans MUST be created for:
|
|
|
|
- Every service method (business logic layer)
|
|
- Every database query
|
|
- Every external API call
|
|
- Every message queue publish/consume
|
|
- Any operation that takes measurable time or could fail
|
|
|
|
## HTTP Middleware with `otelhttp`
|
|
|
|
Automatically creates spans for incoming and outgoing HTTP requests:
|
|
|
|
```go
|
|
import "go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"
|
|
|
|
// Incoming requests — wrap your handler
|
|
mux.Handle("/orders", otelhttp.NewHandler(orderHandler, "CreateOrder"))
|
|
|
|
// Outgoing requests — HTTP clients MUST use otelhttp for automatic span propagation
|
|
client := &http.Client{
|
|
Transport: otelhttp.NewTransport(http.DefaultTransport),
|
|
}
|
|
```
|
|
|
|
## Span Status and Recording Errors
|
|
|
|
```go
|
|
import (
|
|
"go.opentelemetry.io/otel/codes"
|
|
)
|
|
|
|
// On success — no need to set status (Unset is fine)
|
|
|
|
// On error — MUST call both RecordError() and SetStatus(Error)
|
|
if err != nil {
|
|
span.RecordError(err)
|
|
span.SetStatus(codes.Error, "operation failed")
|
|
return err
|
|
}
|
|
```
|
|
|
|
## Structured Errors with `samber/oops`
|
|
|
|
Standard Go errors lose critical debugging information: there's no stack trace, no structured context, and no way to attach request-scoped metadata. When an error surfaces in a trace, you see `"connection refused"` but not where it originated or which user/tenant was affected.
|
|
|
|
[`samber/oops`](https://github.com/samber/oops) is a drop-in error library that fills these gaps. Every `oops` error carries a stack trace, structured attributes, and integrates naturally with both OpenTelemetry spans and `slog`:
|
|
|
|
```go
|
|
import "github.com/samber/oops"
|
|
|
|
func (s *OrderService) Create(ctx context.Context, req CreateOrderRequest) (*Order, error) {
|
|
ctx, span := tracer.Start(ctx, "OrderService.Create")
|
|
defer span.End()
|
|
|
|
order, err := s.repo.Insert(ctx, req.ToOrder())
|
|
if err != nil {
|
|
// oops wraps the error with stack trace, structured context, and error code
|
|
return nil, oops.
|
|
In("order-service").
|
|
Code("order_insert_failed").
|
|
With("order_id", req.OrderID).
|
|
With("user_id", req.UserID).
|
|
Wrapf(err, "inserting order")
|
|
}
|
|
|
|
return order, nil
|
|
}
|
|
```
|
|
|
|
When this error is logged or recorded on a span, you get the full stack trace, the domain (`order-service`), an error code (`order_insert_failed`), and structured attributes (`order_id`, `user_id`) — all machine-parseable and searchable in your observability platform.
|
|
|
|
`oops` errors work with `span.RecordError()`, `errors.Is`/`errors.As`, and `slog` — see the `samber/cc-skills-golang@golang-error-handling` and `samber/cc-skills-golang@golang-samber-oops` skills for full usage patterns.
|
|
|
|
## Trace Sampling
|
|
|
|
In high-throughput services, tracing every request is expensive. Use sampling to control the volume:
|
|
|
|
```go
|
|
tp := sdktrace.NewTracerProvider(
|
|
// Sample 10% of traces in production
|
|
sdktrace.WithSampler(sdktrace.TraceIDRatioBased(0.1)),
|
|
sdktrace.WithBatcher(exporter),
|
|
sdktrace.WithResource(res),
|
|
)
|
|
```
|
|
|
|
For more nuanced control, use `sdktrace.ParentBased()` to respect the parent's sampling decision — this keeps traces complete across service boundaries.
|
|
|
|
## Cost of Tracing
|
|
|
|
Tracing can be one of the most expensive observability signals. Every span generates data that must be serialized, transmitted, stored, and indexed. In a microservices architecture, a single user request can produce dozens or hundreds of spans across services.
|
|
|
|
**Cost factors:**
|
|
|
|
- **Span volume** — a service handling 10k req/s with 5 spans per request generates 50k spans/s. At 100% sampling, this is enormous.
|
|
- **Span attributes** — each attribute adds to the payload size. Large attributes (request/response bodies) multiply cost.
|
|
- **Storage and indexing** — tracing backends (Jaeger, Tempo, Datadog) charge by volume. Unsampled traces can easily become the largest line item in your observability bill.
|
|
|
|
**Mitigation:**
|
|
|
|
- Use sampling (see above) — start with 10% (`TraceIDRatioBased(0.1)`) and adjust based on traffic volume and budget
|
|
- For high-throughput services, consider head-based sampling (decide at trace start) or tail-based sampling (decide after the trace completes, keeping only interesting traces like errors or slow requests)
|
|
- Avoid attaching large payloads as span attributes — log them instead and correlate via trace_id
|