Observability

Observability in Go Micro spans logs, metrics, and traces. The goal is rapid insight into service behavior with minimal configuration.

Observability

Observability in Go Micro spans logs, metrics, and traces. The goal is rapid insight into service behavior with minimal configuration.

Core Principles

  1. Structured Logs – Machine-parsable, leveled output
  2. Metrics – Quantitative trends (counters, gauges, histograms)
  3. Traces – Request flows across service boundaries
  4. Correlation – IDs flowing through all three signals

Logging

The default logger can be replaced. Use env vars to adjust level:

MICRO_LOG_LEVEL=debug go run main.go

Recommended fields:

  • service – service name
  • version – release identifier
  • trace_id – propagated context id
  • span_id – current operation id

Metrics

Patterns:

  • Emit counters for request totals
  • Use histograms for latency
  • Track error rates per endpoint

Example (pseudo-code):

// Wrap handler to record metrics
func MetricsWrapper(fn micro.HandlerFunc) micro.HandlerFunc {
    return func(ctx context.Context, req micro.Request, rsp interface{}) error {
        start := time.Now()
        err := fn(ctx, req, rsp)
        latency := time.Since(start)
        metrics.Inc("requests_total", req.Endpoint(), errorLabel(err))
        metrics.Observe("request_latency_seconds", latency, req.Endpoint())
        return err
    }
}

Tracing

Distributed tracing links calls across services.

Propagation strategy:

  • Extract trace context from incoming headers
  • Inject into outgoing RPC calls/broker messages
  • Create spans per handler and client call

Local Development Strategy

Start with only structured logs. Add metrics when operating multiple services. Introduce tracing once debugging multi-hop latency or failures.

Roadmap (Planned Enhancements)

  • Native OpenTelemetry exporter helpers
  • Automatic handler/client wrapping for spans
  • Default correlation IDs across broker messages

Deployment Recommendations

ScaleSuggested Stack
DevConsole logs only
StagingLogs + basic metrics (Prometheus)
Prod (basic)Logs + metrics + sampling traces
Prod (complex)Full tracing + profiling + anomaly detection

Troubleshooting

SymptomCauseFix
Missing trace IDs in logsContext not propagatedEnsure wrappers add IDs
Metrics server emptyEndpoint not scrapedVerify Prometheus config
High cardinality metricsDynamic labelsReduce labeled dimensions