Skip to content

Concepts

OpenTelemetry is a foundational framework for observability in distributed systems, enabling developers and SRE teams to collect and analyze traces, metrics, and logs. These three pillars of observability work together to provide visibility into system behavior, performance, and reliability. Understanding their definitions and relationships is critical for implementing effective monitoring and debugging strategies.

Traces: Tracking Request Flow Across Services

A trace represents the end-to-end journey of a single request through a distributed system. It is composed of spans, which are individual operations (e.g., HTTP calls, database queries) that make up the request’s lifecycle.

Key Concepts:

  • Trace ID: A unique identifier for an entire trace.
  • Span: A logical unit of work (e.g., a function call) with start and end timestamps, and metadata (e.g., status, tags).
  • Context Propagation: Spans are linked via parent-child relationships, enabling context (e.g., headers) to propagate across service boundaries.

Example:
A user request to a web app might generate a trace with spans for the frontend, backend service, and database.

# Example OpenTelemetry trace exported to Jaeger  
{
  "trace_id": "123e4567-e89b-12d3-a456-426614174000",
  "spans": [
    {
      "name": "HTTP Request /api/data",
      "kind": "CLIENT",
      "timestamp": 1620000000000,
      "duration": 15000000
    },
    {
      "name": "Database Query",
      "kind": "SERVER",
      "timestamp": 1620000000000,
      "duration": 5000000
    }
  ]
}

Metrics: Quantifying System Behavior

Metrics are aggregated numerical values that represent system performance, resource usage, or operational health. They are typically collected at regular intervals and used for alerting, capacity planning, and trend analysis.

Key Concepts:

  • Metric Name: A human-readable identifier (e.g., http_request_duration_seconds).
  • Dimensions: Labels (e.g., method="GET", status_code="200") to categorize data.
  • Aggregation: Metrics are summed, averaged, or counted over time windows (e.g., 1-minute intervals).

Example:
A metric tracking HTTP request latency:

# Prometheus query to calculate average latency  
avg_over_time(http_request_duration_seconds{job="my-service"}[1m])

Logs: Capturing Detailed Events

Logs are unstructured textual records of events, errors, or debug information generated by applications or infrastructure. They provide granular context for troubleshooting specific issues.

Key Concepts:

  • Log Level: Indicates severity (e.g., INFO, ERROR, DEBUG).
  • Structured Logging: Logs are often enriched with metadata (e.g., {"level": "ERROR", "message": "Database connection failed", "service": "auth-service"}).
  • Correlation: Logs are tied to traces and metrics via identifiers like trace_id or span_id.

Example:
A log entry from a service:

{
  "timestamp": "2023-10-05T12:00:00Z",
  "level": "ERROR",
  "message": "Failed to connect to database",
  "trace_id": "123e4567-e89b-12d3-a456-426614174000",
  "service": "order-service"
}

Relationship Between Traces, Metrics, and Logs

These three data types complement each other:
1. Traces provide context for the flow of requests and dependencies between services.
2. Metrics summarize performance and resource usage at scale (e.g., error rates, latency).
3. Logs offer detailed, contextual insights into specific events or errors.

Together, they enable observability by answering questions like:
- What happened? (Logs)
- Where did it happen? (Traces)
- How often did it happen? (Metrics)

Tools and Integration

OpenTelemetry integrates with tools like:
- Prometheus for metrics collection and storage.
- Grafana for visualizing metrics and logs.
- Jaeger or Zipkin for trace visualization.
- ELK Stack (Elasticsearch, Logstash, Kibana) for log analysis.

Example: Exporting metrics to Prometheus:

# metrics_exporter_config.yaml
metrics:
  exporter:
    type: prometheus
  prometheus:
    endpoint: "0.0.0.0:9090"

Key takeaways

  • Traces track request flow across services, enabling root-cause analysis.
  • Metrics quantify system behavior for monitoring and alerting.
  • Logs provide detailed, contextual insights into specific events.
  • Together, they form a holistic observability strategy for distributed systems.
  • OpenTelemetry unifies these data types into a single observability pipeline.