📚 BusinessExplain 🏠 Home 📊 Analytics
← Back to Skills

Monitoring Observability Mastery

⚙️ DevOps & Infrastructure 21 min read v1.0.0
📋 Contents

Monitoring & Observability Mastery

Three Pillars of Observability

| Pillar | Purpose | Key Tools | Query Language |
|--------|---------|-----------|----------------|
| Metrics | Numeric time-series data, aggregation, alerting | Prometheus, Thanos | PromQL |
| Logs | Discrete events, debugging context | Loki, Elasticsearch, Fluentd | LogQL, KQL |
| Traces | Request flow across services, latency attribution | Jaeger, Tempo, Zipkin | TraceQL |

Interconnection Pattern

Alert (metric) → Correlate (trace ID in log) → Trace (cross-service path) → Log (root cause)

Prometheus

Metric Types

# Counter — only increases; use rate() for throughput
http_requests_total{method="GET", path="/api/v1/users"}

Gauge — point-in-time value; can go up or down

node_memory_available_bytes

Histogram — bucketed observations; enables quantiles

http_request_duration_seconds_bucket{le="0.1"}
http_request_duration_seconds_sum
http_request_duration_seconds_count

Summary — pre-computed quantiles on client side

http_request_duration_seconds{quantile="0.99"}

Rule: Prefer histograms over summaries for aggregation across instances.

PromQL Essentials

# Rate of requests over 5m window
rate(http_requests_total[5m])

99th percentile latency from histogram

histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le, service))

Error ratio (5xx / total)

sum(rate(http_requests_total{status=~"5.."}[5m]))
/
sum(rate(http_requests_total[5m]))

Top 5 services by CPU

topk(5, sum(rate(container_cpu_usage_seconds_total[5m])) by (service))

Predict disk full in 4h based on growth rate

predict_linear(node_filesystem_avail_bytes[1h], 4*3600) < 0

Subquery: 5m average over 1h, then max

max_over_time(avg_over_time(rate(http_requests_total[5m])[1h:]))

Offset for comparing to previous week

rate(http_requests_total[5m] offset 1w)

Recording Rules

# /etc/prometheus/rules/recording.yml
groups:
  - name: service:request_rate
    interval: 30s
    rules:
      - record: service:http_requests:rate5m
        expr: sum(rate(http_requests_total[5m])) by (service, method, status)
        labels:
          team: platform

- record: service:http_error_ratio
        expr: |
          sum(rate(http_requests_total{status=~"5.."}[5m])) by (service)
          /
          sum(rate(http_requests_total[5m])) by (service)

When to use recording rules:

  • Repeatedly used complex expressions

  • Dashboard queries that must be fast

  • Federated aggregation queries

    • Alerting Rules

    # /etc/prometheus/rules/alerts.yml
groups:
  - name: service-alerts
    rules:
      - alert: HighErrorRate
        expr: service:http_error_ratio > 0.05
        for: 5m
        labels:
          severity: critical
          team: platform
        annotations:
          summary: "High error rate on {{ $labels.service }}"
          runbook: "https://runbooks.internal/high-error-rate"
          dashboard: "https://grafana.internal/d/service-overview"

    - alert: DiskSpaceLow
            expr: |
              (node_filesystem_avail_bytes / node_filesystem_size_bytes) < 0.15
              and
              predict_linear(node_filesystem_avail_bytes[1h], 4*3600) < 0
            for: 10m
            labels:
              severity: warning
            annotations:
              summary: "Filesystem {{ $labels.mountpoint }} filling on {{ $labels.instance }}"

    Service Discovery

    # prometheus.yml — Kubernetes SD
scrape_configs:
  - job_name: kubernetes-pods
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: true
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
        action: replace
        target_label: __metrics_path__
        regex: (.+)
      - source_labels: [__meta_kubernetes_namespace]
        action: replace
        target_label: namespace

    - job_name: consul
        consul_sd_configs:
          - server: 'consul.service.consul:8500'
            services: ['api', 'web', 'worker']

    Federation

    # Global Prometheus federating from regional clusters
scrape_configs:
  - job_name: federate-us-east
    honor_labels: true
    metrics_path: /federate
    params:
      'match[]':
        - '{job="kubernetes-pods"}'
        - '{__name__=~"service:.*"}'
    static_configs:
      - targets:
          - prometheus-us-east.internal:9090

    Thanos — Long-Term Storage

    # Thanos Sidecar (runs alongside Prometheus)
#docker run thanosio/thanos:latest sidecar \

     --tsdb.path=/prometheus/data \
    

     --objstore.config-file=bucket.yml \
    

     --prometheus.http-address=http://prometheus:9090 \
    

     --grpc-address=0.0.0.0:10901
    

    bucket.yml
    
type: S3
config:
  bucket: thanos-storage
  endpoint: s3.amazonaws.com
  access_key: "${AWS_ACCESS_KEY_ID}"
  secret_key: "${AWS_SECRET_ACCESS_KEY}"

    Thanos Querier — global view
    
#docker run thanosio/thanos:latest query \

     --grpc-address=0.0.0.0:10901 \
    

     --store=thanos-sidecar-us-east:10901 \
    

     --store=thanos-sidecar-eu-west:10901 \
    

     --store=thanos-store-gateway:10901

    Architecture:

    Prometheus + Sidecar → Object Store (S3/GCS)
                          ↕
                  Thanos Store Gateway (reads historical)
                          ↕
                  Thanos Querier (global view)
                          ↕
                  Grafana (visualization)

    Grafana

    Dashboards as Code (Jsonnet/Grafonnet)

    // dashboards/api-overview.jsonnet
local grafonnet = import 'grafonnet/grafonnet.libsonnet';
local dashboard = grafonnet.dashboard;
local row = grafonnet.row;
local graphPanel = grafonnet.graphPanel;
local singlestat = grafonnet.singlestat;

    dashboard.new(
      'API Service Overview',
      tags=['api', 'production'],
      refresh='30s',
      time_from='now-1h',
    )
    + {
      templating: {
        list: [
          {
            name: 'service',
            type: 'query',
            query: 'label_values(up{job="kubernetes-pods"},service)',
            refresh: 2,
          },
        ],
      },
    }
    + row.new('Throughput')
    + graphPanel.new(
      'Request Rate',
      datasource='Prometheus',
      targets=[
        { expr: 'sum(rate(http_requests_total{service="$service"}[5m])) by (method)' },
      ],
    )
    + singlestat.new(
      'Error Rate',
      datasource='Prometheus',
      targets=[
        { expr: 'service:http_error_ratio{service="$service"}' },
      ],
      thresholds='0.01,0.05',
      colors=['green', 'yellow', 'red'],
    )

    Provisioning via YAML

    # /etc/grafana/provisioning/dashboards/dashboards.yml
apiVersion: 1
providers:
  - name: 'default'
    orgId: 1
    folder: ''
    type: file
    disableDeletion: false
    editable: true
    updateIntervalSeconds: 30
    options:
      path: /var/lib/grafana/dashboards
      foldersFromFilesStructure: true

    # /etc/grafana/provisioning/datasources/datasources.yml
apiVersion: 1
datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true
    jsonData:
      httpMethod: POST
      timeInterval: "15s"
  - name: Loki
    type: loki
    access: proxy
    url: http://loki:3100
  - name: Jaeger
    type: jaeger
    access: proxy
    url: http://jaeger:16686

    Dashboard Variables

    | Variable Type | Use Case | Example Query |
    |--------------|----------|---------------|
    | Query | Dynamic label values | label_values(up, namespace) |
    | Interval | Time granularity | 1m,5m,10m,30m,1h |
    | Custom | Static options | prod,staging,dev |
    | Datasource | Switch backends | Prometheus,Loki |
    | Text box | Free-form input | Used in annotations |

    Grafana Alerting (Unified)

    # Provisioned alert rule
apiVersion: 1
groups:
  - orgId: 1
    name: service-health
    rules:
      - uid: high-error-rate
        title: High Error Rate
        condition: C
        data:
          - refId: A
            relativeTimeRange:
              from: 600
            datasourceUid: prometheus-uid
            model:
              expr: service:http_error_ratio > 0.05
          - refId: B
            relativeTimeRange:
              from: 600
            datasourceUid: prometheus-uid
            model:
              expr: service:http_error_ratio > 0.1
          - refId: C
            relativeTimeRange:
              from: 600
            model:
              type: reduce
              reducer: last
              expression: A
        noDataState: OK
        executionErrorState: Alerting
        for: 5m

    Alertmanager

    Full Configuration

    # alertmanager.yml
global:
  resolve_timeout: 5m
  smtp_smarthost: 'smtp.internal:587'
  smtp_from: 'alerts@internal'
  slack_api_url: 'https://hooks.slack.com/services/XXX'

    templates:
      - '/etc/alertmanager/templates/*.tmpl'
    

    route:
      receiver: 'default-slack'
      group_by: ['alertname', 'cluster', 'namespace']
      group_wait: 30s
      group_interval: 5m
      repeat_interval: 4h
      routes:
        - match:
            severity: critical
          receiver: 'pagerduty-critical'
          group_wait: 10s
          repeat_interval: 1h
        - match:
            team: platform
          receiver: 'platform-slack'
          group_by: ['alertname', 'namespace']
        - match_re:
            alertname: Watchdog|InfoInhibitor
          receiver: 'null'
    

    receivers:
      - name: 'default-slack'
        slack_configs:
          - channel: '#alerts'
            send_resolved: true
            title: '{{ .GroupLabels.alertname }}'
            text: >-
              {{ range .Alerts }}
              Summary: {{ .Annotations.summary }}
              Severity: {{ .Labels.severity }}
              Runbook: {{ .Annotations.runbook }}
              {{ end }}
    

    - name: 'pagerduty-critical'
        pagerduty_configs:
          - routing_key: ''
            severity: '{{ .GroupLabels.severity }}'
    

    - name: 'platform-slack'
        slack_configs:
          - channel: '#platform-alerts'
            send_resolved: true
    

    - name: 'null'
    

    inhibit_rules:
      - source_match:
          severity: 'critical'
        target_match:
          severity: 'warning'
        equal: ['alertname', 'cluster', 'namespace']

    Silences

    # Silence all warnings for a namespace during maintenance
amtool silence add \
  --author="oncall@internal" \
  --comment="Planned maintenance window" \
  --duration=4h \
  namespace=staging severity=warning

    Silence specific alert
    
amtool silence add \
  --author="oncall@internal" \
  --comment="Known issue, tracking in JIRA-12345" \
  --duration=72h \
  alertname=DiskSpaceLow instance=node-03.internal

    List and expire silences
    
amtool silence query
amtool silence expire

    Alert Template

    {{ define "slack.title" }}
[{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ end }}] {{ .GroupLabels.alertname }}
{{ end }}

    {{ define "slack.text" }}
    {{ range .Alerts }}
    Alert: {{ .Annotations.summary }}
    Severity: {{ .Labels.severity }}
    Namespace: {{ .Labels.namespace }}
    Runbook: {{ .Annotations.runbook }}
    Dashboard: {{ .Annotations.dashboard }}
    {{ end }}
    {{ end }}

    Loki

    LogQL

    # Basic label selector
{app="api-service", namespace="production"}

    Filter with line filter
    
{app="api-service"} |= "error" | json | status >= 500

    Extract and aggregate
    
sum(count_over_time({app="api-service"} | json | level="error" [5m])) by (service)

    Pattern parsing (v2 pattern syntax)
    
{app="api-service"} | pattern  - - [] "  "   |
  line_format "{{.method}} {{.path}} {{.status}}"

    Metric queries from logs
    
sum(rate({app="api-service"} | json | status >= 500 [5m])) /
sum(rate({app="api-service"} | json [5m]))

    Unwrap for aggregations
    
avg_over_time({app="api-service"} | json | unwrap latency_ms [5m])

    Deduplication for repeated log lines
    
{app="api-service"} |= "timeout" | dedup

    Structured Logging Best Practices

    // Go — slog structured logging
slog.Info("request completed",
    slog.String("method", r.Method),
    slog.String("path", r.URL.Path),
    slog.Int("status", statusCode),
    slog.Duration("latency", elapsed),
    slog.String("trace_id", traceID),
    slog.String("span_id", spanID),
)

    // Output — JSON for Loki consumption
{
  "ts": "2025-01-15T10:30:00.123Z",
  "level": "info",
  "msg": "request completed",
  "method": "GET",
  "path": "/api/v1/users",
  "status": 200,
  "latency_ms": 42,
  "trace_id": "abc123def456",
  "span_id": "789ghi012",
  "service": "user-api",
  "version": "2.3.1"
}

    Rules for structured logs:

  • Always include trace_id and span_id for correlation

  • Use consistent field names across services

  • Use level not severity (or map consistently)

  • Include deployment metadata: service, version, namespace

  • Never log secrets, tokens, PII

    • Promtail / Fluent-Bit Shipping

    # promtail-config.yml
server:
  http_listen_port: 9080
  grpc_listen_port: 0

    positions:
      filename: /tmp/positions.yaml
    

    clients:
      - url: http://loki:3100/loki/api/v1/push
        tenant_id: production
    

    scrape_configs:
      - job_name: kubernetes-pods
        kubernetes_sd_configs:
          - role: pod
        pipeline_stages:
          - cri: {}
          - json:
              expressions:
                level: level
                msg: msg
                trace_id: trace_id
                service: service
          - labels:
              level:
              service:
          - timestamp:
              source: ts
              format: RFC3339
          - label_drop:
              - filename

    # fluent-bit.conf
[INPUT]
    Name              tail
    Path              /var/log/containers/*.log
    Parser            docker
    Tag               kube.*
    Mem_Buf_Limit     64MB
    Skip_Long_Lines   On

    [FILTER]
        Name              kubernetes
        Match             kube.*
        Kube_URL          https://kubernetes.default:443
        Kube_CA_File      /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        Kube_Token_File   /var/run/secrets/kubernetes.io/serviceaccount/token
    

    [OUTPUT]
        Name              loki
        Match             *
        Host              loki.internal
        Port              3100
        Labels            job=fluent-bit, cluster=prod
        Auto_Kubernetes_Labels On

    Jaeger — Distributed Tracing

    Architecture

    [Instrumented App] → [jaeger-agent:6831 (UDP)] → [jaeger-collector:14268] → [Storage Backend]
                                                                            ↓
                                                                    [jaeger-query:16686]

    Deployment (All-in-One for dev)

    # docker-compose.yml
services:
  jaeger:
    image: jaegertracing/all-in-one:1.54
    environment:
      - COLLECTOR_ZIPKIN_HOST_PORT=:9411
      - COLLECTOR_OTLP_ENABLED=true
      - SPAN_STORAGE_TYPE=elasticsearch
      - ES_SERVER_URLS=http://elasticsearch:9200
    ports:
      - "5775:5775/udp"   # agent zipkin compact
      - "6831:6831/udp"   # agent jaeger compact
      - "6832:6832/udp"   # agent jaeger binary
      - "5778:5778"       # agent configs
      - "16686:16686"     # query UI
      - "14268:14268"     # collector HTTP
      - "14250:14250"     # collector gRPC
      - "4317:4317"       # OTLP gRPC
      - "4318:4318"       # OTLP HTTP

    Sampling Strategies

    // sampling-strategies.json
{
  "default_strategy": {
    "type": "probabilistic",
    "param": 0.1
  },
  "service_strategies": [
    {
      "service": "payment-api",
      "type": "probabilistic",
      "param": 1.0
    },
    {
      "service": "user-api",
      "type": "ratelimiting",
      "param": 10
    }
  ]
}

    Sampling types:

  • Probabilistic: Fraction of traces (e.g., 0.1 = 10%)

  • Rate-limiting: Max N traces per second

  • Adaptive: Auto-adjusts based on traffic (Jaeger backend)

  • Per-endpoint: Different rates per operation (advanced)

    • Span Architecture

    Trace: abc123
├── Span A (gateway, 50ms total)
│   ├── Span B (user-api, 20ms)
│   │   └── Span D (db-query, 15ms)
│   └── Span C (payment-api, 25ms)
│       └── Span E (external-provider, 22ms)

    Key concepts:

  • Trace: End-to-end request journey across all services

  • Span: Single unit of work with operation name, start time, duration

  • Context propagation: trace_id + span_id passed via HTTP headers or message metadata

  • Baggage: Arbitrary key-value pairs propagated across all spans

    • OpenTelemetry

    SDK — Go Example

    import (
    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc"
    "go.opentelemetry.io/otel/sdk/resource"
    sdktrace "go.opentelemetry.io/otel/sdk/trace"
    semconv "go.opentelemetry.io/otel/semconv/v1.24.0"
)

    func initTracer() (*sdktrace.TracerProvider, error) {
        exporter, err := otlptracegrpc.New(
            context.Background(),
            otlptracegrpc.WithEndpoint("otel-collector:4317"),
            otlptracegrpc.WithInsecure(),
        )
        if err != nil {
            return nil, err
        }
    

    res := resource.NewWithAttributes(
            semconv.SchemaURL,
            semconv.ServiceNameKey.String("user-api"),
            semconv.ServiceVersionKey.String("2.3.1"),
            semconv.DeploymentEnvironmentKey.String("production"),
        )
    

    tp := sdktrace.NewTracerProvider(
            sdktrace.WithBatcher(exporter),
            sdktrace.WithResource(res),
            sdktrace.WithSampler(sdktrace.TraceIDRatioBased(0.1)),
        )
        otel.SetTracerProvider(tp)
        return tp, nil
    }

    Collector Configuration

    # otel-collector-config.yml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

    prometheus:
        config:
          scrape_configs:
            - job_name: 'otel-collector'
              scrape_interval: 15s
              static_configs:
                - targets: ['0.0.0.0:8888']
    

    processors:
      batch:
        send_batch_size: 1024
        send_batch_max_size: 2048
        timeout: 5s
    

    memory_limiter:
        check_interval: 1s
        limit_mib: 512
    

    attributes:
        actions:
          - key: env
            value: production
            action: upsert
    

    filter:
        error_mode: ignore
        traces:
          span:
            - 'attributes["http.route"] == "/healthz"'
            - 'attributes["http.route"] == "/readyz"'
    

    exporters:
      otlp/jaeger:
        endpoint: jaeger-collector:4317
        tls:
          insecure: true
    

    prometheusremotewrite:
        endpoint: http://prometheus-remote-write:9090/api/v1/write
    

    loki:
        endpoint: http://loki:3100/loki/api/v1/push
        default_labels_enabled:
          exporter: false
    

    service:
      pipelines:
        traces:
          receivers: [otlp]
          processors: [memory_limiter, filter, batch]
          exporters: [otlp/jaeger]
        metrics:
          receivers: [otlp, prometheus]
          processors: [memory_limiter, attributes, batch]
          exporters: [prometheusremotewrite]
        logs:
          receivers: [otlp]
          processors: [memory_limiter, batch]
          exporters: [loki]

    Auto-Instrumentation

    | Language | Method | Command |
    |----------|--------|---------|
    | Java | JAR agent | java -javaagent:opentelemetry-javaagent.jar -Dotel.service.name=user-api -jar app.jar |
    | Python | Pip package | opentelemetry-instrument --service_name user-api python app.py |
    | Node.js | Require module | node --require @opentelemetry/auto-instrumentations/register app.js |
    | .NET | NuGet package | OTEL_SERVICE_NAME=user-api dotnet run |
    | Go | Requires manual SDK | (no auto-instrumentation — use otel-go SDK) |

    Java auto-instrumentation env vars:

    OTEL_SERVICE_NAME=user-api
    OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4317
    OTEL_TRACES_SAMPLER=parentbased_traceidratio
    OTEL_TRACES_SAMPLER_ARG=0.1
    OTEL_PROPAGATORS=tracecontext,baggage
    OTEL_RESOURCE_ATTRIBUTES=service.version=2.3.1,deployment.environment=production

    SLOs / SLIs / SLAs

    Definitions

    | Term | Definition | Example |
    |------|-----------|---------|
    | SLI | Service Level Indicator — quantitative measure | 99.5% of requests < 200ms |
    | SLO | Service Level Objective — target for an SLI | 99.9% availability per 30-day window |
    | SLA | Service Level Agreement — contractual consequence | 99.9% uptime or credit 10% bill |

    SLO Implementation with Prometheus

    # Availability SLO: 99.9% of requests succeed

  • record: sli:api:availability:rate5m
    • 
  expr: |
    sum(rate(http_requests_total{status!~"5.."}[5m])) by (service)
    /
    sum(rate(http_requests_total[5m])) by (service)

  • record: sli:api:latency_p99:rate5m
    • 
  expr: |
    histogram_quantile(0.99,
      sum(rate(http_request_duration_seconds_bucket[5m])) by (le, service)
    )

    SLO burn rate alerting
    

  • alert: SLOBurnRateFast
    • 
  expr: |
    (
      sum(rate(http_requests_total{status=~"5.."}[1h])) by (service)
      /
      sum(rate(http_requests_total[1h])) by (service)
    ) > (1 - 0.999) * 14.4
  for: 5m
  labels:
    severity: critical

  • alert: SLOBurnRateSlow
    • 
  expr: |
    (
      sum(rate(http_requests_total{status=~"5.."}[6h])) by (service)
      /
      sum(rate(http_requests_total[6h])) by (service)
    ) > (1 - 0.999) * 6
  for: 30m
  labels:
    severity: warning

    Error Budget Calculation

    Monthly error budget (30 days) for 99.9% SLO:
  Total minutes: 30 × 1440 = 43,200 minutes
  Allowed downtime: 43,200 × 0.001 = 43.2 minutes/month

    At 10,000 req/min:
      Total requests: 432,000,000/month
      Allowed failures: 432,000/month
      Error budget per day: 14,400 failed requests

    Error Budget Policy Example

    | Budget Remaining | Action |
    |-----------------|--------|
    | > 50% | Normal operations |
    | 25–50% | Increase test coverage, no feature freezes |
    | 10–25% | Feature freeze, focus on reliability |
    | < 10% | Code freeze, all hands on reliability |
    | 0% | Moratorium on non-reliability changes |

    Incident Management

    Runbook Template

    # Runbook: High Error Rate on {{ service }}

    Overview
    
Brief description of the service and what this alert means.

    Severity
    

  • SEV1: >10% error rate — page on-call immediately
    • 

  • SEV2: 5–10% error rate — page on-call within 15 min
    • 

  • SEV3: 2–5% error rate — notify during business hours
    • 

    Quick Triaging (first 5 minutes)
    

  • Check Grafana dashboard: {{ dashboard_url }}
    • 

  • Identify affected endpoints: rate(http_requests_total{status=~"5..", service="{{ service }}"}[5m])
    • 

  • Check recent deployments: kubectl rollout history deployment/{{ service }}
    • 

  • Rollback if correlated: kubectl rollout undo deployment/{{ service }}
    • 

    Detailed Investigation
    

  • Check logs: {service="{{ service }}"} |= "error" | json
    • 

  • Check traces: Jaeger → search service={{ service }}, status=error
    • 

  • Check dependencies: Are downstream services healthy?
    • 

  • Check infrastructure: CPU, memory, disk on nodes
    • 

    Resolution Steps
    

    Step 1: Rollback
    bash kubectl rollout undo deployment/{{ service }} -n production 
    

    Step 2: Scale
    bash kubectl scale deployment/{{ service }} --replicas=10 -n production 
    

    Step 3: Circuit Break
    
If downstream is flaky, enable circuit breaker via feature flag.

    Communication
    

  • Slack: #incident-{{ service }}
    • 

  • PagerDuty: Auto-created on SEV1
    • 

  • StatusPage: Update if customer-facing
    • 

Created within 48h of resolution.

    Postmortem Template

    # Postmortem: [Incident Title]

    Date: YYYY-MM-DD
    Duration: Xm (from detection to resolution)
    Impact: [quantified — X users, Y requests failed, Z revenue lost]
    Detection: [alert / customer report / internal discovery]
    Responders: [list of on-call and additional responders]
    

    Timeline (all times UTC)
    
| Time | Event |
|------|-------|
| 00:00 | Alert fired |
| 00:02 | On-call acknowledged |
| 00:05 | Root cause identified |
| 00:10 | Mitigation applied |
| 00:15 | Full recovery confirmed |

    Root Cause
    
[Detailed technical explanation of what happened]

    Contributing Factors
    

  • [Factor 1]
    • 

  • [Factor 2]
    • 

    Resolution
    
[What was done to fix and verify]

    Action Items
    
| Action | Owner | Priority | Due Date | Issue |
|--------|-------|----------|----------|-------|
| Add rate limiting | @team-1 | P1 | YYYY-MM-DD | JIRA-123 |
| Improve alert threshold | @team-2 | P2 | YYYY-MM-DD | JIRA-124 |
| Add runbook | @team-3 | P2 | YYYY-MM-DD | JIRA-125 |

    Lessons Learned
    

  • What went well
    • 

  • What went wrong
    • 

  • Where we got lucky
    •

    Blameless Culture Principles

  • Assume good intent — focus on the "what" not the "who"
  • Focus on systems — human error = system design error
  • No punitive action — postmortems are learning tools, not evidence
  • Psychological safety — people must feel safe to report mistakes
  • Action items over blame — every incident must produce improvements
  • Celebrate learning — recognize those who discover and share failure modes

    • On-Call Practices

    On-Call Rotation Setup

    # oncall-config.yml (PagerDuty/Tech-Mate)
schedules:
  - name: platform-primary
    rotation: weekly
    layers:
      - users: [alice, bob, carol, dave]
        rotation_length: 1w
        start: "2025-01-06T09:00:00Z"
    restrictions:
      - start: "09:00"
        end: "17:00"
        day_of_week: [mon, tue, wed, thu, fri]
  - name: platform-secondary
    rotation: daily
    layers:
      - users: [eve, frank, grace]
        rotation_length: 1d

    escalation_policies:
      - name: platform
        rules:
          - delay: 0
            targets: [oncall-primary]
          - delay: 300    # 5 min
            targets: [oncall-secondary]
          - delay: 900    # 15 min
            targets: [engineering-manager]

    Pre-Shift Checklist

  • [ ] Verify PagerDuty/OpsGenie notification settings (push, SMS, call)
  • [ ] Review open incidents and ongoing maintenance windows
  • [ ] Bookmark key dashboards: SLO, infra, service mesh
  • [ ] Confirm runbook access and SSH/kubectl credentials
  • [ ] Set up "on-call" status in Slack/Teams
  • [ ] Review recent changes: deploys, feature flags, config changes
  • [ ] Test alert delivery with a test notification

    • Post-Shift Handoff

    ## On-Call Handoff: [Date]

    Summary
    

  • Total alerts: X
    • 

  • Real incidents: Y
    • 

  • False alarms: Z
    • 

    Incidents
    
| Time | Alert | Action | Status |
|------|-------|--------|--------|
| 02:30 | HighErrorRate | Rolled back deploy | Resolved |
| 06:15 | DiskSpaceLow | Cleaned logs, created PR | Mitigated |

    Follow-ups
    

  • [ ] JIRA-123: Update alert threshold for DiskSpaceLow
    • 

  • [ ] JIRA-124: Investigate memory leak in worker service
    • 

    Notes
    

  • Maintenance window for DB upgrade scheduled tomorrow 02:00 UTC
    • 

  • New deploy (v2.4.0) rolled out at 04:00, watch for latency spikes
    •

    PagerDuty / OpsGenie Integration

    PagerDuty Integration

    # Prometheus Alertmanager → PagerDuty
receivers:
  - name: pagerduty-oncall
    pagerduty_configs:
      - routing_key: ''
        severity: '{{ .GroupLabels.severity }}'
        group: '{{ .GroupLabels.team }}'
        component: '{{ .GroupLabels.service }}'
        class: '{{ .GroupLabels.alertname }}'
        details:
          runbook: '{{ .Annotations.runbook }}'
          dashboard: '{{ .Annotations.dashboard }}'

    PagerDuty Event API v2 (custom integration)
    

    curl -X POST https://events.pagerduty.com/v2/enqueue \
    

      -H 'Content-Type: application/json' \
    

      -d '{
    

        "routing_key": "YOUR_INTEGRATION_KEY",
    

        "event_action": "trigger",
    

        "payload": {
    

          "summary": "High error rate on user-api",
    

          "severity": "critical",
    

          "source": "prometheus",
    

          "component": "user-api",
    

          "group": "platform",
    

          "class": "HighErrorRate",
    

          "custom_details": {
    

            "runbook": "https://runbooks.internal/high-error-rate",
    

            "error_ratio": "0.12",
    

            "affected_namespace": "production"
    

          }
    

        }
    

      }'

    OpsGenie Integration

    # Alertmanager → OpsGenie
receivers:
  - name: opsgenie-oncall
    opsgenie_configs:
      - api_key: ''
        severity: '{{ .GroupLabels.severity }}'
        tags: '{{ .GroupLabels.alertname }},{{ .GroupLabels.team }}'
        description: '{{ .Annotations.summary }}'
        details:
          runbook: '{{ .Annotations.runbook }}'
          dashboard: '{{ .Annotations.dashboard }}'
        priority: '{{ if eq .GroupLabels.severity "critical" }}P1{{ else if eq .GroupLabels.severity "warning" }}P3{{ else }}P5{{ end }}'

    Flowcharts & Decision Diagrams

    Observability Three Pillars Flowchart

    Incident Detected ────────────────────────────────────────────┐
  │                                                             │
  ├─ Alert fires (PagerDuty/OpsGenie)                          │
  │                                                             │
  ├─ Step 1: METRICS ──→ Prometheus/Grafana ──→ CPU? Memory?   │
  │     "Is it resource saturation?"                           │
  │         │                                                  │
  │     No ↓                     Yes ↓                         │
  │   Step 2: LOGS            Scale resources                  │
  │   Loki/ELK ──→ "What error messages?"                      │
  │         │                                                  │
  │   Found the error? ──Yes──→ Fix known pattern              │
  │         │                                                   │
  │        No                                                   │
  │         ↓                                                  │
  │   Step 3: TRACES ──→ Jaeger/Tempo ──→ "Where's the latency?"│
  │   "Which service is the bottleneck?"                       │
  │         │                                                  │
  │   Identify root cause ──→ Apply fix ──→ Verify in metrics │
  └────────────────────────────────────────────────────────────┘

    Alert Routing Decision Tree

    Alert fires ──→ Severity? ─┬─ P1 (critical) ──→ Page on-call immediately (PagerDuty)
                             ├─ P2 (warning) ──→ Slack #incidents + JIRA ticket
                             ├─ P3 (info) ──→ Log to dashboard only
                             └─ Unknown ──→ Route to team Slack for triage

    Monitoring Stack Selection

    Select monitoring stack ─┬─ Cloud-native (AWS)? ──→ CloudWatch + X-Ray + Bedrock
                          ├─ Cloud-native (GCP)? ──→ Cloud Monitoring + Cloud Trace
                          ├─ K8s-first? ──→ Prometheus Operator + Loki + Tempo
                          ├─ Enterprise? ──→ Datadog APM + Logs + Metrics
                          └─ Budget-constrained? ──→ VictoriaMetrics + Grafana LGTM stack

    Testing Strategy for Monitoring

    Alert Testing

  • Alert Unit Tests: Validate PromQL expressions return expected results
    •    # Test alert rule with promtool
   promtool test rules alert_test.yml
   
   # alert_test.yml
   rule_files:
     - alerts.yml
   tests:
     - interval: 1m
       input_series:
         - series: 'up{job="myapp",instance="1"}'
           values: "0 0 0 0 0"
       alert_rule_test:
         - alertname: InstanceDown
           exp_labels:
             severity: critical
           exp_annotations:
             summary: "Instance down"
  • End-to-End Alert Pipeline Test: Use Alertmanager webhook to verify routing
    •    # Verify alert reaches PagerDuty
   receiver: pagerduty-test
   routes:
     - match:
         severity: critical
         test: "true"
       receiver: pagerduty-test
  • Dashboard Testing: Verify Grafana dashboards render without errors
    •    # grr — Grafana dashboard linter
   grr lint dashboards/*.json
   
   # Verify all PromQL queries in dashboard
   grafana-dashboard-testing --url=http://grafana:3000 --dashboard-uid=myapp

    Load Testing with k6 + Prometheus

    // k6-load-test.js — inject metrics during load test
import http from 'k6/http';
import { check } from 'k6';
import { Trend } from 'k6/metrics';

    const apiLatency = new Trend('api_latency_ms');
    

    export let options = {
      stages: [
        { duration: '2m', target: 100 },   // ramp to 100
        { duration: '5m', target: 100 },   // sustain
        { duration: '2m', target: 0 },      // ramp down
      ],
      thresholds: {
        http_req_duration: ['p(99)<500'],
        http_req_failed: ['rate<0.01'],
      },
    };
    

    export default function () {
      let start = Date.now();
      let res = http.get('http://myapp:8080/api/data');
      apiLatency.add(Date.now() - start);
      check(res, { 'status 200': (r) => r.status === 200 });
    }

    Synthetic Monitoring (Probes)

    # Prometheus Blackbox Exporter probe config
modules:
  http_2xx:
    prober: http
    timeout: 5s
    http:
      valid_status_codes: [200]
      method: GET
      headers:
        User-Agent: "synthetic-monitor/1.0"

    Probe scrape config
    
scrape_configs:
  - job_name: 'blackbox-http'
    metrics_path: /probe
    params:
      module: [http_2xx]
    static_configs:
      - targets:
          - https://myapp.com/health
          - https://api.myapp.com/ready
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: blackbox-exporter:9115

    Performance Optimization for Monitoring

  • Prometheus Cardinality Control: Limit high-cardinality labels
    •    # Use metric_relabel_configs to drop high-cardinality labels
   metric_relabel_configs:
     - source_labels: [__name__]
       regex: 'http_request_duration_seconds_bucket'
       action: keep
     - source_labels: [path]
       regex: '/api/v[0-9]+/.*'
       action: keep
     - source_labels: [user_id]
       action: drop  # Never keep user_id as label
  • Loki Label Strategy: Keep labels low-cardinality (<10K unique values)
  • Sampling Strategies: Tail-based sampling in OTEL Collector for production
  • Retention: Prometheus 15d, Thanos long-term, Loki 30d, Tempo 7d
  • Query Optimization: Use recording rules for frequently-queried dashboards
  • Resource Limits: Set --storage.tsdb.retention.size=50GB and query.max-samples=50000000

    • Best Practices

    Metrics

  • USE Method (Utilization, Saturation, Errors) for infrastructure
    • - Utilization: % of resource busy - Saturation: queue depth / wait time - Errors: error count / error rate
  • RED Method (Rate, Errors, Duration) for services
    • - Rate: requests/sec - Errors: failed requests/sec or % - Duration: latency distributions (p50, p95, p99)
  • Label cardinality ≤ 10 values per label; avoid unbounded labels (user IDs, request IDs)
  • Use recording rules for any query appearing on dashboards
  • Set scrape_interval: 15s minimum; 5s only for critical latency-sensitive services

    • Logging

  • Ship JSON structured logs — never plain text in production
  • Include trace_id + span_id in every log line for correlation
  • Set retention per severity: errors 90d, info 30d, debug 7d
  • Use sampling for high-volume non-error logs
  • Never log secrets, tokens, PII (use redaction pipelines)

    • Tracing

  • Always propagate trace context (W3C traceparent header)
  • Sample at the edge (gateway/service mesh), not mid-pipeline
  • Add trace_id to error logs and alert annotations
  • Use tail-based sampling for error traces (keep 100% of errors)
  • Set reasonable span limits (Jaeger default: max 1,000 spans/trace)

    • Alerting

  • Alert on symptoms, not causes (user impact, not CPU usage)
  • Use for: clause to avoid flapping (5m for critical, 10m for warning)
  • Every alert must have a runbook link — no runbook = delete the alert
  • Aim for < 5 pages per on-call shift; tune thresholds aggressively
  • Use multi-window multi-burn-rate alerts for SLOs (Google SRE approach)

    • Common Pitfalls

    | Pitfall | Impact | Fix |
    |---------|--------|-----|
    | High cardinality labels | Prometheus OOM, slow queries | Limit label values, use recording rules to aggregate |
    | No for: in alerts | Alert fatigue from flapping | Always use for: 5m minimum |
    | Alerting on CPU/memory | Noisy, low-signal alerts | Alert on SLO burn rate instead |
    | Unbounded log volumes | Loki/storage cost explosion | Use sampling, set retention policies |
    | Missing trace propagation | Broken traces across services | Use OpenTelemetry auto-instrumentation |
    | Single point of failure in monitoring | Can't monitor monitoring | Use Thanos/redundant Prometheus, external health checks |
    | Stale dashboards | Misleading decisions | Review dashboards quarterly, remove unused panels |
    | Skip postmortems | Recurring incidents | Mandate postmortem within 48h for all SEV1/2 |
    | No silence hygiene | Missed alerts after maintenance | Auto-expire silences, audit weekly |
    | Alerting on raw rates | Noisy during low traffic | Use ratio queries (errors/total) instead |

    Quick Reference Commands

    # Prometheus
promtool check config prometheus.yml
promtool check rules rules/*.yml
promtool query instant http://prometheus:9090 'up == 0'
curl -s http://prometheus:9090/api/v1/targets | jq '.data.activeTargets[] | select(.health!="up")'

    Alertmanager
    
amtool check-config alertmanager.yml
amtool alert query --output=extended
amtool silence add --author="ops" --duration=2h alertname=DiskSpaceLow

    Loki
    
logcli query '{app="api"} |= "error"' --limit=100 --since=1h
logcli labels --limit=50 app
logcli series '{app="api"}' --from=2025-01-01T00:00:00Z

    Jaeger
    
curl -s http://jaeger:16686/api/traces?service=user-api&limit=20 | jq '.data[].traceID'

    Thanos
    
thanos tools bucket ls --objstore.config-file=bucket.yml
thanos tools bucket verify --objstore.config-file=bucket.yml

    Key Ports Quick Reference

    | Component | Port | Purpose |
    |-----------|------|---------|
    | Prometheus | 9090 | Web UI / API |
    | Prometheus | 9091 | Federation push |
    | Alertmanager | 9093 | Web UI |
    | Alertmanager | 9094 | Cluster mesh |
    | Grafana | 3000 | Web UI |
    | Loki | 3100 | API / push |
    | Promtail | 9080 | Metrics |
    | Jaeger UI | 16686 | Query UI |
    | Jaeger Collector | 14268 | HTTP spans |
    | Jaeger Collector | 14250 | gRPC spans |
    | Jaeger Agent | 6831 | UDP compact |
    | OTel Collector | 4317 | OTLP gRPC |
    | OTel Collector | 4318 | OTLP HTTP |
    | Thanos Query | 10902 | Web UI |
    | Thanos Sidecar | 10901 | gRPC store API |

    💬 Ask about Monitoring Observability Mastery