Observability

The cheat table: Azure Monitor → AWS observability

CloudWatch Metrics

CloudWatch's data model has four nouns. Memorize them and the AWS console stops feeling random.

Auto-emitted vs custom metrics

Most AWS services emit a set of metrics for free, at 1-minute resolution. Anything you emit yourself is a custom metric and costs $0.30 per metric per month (where a "metric" is one unique namespace + name + dimension combo).

Retention - the surprisingly long tail

CloudWatch metrics are kept forever, but resolution decays over time. This is similar to Azure Monitor's rollup behavior but with different cliffs.

Side-by-side: emit a custom metric

# Custom metrics in Azure Monitor
# Usually emitted via App Insights SDK,
# or the Application Insights TrackMetric API:

from applicationinsights import TelemetryClient

tc = TelemetryClient("<ikey>")
tc.track_metric(
  name="queue_depth",
  value=42,
  properties={"queue": "reports-intake"},
)
tc.flush()

# Or write the metric via Azure Monitor
# Metrics REST API (POST .../metrics).

# Option A: classic PutMetricData (counted)
aws cloudwatch put-metric-data \
  --namespace "Compass/Reports" \
  --metric-name QueueDepth \
  --value 42 \
  --unit Count \
  --dimensions Queue=reports-intake

# Option B: EMF - emit via a log line, no separate metric call
# (this gets parsed by CloudWatch into a real metric):
echo '{
  "_aws": {
    "Timestamp": 1716331200000,
    "CloudWatchMetrics": [{
      "Namespace": "Compass/Reports",
      "Dimensions": [["Queue"]],
      "Metrics": [{"Name": "QueueDepth", "Unit": "Count"}]
    }]
  },
  "Queue": "reports-intake",
  "QueueDepth": 42
}'

CloudWatch Logs & Logs Insights

The KQL-to-Insights translation

If you spend most of your day with KQL, here's the shortest possible translation table. Internalize the verbs first, the syntax second.

Side-by-side: count errors per 5 minutes

AppTraces
| where TimeGenerated > ago(1h)
| where SeverityLevel >= 3
| where AppRoleName == "gfn-reports"
| summarize errors = count()
    by bin(TimeGenerated, 5m)
| order by TimeGenerated asc

fields @timestamp, @message
| filter level = "ERROR"
| filter service = "gfn-reports"
| stats count(*) as errors by bin(5m)
| sort @timestamp asc

Structured JSON is the unlock

Logs Insights auto-parses any line that's valid JSON. Plain string lines work, but you'll spend your life writing parse rules. If you control the application, emit JSON. If you don't, write the parse rule once and save it as a query.

{
  "timestamp": "2026-05-22T14:32:01Z",
  "level": "ERROR",
  "service": "gfn-reports",
  "route": "/reports/aggregate",
  "user_id": "u_42",
  "duration_ms": 1820,
  "error": "DynamoDB throttled"
}

With that shape, filter level = "ERROR" | stats avg(duration_ms) by route works out of the box. Every JSON key is a queryable field.

CloudWatch Alarms

An alarm has three states: OK, ALARM, and INSUFFICIENT_DATA. State transitions can each have separate actions - which is a feature and a foot-gun. There are three kinds of alarms:

Side-by-side: alert on error rate

# 1. Action Group (who gets paged)
az monitor action-group create \
  --name oncall-ag \
  --resource-group rg-obs \
  --action email primary sre@example.com

# 2. Metric alert rule
az monitor metrics alert create \
  --name "gfn-reports-errors-high" \
  --resource-group rg-obs \
  --scopes "<function-app-id>" \
  --condition "avg exceptions/server > 10" \
  --window-size 5m \
  --evaluation-frequency 1m \
  --action oncall-ag

# 1. SNS topic (the fan-out)
aws sns create-topic --name oncall-topic
aws sns subscribe \
  --topic-arn arn:aws:sns:us-east-1:123:oncall-topic \
  --protocol email \
  --notification-endpoint sre@example.com

# 2. CloudWatch alarm
aws cloudwatch put-metric-alarm \
  --alarm-name gfn-reports-errors-high \
  --namespace AWS/Lambda \
  --metric-name Errors \
  --dimensions Name=FunctionName,Value=gfn-reports-worker \
  --statistic Sum \
  --period 60 \
  --evaluation-periods 3 \
  --threshold 5 \
  --comparison-operator GreaterThanThreshold \
  --treat-missing-data notBreaching \
  --alarm-actions arn:aws:sns:us-east-1:123:oncall-topic

{
  "AlarmName": "api-latency-high",
  "Namespace": "AWS/ApiGateway",
  "MetricName": "Latency",
  "Dimensions": [
    { "Name": "ApiName", "Value": "reports-api" }
  ],
  "Statistic": "Sum",
  "Period": 60,
  "EvaluationPeriods": 1,
  "Threshold": 5000,
  "ComparisonOperator": "GreaterThanThreshold",
  "AlarmActions": ["arn:aws:sns:us-east-1:123:oncall-topic"]
}

CloudTrail

CloudTrail logs two flavors of event:

Trails - the persistent capture

Out of the box, AWS keeps a rolling 90-day window of management events you can browse in the console. To retain longer, search programmatically, or capture data events, you create a trail - a configured capture that lands events in S3 (and optionally CloudWatch Logs).

# A multi-region trail captures events from every region with one config.
# An org-wide trail captures all member accounts too (set up via Organizations).
aws cloudtrail create-trail \
  --name compass-org-trail \
  --s3-bucket-name compass-cloudtrail-archive-123 \
  --is-multi-region-trail \
  --is-organization-trail \
  --enable-log-file-validation

aws cloudtrail start-logging --name compass-org-trail

# Look up a specific event in the rolling 90-day view (no trail needed)
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=EventName,AttributeValue=DeleteBucket \
  --max-results 10

X-Ray distributed tracing

X-Ray is AWS's distributed tracing service. It's similar in scope to Application Insights' distributed traces, with one big architectural difference: X-Ray is just the backend. In a modern setup you instrument with OpenTelemetry (via ADOT), which sends traces to X-Ray for storage and visualization. The X-Ray SDK still exists but is being de-emphasized.

Sampling - why "trace everything" is a bad idea

At AWS-scale request rates, tracing 100% of requests is both expensive and noisy. X-Ray's default sampling rule keeps the first request per second per service, plus 5% of the rest:

{
  "version": 2,
  "default": {
    "fixed_target": 1,         // 1 trace per second per service
    "rate": 0.05             // + 5% of all additional requests
  },
  "rules": []           // per-path overrides go here
}

The default is sensible. Override with custom rules to over-sample specific endpoints (the checkout flow, the new feature you just shipped) and under-sample others (health checks, the noisy /metrics Prometheus scrape).

Container Insights & Lambda Insights

The base CloudWatch metric set for ECS, EKS, and Lambda is functional but sparse. AWS sells two opt-in Insights bundles that emit richer schemas:

Note: AWS reuses the name "Application Insights" - it's a different product from Azure's App Insights despite the identical brand. AWS Application Insights is an opt-in auto-monitoring layer; Azure App Insights is the core APM service.

Try it: poke the observability stack

aws cloudwatch list-metrics --max-items 20

# Filter to one namespace
aws cloudwatch list-metrics \
  --namespace AWS/Lambda \
  --max-items 10

# See which dimensions exist for one metric
aws cloudwatch list-metrics \
  --namespace AWS/Lambda \
  --metric-name Invocations

aws logs describe-log-groups --max-items 20 \
  --query "logGroups[*].[logGroupName, retentionInDays, storedBytes]" \
  --output table

# Look for groups with no retention set (= forever, $$)
aws logs describe-log-groups \
  --query "logGroups[?retentionInDays==null].logGroupName" \
  --output table

# Start the query - returns a queryId
QID=$(aws logs start-query \
  --log-group-name "/aws/lambda/some-existing-function" \
  --start-time $(date -v-1H +%s) \
  --end-time $(date +%s) \
  --query-string 'fields @timestamp, @message | stats count(*) by bin(5m)' \
  --query queryId --output text)

# Poll for completion (usually <5s for small windows)
aws logs get-query-results --query-id $QID

# Alarms - filter to anything currently firing
aws cloudwatch describe-alarms \
  --state-value ALARM \
  --query "MetricAlarms[*].[AlarmName, StateValue, StateReason]" \
  --output table

# Trails - how is auditing configured?
aws cloudtrail describe-trails \
  --query "trailList[*].[Name, IsMultiRegionTrail, IsOrganizationTrail, S3BucketName]" \
  --output table

# Recent root-user activity (security hygiene check)
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=Username,AttributeValue=root \
  --max-results 5

Quick check

Gotchas for Azure devs

Project Compass: add alarms & a dashboard

By chapter 8 the gfn-reports service has a Lambda worker, an SQS intake, a DynamoDB table, an S3 bucket, and KMS encryption. It does work. But if it breaks at 2am, nobody knows. This chapter's slice adds the visibility layer: an SNS topic for paging, three alarms on Lambda health, and a CloudWatch dashboard.

# Create the SNS topic that all our alarms will publish to
TOPIC_ARN=$(aws sns create-topic \
  --name compass-oncall \
  --tags Key=Project,Value=Compass \
  --profile compass \
  --query TopicArn --output text)

# Subscribe your email (you'll get a confirmation link)
aws sns subscribe \
  --topic-arn $TOPIC_ARN \
  --protocol email \
  --notification-endpoint you@example.com \
  --profile compass

echo "Topic ARN: $TOPIC_ARN"

# Use a metric math expression: errors / invocations > 0.01
aws cloudwatch put-metric-alarm \
  --alarm-name compass-error-rate-high \
  --alarm-description "gfn-reports error rate exceeds 1%" \
  --evaluation-periods 3 \
  --datapoints-to-alarm 2 \
  --threshold 0.01 \
  --comparison-operator GreaterThanThreshold \
  --treat-missing-data notBreaching \
  --metrics '[
    {
      "Id": "errorRate",
      "Expression": "errors / invocations",
      "Label": "ErrorRate",
      "ReturnData": true
    },
    {
      "Id": "errors",
      "MetricStat": {
        "Metric": {
          "Namespace": "AWS/Lambda",
          "MetricName": "Errors",
          "Dimensions": [{"Name": "FunctionName", "Value": "gfn-reports-worker"}]
        },
        "Period": 60,
        "Stat": "Sum"
      },
      "ReturnData": false
    },
    {
      "Id": "invocations",
      "MetricStat": {
        "Metric": {
          "Namespace": "AWS/Lambda",
          "MetricName": "Invocations",
          "Dimensions": [{"Name": "FunctionName", "Value": "gfn-reports-worker"}]
        },
        "Period": 60,
        "Stat": "Sum"
      },
      "ReturnData": false
    }
  ]' \
  --alarm-actions $TOPIC_ARN \
  --profile compass

aws cloudwatch put-metric-alarm \
  --alarm-name compass-throttles-any \
  --alarm-description "gfn-reports has any throttled invocations" \
  --namespace AWS/Lambda \
  --metric-name Throttles \
  --dimensions Name=FunctionName,Value=gfn-reports-worker \
  --statistic Sum \
  --period 60 \
  --evaluation-periods 1 \
  --threshold 0 \
  --comparison-operator GreaterThanThreshold \
  --treat-missing-data notBreaching \
  --alarm-actions $TOPIC_ARN \
  --profile compass

aws cloudwatch put-metric-alarm \
  --alarm-name compass-dlq-depth-high \
  --alarm-description "More than 10 messages stuck in the gfn-reports DLQ" \
  --namespace AWS/SQS \
  --metric-name ApproximateNumberOfMessagesVisible \
  --dimensions Name=QueueName,Value=gfn-reports-dlq \
  --statistic Maximum \
  --period 300 \
  --evaluation-periods 1 \
  --threshold 10 \
  --comparison-operator GreaterThanThreshold \
  --treat-missing-data notBreaching \
  --alarm-actions $TOPIC_ARN \
  --profile compass

cat > /tmp/compass-dashboard.json <<'EOF'
{
  "widgets": [
    {
      "type": "metric", "x": 0, "y": 0, "width": 12, "height": 6,
      "properties": {
        "title": "Lambda invocations and errors",
        "metrics": [
          ["AWS/Lambda", "Invocations", "FunctionName", "gfn-reports-worker", {"stat": "Sum"}],
          [".",          "Errors",      ".",            ".",                  {"stat": "Sum"}]
        ],
        "period": 60, "region": "us-east-1", "view": "timeSeries", "stacked": false
      }
    },
    {
      "type": "metric", "x": 12, "y": 0, "width": 12, "height": 6,
      "properties": {
        "title": "Lambda duration p95",
        "metrics": [
          ["AWS/Lambda", "Duration", "FunctionName", "gfn-reports-worker", {"stat": "p95"}]
        ],
        "period": 60, "region": "us-east-1", "view": "timeSeries"
      }
    },
    {
      "type": "metric", "x": 0, "y": 6, "width": 12, "height": 6,
      "properties": {
        "title": "SQS intake + DLQ depth",
        "metrics": [
          ["AWS/SQS", "ApproximateNumberOfMessagesVisible", "QueueName", "gfn-reports-intake", {"stat": "Maximum"}],
          [".",       ".",                                  ".",         "gfn-reports-dlq",    {"stat": "Maximum"}]
        ],
        "period": 60, "region": "us-east-1", "view": "timeSeries"
      }
    }
  ]
}
EOF

aws cloudwatch put-dashboard \
  --dashboard-name Compass \
  --dashboard-body file:///tmp/compass-dashboard.json \
  --profile compass

# Confirm
aws cloudwatch describe-alarms \
  --alarm-name-prefix compass- \
  --query "MetricAlarms[*].[AlarmName, StateValue]" \
  --output table \
  --profile compass

Recap & next

What stuck?

• The map: Azure Monitor splits into CloudWatch (metrics + logs + alarms + dashboards), CloudTrail (audit), X-Ray (traces). All four are first-class services with their own CLIs.
• Metrics: namespace + name + dimensions + statistic + period. Standard retention decays from 1-sec (3h) to 1-min (15d) to 5-min (63d) to 1-hour (15mo). EMF folds metrics into log lines to dodge the per-call API cost.
• Logs Insights: not KQL, not SQL - its own pipe syntax with fields, filter, stats, sort, parse. JSON-structured logs make queries painless.
• Alarms: three flavors (metric, composite, anomaly). Always wire to an SNS topic, never directly to email - SNS is the fan-out.
• CloudTrail: management events free for the first trail, data events cost real money. One org-wide multi-region trail covers most needs.
• X-Ray: default sampling = 1/sec/service + 5%. ADOT (AWS Distro for OpenTelemetry) is the recommended SDK going forward.

The mental model in one sentence

Common pitfalls so far