Messaging & Events

The cheat table: Azure messaging → AWS messaging

SQS - the workhorse queue

SQS comes in two flavors. Pick the wrong one and you'll spend a chapter swearing at "near-duplicate" messages.

The consumer poll loop

Side-by-side: create a queue

# Create namespace then queue
az servicebus namespace create \
  --name sb-orders \
  --resource-group rg-app \
  --sku Standard

az servicebus queue create \
  --namespace-name sb-orders \
  --resource-group rg-app \
  --name order-intake \
  --max-delivery-count 5 \
  --lock-duration PT30S \
  --enable-dead-lettering-on-message-expiration true

# Dead letter is a sub-entity:
# order-intake/$DeadLetterQueue

# 1. Create the DLQ first (you need its ARN)
aws sqs create-queue \
  --queue-name order-intake-dlq

DLQ_ARN=$(aws sqs get-queue-attributes \
  --queue-url <dlq-url> \
  --attribute-names QueueArn \
  --query 'Attributes.QueueArn' --output text)

# 2. Create the main queue, point at DLQ
aws sqs create-queue \
  --queue-name order-intake \
  --attributes "{
    \"VisibilityTimeout\": \"30\",
    \"RedrivePolicy\": \"{\\\"deadLetterTargetArn\\\":\\\"$DLQ_ARN\\\",\\\"maxReceiveCount\\\":\\\"5\\\"}\"
  }"

Send and receive

# Send a message
aws sqs send-message \
  --queue-url https://sqs.us-east-1.amazonaws.com/123/order-intake \
  --message-body '{"orderId":"o-001","amount":42.50}'

# Poll for messages (long polling - waits up to 20s)
aws sqs receive-message \
  --queue-url https://sqs.us-east-1.amazonaws.com/123/order-intake \
  --wait-time-seconds 20 \
  --max-number-of-messages 10

# Returns Messages[].ReceiptHandle - you NEED this to delete

# Delete after successful processing
aws sqs delete-message \
  --queue-url https://sqs.us-east-1.amazonaws.com/123/order-intake \
  --receipt-handle "AQEB..."

#!/usr/bin/env bash
QUEUE_URL="https://sqs.us-east-1.amazonaws.com/123/order-intake"

while true; do
  MSGS=$(aws sqs receive-message \
    --queue-url "$QUEUE_URL" \
    --max-number-of-messages 10)

  # Extract bodies and process them
  echo "$MSGS" | jq -r '.Messages[]?.Body' | while read -r body; do
    process_order "$body"   # succeeds!
  done
done

SNS - pub/sub and fan-out

SNS is push-based pub/sub. You publish to a topic; the topic delivers to every subscription. Compare to Service Bus Topics + Subscriptions - same shape, different vocab.

The SNS + SQS fan-out pattern

Pure SNS is fire-and-forget; the consumer must be online to receive. The standard trick is to subscribe SQS queues to the topic. Each consumer reads from its own queue, processes at its own pace, and SNS becomes the broadcast bus while SQS becomes the durable buffer.

# 1. Create the topic
TOPIC_ARN=$(aws sns create-topic --name order-events --query 'TopicArn' --output text)

# 2. Create the consumer queue
QUEUE_URL=$(aws sqs create-queue --queue-name warehouse-orders --query 'QueueUrl' --output text)
QUEUE_ARN=$(aws sqs get-queue-attributes --queue-url "$QUEUE_URL" \
  --attribute-names QueueArn --query 'Attributes.QueueArn' --output text)

# 3. Subscribe the queue to the topic
aws sns subscribe \
  --topic-arn "$TOPIC_ARN" \
  --protocol sqs \
  --notification-endpoint "$QUEUE_ARN"

# 4. CRITICAL: give SNS permission to send to the queue
# (resource-based policy on the queue - chapter 2 territory)
aws sqs set-queue-attributes \
  --queue-url "$QUEUE_URL" \
  --attributes Policy='{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"Service":"sns.amazonaws.com"},"Action":"sqs:SendMessage","Resource":"'"$QUEUE_ARN"'","Condition":{"ArnEquals":{"aws:SourceArn":"'"$TOPIC_ARN"'"}}}]}'

Cross-account subscriptions

SNS topics can list principals from other AWS accounts. The topic's access policy grants sns:Subscribe to the other account; the other account's queue policy must allow sns:SendMessage from the topic ARN. The same two-sided handshake we covered in chapter 2 for cross-account access.

EventBridge - the event router

EventBridge is AWS's general-purpose event bus. It started life as CloudWatch Events (handling AWS service events like "EC2 instance state changed") and grew into a routing fabric that consumes events from AWS services, partner SaaS apps (Zendesk, PagerDuty, Datadog), and your own custom apps - then routes them to targets based on JSON pattern matching.

Side-by-side: an AWS console event triggers a Lambda

az eventgrid event-subscription create \
  --name on-blob-created \
  --source-resource-id "/subscriptions/.../mystg" \
  --endpoint "https://func.azurewebsites.net/api/handle" \
  --endpoint-type webhook \
  --included-event-types "Microsoft.Storage.BlobCreated" \
  --subject-begins-with "/blobServices/default/containers/inbox/"

# 1. Define the rule with an event pattern
aws events put-rule \
  --name on-object-created \
  --event-pattern '{
    "source": ["aws.s3"],
    "detail-type": ["Object Created"],
    "detail": {
      "bucket": { "name": ["my-bucket"] },
      "object": { "key": [{ "prefix": "inbox/" }] }
    }
  }'

# 2. Attach the Lambda target
aws events put-targets \
  --rule on-object-created \
  --targets "Id=1,Arn=arn:aws:lambda:us-east-1:123:function:handle"

# 3. Grant EventBridge permission to invoke the Lambda
aws lambda add-permission \
  --function-name handle \
  --statement-id allow-eventbridge \
  --action lambda:InvokeFunction \
  --principal events.amazonaws.com

Content-based routing pattern

The killer feature: rules can inspect any field of the event JSON. This is strict matching - exact string, prefix, suffix, anything-but, numeric range, IP CIDR, or null check. No regex, no SQL.

{
  "source": ["com.compass.orders"],
  "detail-type": ["OrderPlaced"],
  "detail": {
    "customer": {
      "tier": ["premium", "enterprise"]
    },
    "amount": [{ "numeric": [">=", 1000] }],
    "region": [{ "anything-but": ["sanctions-blocked"] }]
  }
}

Schema Registry and partner events

EventBridge can infer the schema of events flowing through a bus and generate strongly-typed bindings (Java, Python, TypeScript). You then call aws events put-events with that type and AWS validates. The Schema Registry is the closest AWS gets to "API contracts for events".

Partner event sources are SaaS apps that AWS has integrated. Zendesk pushes ticket-created events directly to your EventBridge bus. PagerDuty, Datadog, Auth0, Shopify - 100+ partners. The partner creates a partner event source in your account, which you "associate" with a partner event bus.

Kinesis - streaming data

Kinesis is the AWS Event Hubs equivalent - high-throughput, multi-consumer, replay-capable stream storage. It's actually three related services with confusingly similar names.

Shards, partition keys, retention

Producer/consumer code shape

# Create a stream with 2 shards
aws kinesis create-stream --stream-name clickstream --shard-count 2

# Producer: send a record
aws kinesis put-record \
  --stream-name clickstream \
  --partition-key user-42 \
  --data "$(echo -n '{"event":"click","ts":1716393600}' | base64)"

# Consumer: read from a shard (low-level - real code uses KCL)
SHARD_ITER=$(aws kinesis get-shard-iterator \
  --stream-name clickstream \
  --shard-id shardId-000000000000 \
  --shard-iterator-type TRIM_HORIZON \
  --query 'ShardIterator' --output text)

aws kinesis get-records --shard-iterator "$SHARD_ITER" --limit 100

Lambda triggers (event source mappings)

The most common pattern in AWS messaging: "X arrives in service Y, invoke Lambda Z with it". The plumbing that makes this work is an event source mapping - a polling agent that AWS runs on your behalf inside the Lambda service, reading from SQS / Kinesis / DynamoDB Streams / MSK / MQ / DocumentDB and invoking your function with batches.

Batching: when does the function actually fire?

The poller fires Lambda when any of these triggers:

1. BatchSize reached - default 10 for SQS, 100 for Kinesis/DynamoDB Streams, configurable up to 10,000.
2. MaximumBatchingWindowInSeconds elapsed - 0 by default for SQS (fire as soon as anything arrives), up to 300s.
3. Payload limit hit - 6 MB total invocation payload.

So if you set BatchSize=100, MaxBatchingWindow=5: the function fires when 100 messages are buffered OR after 5 seconds (whichever first). Trade-off: bigger batches = more efficiency but higher per-message latency.

aws lambda create-event-source-mapping \
  --function-name process-orders \
  --event-source-arn arn:aws:sqs:us-east-1:123:order-intake \
  --batch-size 10 \
  --maximum-batching-window-in-seconds 5 \
  --function-response-types ReportBatchItemFailures   # partial-batch responses

# Lambda's execution role MUST have sqs:ReceiveMessage,
# sqs:DeleteMessage, sqs:GetQueueAttributes on the queue.

Choosing between them

Four services, overlapping pictures on the marketing diagrams. The decision tree below is what most teams actually use.

Try it: list every messaging service

aws sqs list-queues --profile compass
# {
#   "QueueUrls": [
#     "https://sqs.us-east-1.amazonaws.com/123456789012/some-queue"
#   ]
# }

# Bonus: only queues whose name starts with "compass"
aws sqs list-queues --queue-name-prefix compass- --profile compass

aws sns list-topics --profile compass

# For each topic, show subscriptions:
aws sns list-subscriptions --profile compass

aws events list-event-buses --profile compass
# Always at least the "default" bus.

# What rules are on the default bus?
aws events list-rules --event-bus-name default --profile compass

# What targets does a rule have?
aws events list-targets-by-rule --rule <rule-name> --profile compass

aws kinesis list-streams --profile compass

# Describe a stream's shards and retention:
aws kinesis describe-stream-summary --stream-name <name> --profile compass

Quick check

Gotchas for Azure devs

Project Compass: the intake queue

The Compass gfn-reports service has grown across the last six chapters: an IAM role (ch2), a VPC (ch3), a Lambda (ch4), an S3 bucket for reports (ch5), and a DynamoDB table for the report index (ch6). Now we wire up the intake side: a Standard SQS queue that the upstream report producers will write to, plus a DLQ for poison messages.

aws sqs create-queue \
  --queue-name gfn-reports-intake-dlq \
  --attributes '{
    "MessageRetentionPeriod": "1209600"
  }' \
  --tags Project=Compass,ManagedBy=learn-aws \
  --profile compass

# Capture the DLQ ARN for the next step
DLQ_URL=$(aws sqs get-queue-url --queue-name gfn-reports-intake-dlq \
  --profile compass --query 'QueueUrl' --output text)
DLQ_ARN=$(aws sqs get-queue-attributes --queue-url "$DLQ_URL" \
  --attribute-names QueueArn --profile compass \
  --query 'Attributes.QueueArn' --output text)

echo "DLQ ARN: $DLQ_ARN"   # arn:aws:sqs:us-east-1:123:gfn-reports-intake-dlq

aws sqs create-queue \
  --queue-name gfn-reports-intake \
  --attributes '{
    "VisibilityTimeout": "60",
    "MessageRetentionPeriod": "345600",
    "ReceiveMessageWaitTimeSeconds": "20",
    "RedrivePolicy": "{\"deadLetterTargetArn\":\"'"$DLQ_ARN"'\",\"maxReceiveCount\":\"5\"}"
  }' \
  --tags Project=Compass,ManagedBy=learn-aws \
  --profile compass

# Settings explained:
#   VisibilityTimeout=60         - 2x our Lambda timeout (chapter 4 set Lambda to 30s)
#   MessageRetentionPeriod=4d    - default is 4 days; we leave it alone
#   ReceiveMessageWaitTimeSeconds=20 - enables long polling by default (no short-poll cost trap)
#   maxReceiveCount=5            - 5 retries before a message is parked in the DLQ

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowSendToIntake",
      "Effect": "Allow",
      "Action": ["sqs:SendMessage", "sqs:GetQueueAttributes"],
      "Resource": "arn:aws:sqs:us-east-1:123456789012:gfn-reports-intake"
    },
    {
      "Sid": "AllowReceiveFromIntake",
      "Effect": "Allow",
      "Action": [
        "sqs:ReceiveMessage",
        "sqs:DeleteMessage",
        "sqs:GetQueueAttributes",
        "sqs:ChangeMessageVisibility"
      ],
      "Resource": "arn:aws:sqs:us-east-1:123456789012:gfn-reports-intake"
    }
  ]
}

aws iam put-role-policy \
  --role-name gfn-reports-role \
  --policy-name gfn-reports-intake-access \
  --policy-document file://policies/sqs-access-for-reports-role.json \
  --profile compass

# Verify both queues are wired correctly
aws sqs list-queues --queue-name-prefix gfn-reports- --profile compass
aws sqs get-queue-attributes \
  --queue-url "$(aws sqs get-queue-url --queue-name gfn-reports-intake --profile compass --query QueueUrl --output text)" \
  --attribute-names All \
  --profile compass | jq '.Attributes | {VisibilityTimeout, RedrivePolicy, ReceiveMessageWaitTimeSeconds}'

Recap & next

What stuck?

• Four shapes, four services: queue (SQS), pub-sub (SNS), event router (EventBridge), stream (Kinesis). One-to-one with Azure's Service Bus / Service Bus Topics / Event Grid / Event Hubs.
• SQS Standard vs FIFO: Standard is unlimited and best-effort-ordered; FIFO is 300 TPS (un-batched) and strictly ordered per MessageGroupId. Always pair with a DLQ.
• Visibility timeout is the most important SQS setting. Set it to 2-3x your maximum consumer runtime. Default 30s is a footgun for Lambda workloads.
• SNS = blast; EventBridge = route. SNS fans out; EventBridge inspects the payload and decides. Use SNS-to-SQS to add durability behind SNS.
• Kinesis when you need replay or multi-consumer history. 1 MB/s per shard write cap; up to 365-day retention; multiple independent checkpoints.
• Event source mappings are AWS-managed pollers - your Lambda just receives batches. Set partial batch responses to avoid the poison-message trap.

The mental model in one sentence

Common pitfalls so far