AWS in Practice · 07 — Observability

Logs, metrics, alarms, and tracing to see the system running.

About this module

The system is built. What's missing is the part that separates a study project from something you can actually trust: seeing what's happening in real time. CloudWatch Logs, metrics, alarms, X-Ray, and the wrap-up of the course with next steps to keep growing.

Level: Intermediate · Estimated duration: 3-4 hours

Table of Contents

1. Why observability matters
2. CloudWatch: AWS's nervous system
3. CloudWatch Logs
4. Structured JSON logs
5. Metrics: the two categories
6. Custom metrics via EMF
7. Alarms that matter
8. Dashboards
9. X-Ray: distributed tracing
10. Observability costs
11. Tearing down the infrastructure
12. Practice exercises
13. Wrap-up and next steps

01. Why observability matters

You finished module 06 with several Lambdas, microservices on Beanstalk, SQS queues, SNS topics, and DynamoDB tables. All of them talking to each other. But how do you know it's working? And when something breaks — because it will break — how do you find out where?

The difference between logging and observability

Logging is recording events. You throw a string into a file (or a CloudWatch Log) and move on with your life. It's necessary, but insufficient — if you need to know your API's average latency per endpoint over the last 24 hours, logs alone don't answer that well.

Observability is the ability to ask the system new questions without having to change the code. You instrument the system with three kinds of signal — logs, metrics, and traces — and then use tools to correlate those signals and investigate problems you didn't even know existed.

The three pillars

• Logs — Textual or structured records of events. Good for understanding what happened in a specific execution. High volume, storage cost.
• Metrics — Numbers aggregated over time (latency, error rate, CPU usage). Good for answering "how is the system behaving?". Low volume, cheap.
• Traces — A view of a request's journey passing through multiple services. Good for answering "where, exactly, is the bottleneck?". Medium volume, medium cost.

On AWS, these three pillars are offered respectively by CloudWatch Logs, CloudWatch Metrics, and X-Ray — all integrated into the console and billable on demand.

Why looking at the console isn't enough

A common trap: relying on the AWS console to diagnose problems. The console shows current state: how many messages are in the queue right now, what the environment's status is right now. When something breaks, what you need is the history — what happened in the last 5 minutes before the alarm fired. Without prior instrumentation, that history doesn't exist.

02. CloudWatch: AWS's nervous system

CloudWatch is AWS's umbrella service for monitoring. Almost everything you created in previous modules is already being monitored by it passively — logs being collected, basic metrics being recorded. But to extract value, you need to understand its pieces.

The main components

• Logs — Collects, stores, and indexes textual logs. Charges per GB ingested + GB stored.
• Metrics — Stores numeric time series. AWS publishes automatically; you can publish your own.
• Alarms — Trigger actions (email, Lambda, SNS) when a metric crosses a threshold for X consecutive periods.
• Dashboards — Visual panels combining metrics and logs. Useful for a system overview.
• Logs Insights — A query language to ask questions on top of logs (similar to SQL).
• Synthetics — Tests that run continuously simulating users (we won't use it, but it's worth knowing about).

What's already being collected

Without you doing anything, AWS already collects:

• Lambda publishes metrics for invocations, errors, duration, and throttles. Logs go to a log group /aws/lambda/<function-name>.
• SQS publishes the number of visible messages, in-flight messages, and age of the oldest message.
• SNS publishes the number of messages published and delivered.
• DynamoDB publishes RCU/WCU consumed, latency, throttles.
• Beanstalk publishes environment health metrics; EC2 instances publish CPU, network, disk.

All of this stays in CloudWatch for 15 months by default (with decreasing granularity over time). That's a lot for free — the challenge isn't having data, it's knowing which questions to ask and which alarms to create.

Logs generated by your code

Anything your application writes to stdout or stderr ends up in CloudWatch Logs automatically, in both cases: Lambda (via the runtime) and Beanstalk (via the CloudWatch agent that comes installed). You don't need to configure the CloudWatch SDK — just use console.log in Node.js.

03. CloudWatch Logs

Logs are the most flexible and most expensive form of monitoring. It's worth understanding the hierarchy, the features and — above all — how to control the cost.

Hierarchy: log group, log stream, event

CloudWatch Logs organizes data into three levels:

• Log Group — groups related logs, generally by application or function. Example: /aws/lambda/order-event-consumer. The retention and KMS settings live here.
• Log Stream — a sequence of events coming from the same source. For Lambdas, it's usually one per execution instance. For EC2, one per instance.
• Log Event — a log line: timestamp + message. It's the smallest unit.

Configure retention on every log group

By default, CloudWatch Logs retains logs indefinitely. In a course, this becomes your biggest silent cost. Configure retention on every log group:

# List all log groups
aws logs describe-log-groups \
  --query 'logGroups[*].[logGroupName,retentionInDays]' \
  --output table
 
# Set 7-day retention for a specific log group
aws logs put-retention-policy \
  --log-group-name /aws/lambda/order-event-consumer \
  --retention-in-days 7
 
# Apply 7 days to ALL log groups (bash script)
aws logs describe-log-groups \
  --query 'logGroups[*].logGroupName' --output text | \
  tr '\t' '\n' | while read group; do
    aws logs put-retention-policy \
      --log-group-name "$group" --retention-in-days 7
  done

Searching logs in the console

For quick manual inspection, CloudWatch Logs has text search in the console itself: go to Log groups > <your group>, click Search log group, and use the filter syntax:

# Filter by a single word
ERROR
 
# Combine words (all must appear)
ERROR Inventory
 
# Exclude a word
ERROR -Connection
 
# JSON: field equals
{ $.level = "error" }
 
# JSON: numeric field
{ $.duration > 1000 }
 
# JSON: combination
{ $.level = "error" && $.service = "orders" }

For richer analysis — grouping, aggregation, sorting — use CloudWatch Logs Insights, with its own SQL-like syntax:

# Top 10 errors in the last 24 hours
fields @timestamp, @message
| filter level = "error"
| stats count() by error_code
| sort count desc
| limit 10
 
# p99 latency per endpoint in the last hour
fields @timestamp, route, duration
| filter ispresent(duration)
| stats pct(duration, 99) by route
| sort `pct(duration, 99)` desc

04. Structured JSON logs

This is the one logging decision that really matters in this project: logging in structured JSON instead of free text. The difference is night and day when it comes to debugging.

Why structured, not free-form

Compare these two logs of the same event:

# Free-text log (BAD)
[INFO] Order order-123 processed in 234ms by user user-456
 
# Structured JSON log (GOOD)
{"timestamp":"2026-05-07T14:23:11Z","level":"info",
 "service":"orders","event":"order.processed",
 "orderId":"order-123","userId":"user-456",
 "duration_ms":234,"region":"us-east-1"}

Free text forces you to write regex every time you want to ask a question. JSON is parsed automatically by CloudWatch Logs Insights — you query the fields directly.

A logging standard for the project

Adopt these fields as a baseline across all the project's services. It's not an exhaustive list, but it's the minimum viable set for correlation:

• timestamp — ISO 8601 in UTC. CloudWatch already adds it, but having it in the payload helps with external systems.
• level — info, warn, error, debug. Always lowercase.
• service — the microservice name: orders, inventory, payments.
• event — the domain event name: order.created, payment.failed.
• requestId — a unique request ID. Lets you follow one operation across all services.
• userId / orderId / etc. — domain IDs relevant to the event.
• duration_ms — for timed operations: latency in milliseconds.
• error — an object with code, message, stack when applicable.

Use a library, not console.log

For Node.js, use pino or winston. Both support JSON output natively, are performant, and plug easily into CloudWatch:

// pino: minimal and fast
import pino from 'pino';
 
const log = pino({
  level: process.env.LOG_LEVEL || 'info',
  base: {
    service: 'orders',
    region: process.env.AWS_REGION,
  },
});
 
log.info({ orderId, userId, duration_ms: 234 }, 'order.processed');
log.error({ err, orderId }, 'order.failed');

requestId propagation

To be able to trace a request across services, propagate the requestId in HTTP headers and in SQS/SNS message attributes. When Orders publishes an event, it puts the requestId in the message attributes. When the consumer Lambda processes it, it extracts and logs with the same ID. Result: a search for requestId = X returns the entire journey.

05. Metrics: the two categories

Metrics are numbers aggregated over time. They fall into two important categories: native ones, published by AWS automatically, and custom ones, which you publish from your code.

Native metrics (free)

Each AWS service publishes its own set. It's worth knowing the main ones for the services you use:

Dimensions: how to filter

Each metric comes with dimensions — keys that let you filter. For example, Lambda Duration comes with the FunctionName dimension, letting you see duration broken out by function.

Knowing the available dimensions is critical for creating useful charts and alarms. In the console, when you choose a metric, the dimensions appear as tabs. On the CLI:

# List metrics and dimensions for a namespace
aws cloudwatch list-metrics --namespace AWS/Lambda \
  --metric-name Duration

Statistics and periods

Every metric is queried with a statistic (Average, Sum, Min, Max, p95, p99) and a period (in seconds). An important detail:

• Average tends to hide problems. If 99 requests return in 50ms and 1 returns in 5s, the average is ~100ms — it looks OK, but there are frustrated users.
• p99 or p95 are almost always better than Average for latency. They show the worst-case experience, which is what the user feels.
• Sum only makes sense for cumulative counters: requests, errors, bytes.
• Min / Max are useful for detecting outlier values.

06. Custom metrics via EMF

Native metrics cover infrastructure: Lambda latency, queue size. But you also want domain metrics: orders created per minute, total value processed, payment approval rate. Those you publish yourself.

Two ways to publish

The direct way is to call the CloudWatch SDK: PutMetricData. It works, but it has two problems: each call is a separate API call (cost + latency) and it blocks the main flow. The modern way is to use EMF (Embedded Metric Format).

EMF: a metric embedded in the log

EMF is a special JSON format that, when it appears in CloudWatch Logs, is automatically extracted and published as a metric. In other words, you only need to log — CloudWatch does the rest. No extra call.

{
  "_aws": {
    "Timestamp": 1715089391000,
    "CloudWatchMetrics": [{
      "Namespace": "OrderSystem/Orders",
      "Dimensions": [["Service", "Region"]],
      "Metrics": [
        { "Name": "OrderCreated", "Unit": "Count" },
        { "Name": "OrderValue", "Unit": "None" }
      ]
    }]
  },
  "Service": "orders",
  "Region": "us-east-1",
  "OrderCreated": 1,
  "OrderValue": 234.50
}

In real code, nobody writes this by hand. Use the official aws-embedded-metrics library:

import { metricScope, Unit } from 'aws-embedded-metrics';
 
export const handler = metricScope(metrics => async (event) => {
  metrics.setNamespace('OrderSystem/Orders');
  metrics.setDimensions({ Service: 'orders', Region: 'us-east-1' });
 
  const order = await processOrder(event);
 
  metrics.putMetric('OrderCreated', 1, Unit.Count);
  metrics.putMetric('OrderValue', order.total, Unit.None);
  metrics.putMetric('ProcessingTime', order.duration_ms,
    Unit.Milliseconds);
 
  return { statusCode: 200 };
});

Metrics worth having in the project

Don't publish everything — each dimension becomes a costly combination. Focus on what really matters for making a decision:

• OrderCreated in OrderSystem/Orders, dimension Service. Counts orders created per unit of time.
• OrderValue in OrderSystem/Orders. Sum of order value — useful for correlating with volume.
• InventoryLow in OrderSystem/Inventory, dimension ProductId. Counts the times stock dropped below the minimum.
• PaymentFailed in OrderSystem/Payments, dimension Reason. Counts failures broken out by reason.
• EventProcessingTime in all consumer Lambdas, dimension EventType. Latency per event type.

07. Alarms that matter

An alarm is a state machine: it watches a metric and fires an action when it crosses a threshold. Well configured, it wakes you up at 3 a.m. for real problems. Poorly configured, it gets ignored because you've grown used to it.

Anatomy of an alarm

• Metric — which number to watch.
• Statistic + Period — how to aggregate (Average, Sum) and over what window (1min, 5min).
• Threshold + Comparison — which value is "bad". Example: > 5, < 0.99.
• Datapoints to alarm — how many consecutive periods must be violated to fire. Avoids a false positive from a single spike.
• Action — what to do when it fires. Typically: publish to an SNS topic that has you as a subscriber.
• Treat missing data — what to do when no data arrives: ignore it, treat as OK, treat as bad.

Essential alarms for the project

These are the alarms you should have running in this system. They're not the only possible ones — they're the ones most likely to save you from a bad night.

Notifications via SNS + email

The simplest way to receive alerts: an SNS topic called alarms-prod with your email as a subscriber. All alarms publish to it. You confirm the subscription once and you're done.

# Create the topic
aws sns create-topic --name alarms-prod
 
# Subscribe your email (a confirmation email will arrive)
aws sns subscribe \
  --topic-arn arn:aws:sns:us-east-1:123456789012:alarms-prod \
  --protocol email \
  --notification-endpoint seu@email.com
 
# Create an alarm pointing to that topic
aws cloudwatch put-metric-alarm \
  --alarm-name lambda-orders-errors \
  --metric-name Errors --namespace AWS/Lambda \
  --dimensions Name=FunctionName,Value=order-event-consumer \
  --statistic Sum --period 300 --threshold 5 \
  --comparison-operator GreaterThanThreshold \
  --evaluation-periods 2 \
  --alarm-actions arn:aws:sns:us-east-1:123456789012:alarms-prod

08. Dashboards

Dashboards are visual panels that combine metrics and logs. They serve three purposes: a daily overview, investigation when something looks off, and sharing with the team.

Anti-pattern: the dashboard nobody looks at

It's tempting to cram a dashboard with 30 colorful widgets. It doesn't work — nobody opens a crowded dashboard. Think in two categories:

• Operations Dashboard — an overview, 5-8 widgets, a single screen. p99 latency of each service, error rate, queue size. It's the one you open every morning.
• Service Dashboard — one per microservice, more detailed. Everything about Orders in one place: latency, errors, DynamoDB capacity, domain metrics. You only open it when something looks off with that service.

Suggested dashboard for the project

Start with an OrderSystem-Operations dashboard containing:

• Row 1 — OrderCreated and PaymentFailed over the last 24h (lines).
• Row 2 — p95 latency of each microservice (3 lines on the same chart).
• Row 3 — ApproximateNumberOfMessagesVisible across all queues (lines).
• Row 4 — Lambda Errors per function (a gauge for each function).
• Row 5 — Top 10 recent error messages via Logs Insights.

Creating it via CloudFormation

Like everything else in the project, dashboards should be code. The resource is AWS::CloudWatch::Dashboard, with the body in JSON:

OperationsDashboard:
  Type: AWS::CloudWatch::Dashboard
  Properties:
    DashboardName: OrderSystem-Operations
    DashboardBody: !Sub |
      {
        "widgets": [{
          "type": "metric",
          "properties": {
            "metrics": [
              ["OrderSystem/Orders", "OrderCreated"],
              ["OrderSystem/Payments", "PaymentFailed"]
            ],
            "period": 300,
            "stat": "Sum",
            "title": "Throughput"
          }
        }]
      }

09. X-Ray: distributed tracing

X-Ray answers a question that logs and metrics don't answer well: "what happened in this specific request, step by step, across all services?". It's the missing piece for debugging latency in distributed systems.

What X-Ray delivers

You enable X-Ray on the services, and each request produces a trace. A trace is a tree of segments (each service) and subsegments (operations within it: a DynamoDB call, an SNS call, etc.). The result:

• Service map: a view of the services and how they talk to each other, with average latency and error rate on each connection.
• Trace timeline: for a specific request, a waterfall chart showing how long each operation took.
• Annotations and metadata: you add fields to the trace (orderId, userId) that become filters in searches.
• Sampling: to avoid tracing 100% of requests, X-Ray samples. Default: 1 req/sec + 5% of the rest.

Enabling it on Lambda

On Lambda, it's literally a flag. In CloudFormation:

OrderEventConsumer:
  Type: AWS::Lambda::Function
  Properties:
    FunctionName: order-event-consumer
    # ... other props
    TracingConfig:
      Mode: Active   # X-Ray enabled
 
# And make sure the role has permission
LambdaRole:
  Type: AWS::IAM::Role
  Properties:
    ManagedPolicyArns:
      - arn:aws:iam::aws:policy/service-role/
        AWSLambdaBasicExecutionRole
      - arn:aws:iam::aws:policy/AWSXRayDaemonWriteAccess

Enabling it on Beanstalk (Node.js)

On Beanstalk you need to instrument the app manually. Fortunately, the X-Ray SDK provides automatic patches for Express and the AWS SDK:

import express from 'express';
import AWSXRay from 'aws-xray-sdk-core';
import xrayExpress from 'aws-xray-sdk-express';
 
// automatically patches the AWS SDK
AWSXRay.captureAWS(require('aws-sdk'));
 
const app = express();
 
// middleware: opens the segment at the start of the request
app.use(xrayExpress.openSegment('orders-service'));
 
app.post('/orders', handler);
 
// middleware: closes the segment at the end
app.use(xrayExpress.closeSegment());

Useful annotations

Annotations are indexed fields — you can filter traces by them. Metadata is read-only:

const segment = AWSXRay.getSegment();
 
// Annotations (indexed, filterable)
segment.addAnnotation('orderId', order.id);
segment.addAnnotation('region', 'us-east-1');
 
// Metadata (visible, but not filterable)
segment.addMetadata('order', { 
  total: order.total, 
  itemCount: order.items.length
});

10. Observability costs

CloudWatch is the trickiest AWS service when it comes to cost. You rarely get a surprise from Lambda or DynamoDB under normal use — but you can get a nasty shock from Logs if you're not careful.

Where the money goes

• Logs Ingestion — ~US$ 0.50/GB ingested. It's the dominant item for verbose apps.
• Logs Storage — ~US$ 0.03/GB/month. Small, but it grows if retention is infinite.
• Logs Insights — ~US$ 0.005/GB scanned per query. Watch out for queries over large windows.
• Custom Metrics — US$ 0.30 per metric/month. High cardinality (many dimensions) blows up here.
• API requests — US$ 0.01 per 1k calls to PutMetricData. EMF avoids this cost.
• Alarms — US$ 0.10 per alarm/month (standard); composite alarms are US$ 0.50.
• X-Ray — US$ 5 per million traces recorded; US$ 0.50 per million scans.

Habits that keep costs under control

• Short retention on every log group — 7 days for dev, 30 for prod, more than that only with a reason.
• Don't log entire payloads. Large payloads (especially in JSON) inflate ingestion without adding value. Log IDs and counters.
• Configurable log level via env var. In prod, run at info. debug only for one-off troubleshooting.
• Low cardinality on custom metric dimensions — no UserId as a dimension.
• Sampling in X-Ray. Don't trace 100%, especially at high volume.
• Review alarms periodically. A useless alarm is US$ 0.10/month of pure bureaucracy.

Estimate for the course project

With the recommended setup (structured logs, 7-day retention, ~10 alarms, EMF metrics, X-Ray enabled), the monthly observability cost lands between US$ 1 and US$ 3 for study use. If you see more than that, it's a sign of some runaway — usually a log group left in debug mode or a metric with high cardinality.

11. Tearing down the infrastructure

You've reached the end. The system works, it's observed, it's documented. Time for the last important lesson: tear it all down without leaving residue.

Why tearing down is part of learning

Most of the inflated bills you'll hear about aren't from people running systems in production — they're from resources forgotten in study environments. A Beanstalk environment with an ALB can cost US$ 16/month sitting there doing nothing. You don't see it, don't use it, don't want it, but the bill arrives.

Since you did everything via CloudFormation, tearing it down is one command:

# List your stacks
aws cloudformation list-stacks \
  --stack-status-filter CREATE_COMPLETE UPDATE_COMPLETE
 
# Delete a stack
aws cloudformation delete-stack --stack-name order-system-prod
 
# Track the deletion
aws cloudformation describe-stacks --stack-name order-system-prod \
  --query 'Stacks[0].StackStatus'
 
# Wait for it to finish (blocks until DELETE_COMPLETE or DELETE_FAILED)
aws cloudformation wait stack-delete-complete \
  --stack-name order-system-prod

Order matters in nested stacks

If you have nested stacks (a parent stack with child stacks), delete the parent — CloudFormation takes care of tearing down the children in the correct order. If you deleted only the child by mistake and the parent is still referencing it, you fall into a somewhat broken state and will probably need to delete the parent too.

What CloudFormation does NOT delete

• S3 buckets with objects. If the stack created a bucket and then you (or the app) put objects inside, CloudFormation fails to delete it. Empty it first: aws s3 rm s3://bucket --recursive.
• DynamoDB tables with DeletionProtection. If you enabled protection, disable it first via the console or CLI.
• Resources with DeletionPolicy: Retain. You explicitly marked them not to be deleted — so the resource stays, and you need to delete it manually later.
• Resources created manually outside the stack. Always. If you went into the console and created a security group by hand, it isn't managed by the stack and it stays.

Final cleanup checklist

1. Delete all the project's CloudFormation stacks.
2. List and delete any leftover S3 buckets (empty them first).
3. Check Beanstalk: no active environments.
4. Check CloudWatch Logs: old log groups may linger — delete the ones you won't use anymore.
5. Check DynamoDB: no active tables.
6. Check SQS and SNS: nothing listed.
7. Check IAM: roles and policies created by the stack should be gone. Users you created manually, you decide.
8. Check the billing dashboard: confirm that usage went back to zero over the next few days.

12. Practice exercises

The exercises in this final module mix hands-on observability with reflection on what you learned throughout the whole course.

13. Wrap-up and next steps

You've reached the end. Not figuratively — you've actually reached the end, with a functional system built, run, observed, and torn down. It's worth a quick pause to look at what happened.

What you know now that you didn't know before

• Microservices aren't magic. They're an architectural decision with concrete trade-offs: operational independence vs coordination complexity. You felt both sides firsthand.
• NoSQL demands different modeling. You modeled DynamoDB by access patterns, chose partition keys, created GSIs. It's neither worse nor better than SQL — it's different.
• PaaS and FaaS solve different things. Beanstalk for always-on low-latency services, Lambda for event-driven tasks. You used both and understood when each makes sense.
• Messaging decouples. Orders doesn't know about Inventory. You could swap out Inventory entirely tomorrow and Orders wouldn't even notice — because everything goes through SNS+SQS.
• Infrastructure is code. Everything you created can be recreated by create-stack. That changes your relationship with infra: it becomes an artifact, not an asset.
• Observability isn't a luxury. Without structured logs, metrics, and alarms, the system is a black box. With them, it explains itself.

Where to go now

You're at an interesting point: you have a solid foundation to choose specific directions. Some suggestions organized by interest:

Technical depth

• API Gateway + Cognito — real authentication, rate limiting, usage plans. Take the current system and put a gateway with auth in front of it.
• Step Functions — workflow orchestration. Useful when you have multi-step flows with retries, branching, parallelism. Distributed sagas become bread and butter.
• EventBridge — the modern successor to SNS for events. Supports schemas, complex rules, native integration with SaaS. It's worth migrating Orders→Inventory to see the difference.
• ECS / Fargate — managed containers. When Beanstalk starts to feel tight, this is the next step up.

Practices and culture

• The Twelve-Factor App — read and apply it. You already use a good chunk of it without realizing.
• Designing Data-Intensive Applications (Kleppmann) — the bedside book for anyone working with distributed systems.
• AWS's Well-Architected Framework — 6 pillars that organize thinking about AWS projects. Do a review of your project.
• Chaos Engineering — once you trust the system, start breaking it on purpose (FIS on AWS) to validate its resilience.

Certifications (if you want)

This project covers a good chunk of two AWS certifications. If you're thinking about an exam:

• AWS Certified Developer — Associate — 100% aligned with what you did. You covered Lambda, DynamoDB, SQS, SNS, IAM, CloudWatch, and CloudFormation. You're missing a bit of Cognito, API Gateway, Kinesis.
• AWS Certified Solutions Architect — Associate — Covers all of that plus more architecture: VPC, EC2 in depth, RDS, advanced S3, Route 53. It's the most common certification as a "first AWS".

And when it goes wrong

At some point you'll be working on something, it'll go wrong in a way you can't explain, and you'll feel like you're wasting time. Remember: that moment is the learning. Everything you felt in this course — frustration with permissions, a scare from a timeout, the delay in understanding why the event wasn't arriving — is exactly what builds intuition. The engineers you admire went through this thousands of times and simply remember the pattern the next time.

END OF THE COURSE

Thank you for making it this far.

You know AWS in practice now. Use it well.