2G

Observability

Use the decision tree below to classify any new alert. Every alert must have a severity tier (Critical or Warning) assigned before merging.

Alerts are one of the easiest observability tools to get wrong. Too few alerts and incidents go unnoticed. Too many noisy alerts and on-call engineers stop trusting the system. This guide focuses on the middle ground: alerts that reflect real user impact, page at the right severity, and help the responder act quickly.

Quick Definitions

• Critical alert: a page-worthy issue that needs immediate action because user impact is already happening or imminent
• Warning alert: an early signal that something is drifting and should be investigated before it becomes critical
• SLO: Service Level Objective, a target such as availability or latency that defines acceptable service behaviour
• Error budget: the amount of failure allowed before the SLO is considered breached
• Composite alert: an alert that combines multiple conditions, so a page is triggered only when signals correlate

Alert Decision Tree

The tree maps alert type (Application / Infra / Database / SQS) to severity based on impact thresholds:

Note- This is just a sensible default, you need to customise the threshold based on your context. DO NOT FOLLOW THE THRESHOLDS AS IS.

Why These Thresholds?

Thresholds are derived from two principles:

1. Signal the symptoms users feel. A 5% error rate means 1 in 20 requests fails, users notice 1% is noise in healthy distributed systems.
2. Give time to act before SLO burn. If your 30-day SLO error budget allows 0.1% errors, a 5% error rate burns 50× your budget, every minute counts.

Always tune thresholds to your traffic. A threshold that generates > 5 pages/week is miscalibrated, tighten the window or raise the percentage.

Key takeaway: good alert thresholds are based on impact and actionability, not round numbers chosen in isolation.

Application Monitoring

Application alerts should reflect what users are experiencing first: failed requests, slow responses, and unusual traffic changes.

Apdex formula:

Apdex = (Satisfied + Tolerating / 2) / Total samples
Where:
  Satisfied  = requests <= threshold (e.g. 300ms)
  Tolerating = requests <= 4× threshold (e.g. 1200ms)
  Frustrated = requests > 4× threshold

Key takeaway: Application alerts are strongest when they focus on symptoms users feel directly.

Infra Alerts

Infrastructure alerts matter, but they are usually supporting signals rather than the primary page trigger.

Key takeaway: infra alerts are most useful when they support or explain a user-visible symptom instead of paging on harmless fluctuations.

Database Alerts

Database issues amplify quickly because one slow query or an exhausted pool can affect every request path that depends on it.

Real-World Scenario

A service may still appear healthy at the application layer until connection pool usage starts climbing and slow query latency rises at the same time. By the time requests begin failing, the root cause was already visible at the database layer.

Key takeaway: Database alerts should catch saturation early enough to avoid turning localised slowness into full request failure.

SQS / Queue Alerts

Queue alerts are about backlog health. A growing queue is not always an outage, but it is often the earliest sign that producers and consumers are no longer balanced.

Key takeaway: watch both message age and queue depth, because one tells you how long work is waiting and the other tells you how much work is piling up.

Composite Alerts (Alert on Correlation, Not Individual Metrics)

A single metric spike may be a blip. Page when multiple signals align:

Tip

Pattern: High error rate + latency spike + throughput drop together = likely outage.
High error rate alone = investigate before paging (could be a bad deploy to one pod).

Configure Datadog monitors with AND conditions or use composite alerts to reduce false positives.

Key takeaway: correlated signals produce far better pages than isolated metric spikes.

Alert Fatigue => How to Measure and Fix

Alert quality deteriorates over time. Track these metrics monthly:

Key takeaway: if most alerts do not lead to action, the alerting system is training engineers to ignore it.

Configuring Alerts in Datadog

The exact tooling can vary, but every alert definition should make the same things obvious: what is being measured, what threshold matters, who is paged, and where the runbook lives.

Example: 5% error rate alert for a service:

# Datadog monitor (terraform resource style)
name: "[APP] High Error Rate - my-service"
type: metric alert
query: >
  sum(last_5m):
    sum:trace.http.request.errors{service:my-service}.as_rate()
    /
    sum:trace.http.request.hits{service:my-service}.as_rate()
  > 0.05
message: |
  Error rate > 5% for 5 minutes on my-service.
  Runbook: https://wiki.example.com/runbooks/my-service/high-error-rate
  @pagerduty-my-service-critical
thresholds:
  critical: 0.05
  warning: 0.02
notify_no_data: false
evaluation_delay: 60

Maintenance Windows and Suppression

Suppression should be intentional and time-bound. The goal is to avoid false positives during known events, not to hide problems permanently.

Always suppress alerts during:

• Planned deployments (configure a @deploy downtime in Datadog for 15 min)
• Known maintenance windows
• Load tests (suppress only in non-prod environments)

Danger: ANTI-PATTERN

Never mute an alert without an expiry time.

“Just muting this for now” becomes a permanently silenced alert that lets the next real incident go undetected for hours.

Key takeaway: every suppression should have a reason, a scope, and an expiry.

Metrics Reference

Application Metrics

Database Metrics

Infra Metrics

Alert → Runbook Link (Required)

Every P1/Critical alert must link to a runbook. The runbook answers:

1. What does this alert mean?
2. How do I verify the issue?
3. What are the immediate mitigation steps?
4. When do I escalate?

Danger: ANTI-PATTERN

Alerting on every single 5XX individually with threshold = 1.

This creates alert fatigue, on-call engineers start ignoring pages because they’re too frequent. Aggregate over a time window and set a meaningful percentage threshold instead.

Danger: ANTI-PATTERN

Setting thresholds without baselining first.

Don’t set “CPU > 80%” without knowing your service normally runs at 60%. Set thresholds relative to your P95 baseline, not arbitrary round numbers.

Final Takeaways

• Alert on user-visible symptoms first, then use infra and database signals as supporting evidence.
• Assign severity based on impact and urgency, not habit.
• Use composite alerts and alert-fatigue metrics to keep pages meaningful.
• Tie every critical alert to a runbook so responders know what to do next.
• Review thresholds regularly as traffic patterns and service behaviour change.

Written by

Balaji G