TL;DR: A bug in DynamoDB’s internal DNS automation in N. Virginia (us-east-1) temporarily deleted the DNS plan for its regional endpoint. That broke connections to DynamoDB and cascaded into problems launching new EC2 instances, health checks for Network Load Balancer (NLB), and several higher-level services (Lambda, ECS/EKS/Fargate, Connect, Redshift). AWS restored core DNS by ~5:25 AM ET and fully stabilized affected services by the afternoon of Oct 20. AWS has disabled the faulty automation globally and is hardening multiple systems to prevent a repeat.
One-Paragraph Summary
Late Oct 19 Pacific (early Oct 20 Eastern), DynamoDB in us-east-1 hit a latent race condition in its DNS management system that led to an empty DNS record for the public regional endpoint. Clients (including other AWS services) couldn’t resolve or connect to DynamoDB until DNS was manually restored. That initial failure then rippled: EC2 couldn’t launch new instances for hours, NLB health checks flapped and removed capacity, and services that depend on those layers saw errors until backlogs were cleared and throttles were removed.
The Root Cause (Human-speak)
DynamoDB uses an internal DNS Planner (makes “plans”) and DNS Enactors (apply them in Route 53) across multiple AZs for resilience. A rare timing race let an older DNS plan overwrite a newer one, and then an automated cleanup deleted the (now-active) older plan—wiping the endpoint’s IPs and blocking further automated fixes. Engineers intervened to repair the DNS state.
Eastern Time Timeline (Oct 20, 2025)
- 2:48 AM–5:40 AM ET: DynamoDB API errors in us-east-1 due to DNS failure. DNS repaired around 5:25 AM ET; clients recovered as cached DNS expired by 5:40 AM ET.
- 2:30 AM–4:50 PM ET: New EC2 launches failed or were throttled (existing instances stayed healthy). Network configuration backlogs cleared by early afternoon; full EC2 recovery by 4:50 PM ET.
- 8:30 AM–5:09 PM ET: NLB saw increased connection errors driven by health-check behavior during the EC2/network backlog; failover was temporarily disabled to restore capacity, then re-enabled after recovery.
- ~2:51 AM–5:15 PM ET: Lambda experienced API/invocation errors and throttling while capacity was constrained; backlogs drained by 5:15 PM ET.
- 1:45 AM–5:20 PM ET: ECS/EKS/Fargate had task/launch and scaling failures; recovered by 5:20 PM ET.
- 2:56 AM–4:20 PM ET: Amazon Connect had elevated errors for calls/chats/tasks tied to NLB and Lambda issues.
- Overnight window: Redshift also saw a separate IAM-related issue for queries using IAM user credentials (global effect); customers using local users were unaffected.
Note on dates/times: AWS’s post is in PDT; we’ve converted key moments to ET for readability. The incident spans late Oct 19 PDT (early Oct 20 ET) through Oct 20.
Who/What Was Affected
- DynamoDB (us-east-1): New connections failed until DNS was fixed. Global Tables replicas in other Regions continued but had replication lag to/from us-east-1 during the outage.
- EC2 (us-east-1): Couldn’t launch new instances for hours; existing instances kept running. Network state propagation backlogs delayed connectivity for newly launched instances.
- NLB (us-east-1): Health checks flapped, reducing available capacity and causing connection errors for some load balancers.
- Lambda, ECS/EKS/Fargate, Connect (us-east-1): Elevated errors and throttling while EC2/NLB recovered.
- Redshift (all Regions): Temporary inability to execute queries via IAM user credentials due to a Redshift defect coupled to an IAM API in us-east-1; local users unaffected.
What AWS Says They’re Changing
- DynamoDB DNS automation: Disabled worldwide until fixed; adding guards against incorrect plan application and this specific race.
- NLB: Adding “velocity control” to limit how much capacity can be removed during health-check-triggered failover.
- EC2: New recovery test suite for the DropletWorkflow Manager (DWFM) path and improved throttling based on queue depth to avoid “congestive collapse.”
Practical Takeaways for Builders
- Design for Regional Isolation & Failover: If you’re single-Region in us-east-1, consider multi-Region or at least zonal isolation for critical data paths (e.g., DynamoDB Global Tables, cross-Region read fallbacks).
- Cache & Timeouts: Be mindful of DNS caching behavior (client, OS, and resolvers). Shorter TTLs help recovery but can increase dependency on DNS during incidents. Balance accordingly.
- Warm Capacity & Queues: Where possible, maintain warm pools or buffered capacity so launches aren’t your single point of recovery. Also ensure idempotent, retry-safe job queues so backlogs can drain safely post-incident.
- Health Checks With Grace: Aggressive health-check failover can remove too much capacity during partial control-plane impairment. Consider grace windows and multi-AZ balancing behavior when tuning.
- Runbooks for “Control-Plane Slow” Days: Prepare operational paths for when the control plane (launches, config propagation) is slow/unavailable. Prefer designs that keep existing data planes healthy and isolated from launch/config storms.
Source
- AWS Post-Event Summary: “Summary of the Amazon DynamoDB Service Disruption in Northern Virginia (US-EAST-1) Region.” https://aws.amazon.com/message/101925/
