Full Byte Developer.

How a DNS race caused the October 2025 AWS outage

AWS traced the incident to an empty DynamoDB DNS record and documented failures during dependent-service recovery.

Originally posted · Published here · 1 source

AWS attributed its October 2025 Northern Virginia disruption to a race condition in DynamoDB's automated DNS management. An outdated plan overwrote a newer one, and cleanup then removed the active plan. The regional endpoint was left without IP addresses. [1]

Check assumptions again before changing shared state

AWS's report says the process checked that its plan was newer before applying it. Delays meant that check was stale by the time the older plan took effect. The system then needed manual intervention to restore a usable state. [1]

For your own automation, examine the interval between checking a condition and committing a change. Ask what another worker can change during that interval and how the update rejects stale input.

Test recovery as well as normal operation

AWS also documented delayed recovery in dependent services and planned additional recovery testing and throttling changes. Restoring the original dependency did not immediately restore every affected operation. [1]

Include the backlog of failed or queued work in an incident exercise. Check whether retries overwhelm the recovering service and whether operators can access the tools they need while a dependency is down.

Keep this guide

By Full Byte Developer

Original post: Instagram post

Filed under Dev tools

Sources

Next in this stack