Back to overview

Degraded DNS in Composer v4 US-EAST-1

May 17 at 03:46pm EDT
Affected services
Playback API (US-EAST-1)

Resolved
May 17 at 03:46pm EDT

Incident
DNS capacity went down to 50% in EKS cluster for Composer V4 in US-EAST-1 region

Impact
Slow response times and error responses for manifest requests.

Duration
~4 minutes

Resolution
Capacity automatically scaled back up to 100%

Reason
A bad EC2 instance was cycled automatically by Auto Scaling Group unexpectedly.

Changes
Increased redundancy and availability count of DNS capacity in each cluster.
Changed threshold of warning notifications in cases where manual intervention would be necessary.