Issue has resolved. Both AWS eu-west-1 lag zones A and B are now showing operational. NetApp CVS Storage is now operational.
Posted Oct 13, 2021 - 06:26 UTC
Monitoring
One side zone B has come up and connectivity should be restored for all customers in the region now and volume mounts should be working now. However, complete recovery is still underway. We are continuing to monitor this.
Posted Oct 13, 2021 - 05:47 UTC
Update
10:20 PM PDT - DX Update – We have been able to restore one of the three DX devices that were down during this event. We are now monitoring the device to ensure it remains stable. In parallel, we have made progress on restarting one of the chillers and are observing reductions in temperature. The onsite teams are now starting to work on the other chiller.
Posted Oct 13, 2021 - 05:34 UTC
Update
Update from AWS Side:
10:00 PM PDT - DX Update – While we saw initial recovery of temperatures in the facility, we experienced a chiller restart failure. Onsite team is currently troubleshooting the issue. We are now exploring if we can restore one of the DX devices safely given the initial improvements we’ve seen, while the onsite team troubleshoots the chiller issue. We do not have an ETA for chiller restoration and hence do not have an ETA for full recovery.
Posted Oct 13, 2021 - 05:24 UTC
Identified
Issue with AWS Direct Connect Lags: Lags in both zones A and B down causing complete dataplane outage for CVS customers in AWS CVS eu-west-1 region.
We have received the following update from AWS Support:
"Description
[6:05 PM PDT] We are investigating packet loss, impacting AWS Direct Connect connectivity between the Eircom Clonshaugh, Dublin, IRL location and the EU-WEST-1 Region. [7:48 PM PDT] Starting at 6:01 PM PDT, we started experiencing elevated packet loss impacting traffic between the AWS Direct Connect location at the Eircom Clonshaugh, Dublin location and the EU-WEST-1 Region. We have determined the root cause and are working to mitigate the issue. Customers should fail over to their redundant AWS Direct Connect connections or use AWS VPN to restore connectivity to their resources in the EU-WEST-1 Region. [9:00 PM PDT] We have determined the root cause as a thermal event caused due to a failure in our cooling system and are actively working to restore the cooling system at the location. Customers should fail over to their redundant AWS Direct Connect connections or use AWS VPN to restore connectivity to their resources in the EU-WEST-1 Region."