Partial System Outage - Commercial, Private and Gov Cloud
Incident Report for TrackVia
Postmortem

AWS Summary

12:14 PM PST Between 7:14 AM PST and 7:59 AM PST, customers experienced elevated network packet loss that impacted connectivity to a subset of Internet destinations. Traffic within AWS Regions, between AWS Regions, and to other destinations on the Internet was not impacted. The issue was caused by network congestion between parts of the AWS Backbone and a subset of Internet Service Providers, which was triggered by AWS traffic engineering, executed in response to congestion outside of our network. This traffic engineering incorrectly moved more traffic than expected to parts of the AWS Backbone that affected connectivity to a subset of Internet destinations. The issue has been resolved, and we do not expect a recurrence

Posted Dec 15, 2021 - 15:22 MST

Resolved
AWS is reporting the issue is resolved and TrackVia has confirmed all services are processing normally.
Posted Dec 15, 2021 - 09:31 MST
Update
All TrackVia environments are now reachable. We are continuing to monitor the situation for additional impact.
Posted Dec 15, 2021 - 09:09 MST
Monitoring
AWS has identified Internet connectivity Issues in their California and Oregon sites. TrackVia continues to monitor the situation and will provide updates as more information becomes available.
Posted Dec 15, 2021 - 08:59 MST
Update
The TrackVia commercial environment is now experiencing the same issue.
Posted Dec 15, 2021 - 08:51 MST
Investigating
TrackVia infrastructure is temporarily unreachable for some of our customers. We are investigating the issue and will post updates as we work towards a solution.
Posted Dec 15, 2021 - 08:42 MST
This incident affected: Commercial Cloud, Private Cloud, and Gov Cloud.