Incident Report

May 10, 2024

Summary

On May 8 at 8:07 AM MT, TrackVia Support notified the TrackVia Operations team that integration connections were being dropped. An investigation of the issue determined that a TrackVia vendor was reporting issues with their API service.TrackVia’s Vendor first reported an issue at 7:21 MT and resolved the issue. Full functionality was restored by 7:59 MT.

Root cause

TrackVia’s Vendor reports unexpected behavior of the network controller managing load balancers recreated some resources with invalid configurations.

Resolution and recovery

TrackVia’s Vendor corrected the mis-configuration and restored API operations by 7:59 MT.

Corrective and preventive measures

• TrackVia’s Vendor replaced the network controller which caused the mis-configuration.
• TrackVia added additional monitoring to identify vendor issues in a more timely manner. 

Incident timeline

• May 8, 2023 6:58 MT: Vendor update of a production cluster.
• May 8, 2023 7:17 MT: Vendor started receiving alerts for the internal API, Echo API and ActiveDocs Proxy.
• May 8, 2023 7:58 MT: Root cause identified and solution deployed.
• May 8, 2023 8:00 MT: Service is restored.