Vendor Incident - API services
Incident Report for TrackVia

Incident Report

May 10, 2024


On May 8 at 8:07 AM MT, TrackVia Support notified the TrackVia Operations team that integration connections were being dropped. An investigation of the issue determined that a TrackVia vendor was reporting issues with their API service.TrackVia’s Vendor first reported an issue at 7:21 MT and resolved the issue. Full functionality was restored by 7:59 MT.

Root cause

TrackVia’s Vendor reports unexpected behavior of the network controller managing load balancers recreated some resources with invalid configurations.

Resolution and recovery

TrackVia’s Vendor corrected the mis-configuration and restored API operations by 7:59 MT.

Corrective and preventive measures

  • TrackVia’s Vendor replaced the network controller which caused the mis-configuration.
  • TrackVia added additional monitoring to identify vendor issues in a more timely manner. 

Incident timeline

  • May 8, 2023 6:58 MT: Vendor update of a production cluster.
  • May 8, 2023 7:17 MT: Vendor started receiving alerts for the internal API, Echo API and ActiveDocs Proxy.
  • May 8, 2023 7:58 MT: Root cause identified and solution deployed.
  • May 8, 2023 8:00 MT: Service is restored.
Posted May 10, 2024 - 12:00 MDT

A TrackVia vendor experienced a service disruption in the ActiveDocs Proxy which may have caused interruption with TrackVia api services. The disruption lasted for approximately 38 minutes and was operational by 7:59 AM MT.
Posted May 07, 2024 - 07:00 MDT