TrackVia Incident Report

Service Impairment/Outage

May 26,2021

Summary

On May 26, 2021 from 7:42 AM MT to 8:26 AM MT, TrackVia experienced performance degradation causing a 3 minute outage from 7:52 AM to 7:55 AM MT.  TrackVia Operations observed high CPU load affecting application performance. Additional processing capacity was added while TrackVia investigated the issue and full functionality was restored by 8:26 AM MT. 

Root Cause Analysis

An investigation of the event concluded optimization efforts conducted on the previous day left the database with a corrupted table cache leading to suboptimal performance. Application processing slowed eventually leading to a small outage. Trackvia Operations flush the table cache restoring database performance.

Corrective Actions

1. TrackVia Operations initiated an application autoscaling event adding two additional app servers in each Availability Zone.
2. TrackVia Operations forced a database failover
3. TrackVia Operations executed a restart of existing application servers.
4. TrackVia Operations placed account 21227 in maintenance mode to safe guard data integrity.
5. TrackVia Operations executed a flush tables against each database reader which ultimately returned database performance back to normal