Performance Issues
Incident Report for TrackVia

TrackVia Incident Report

Service Impairment/Outage

May 26,2021


On May 26, 2021 from 7:42 AM MT to 8:26 AM MT, TrackVia experienced performance degradation causing a 3 minute outage from 7:52 AM to 7:55 AM MT.  TrackVia Operations observed high CPU load affecting application performance. Additional processing capacity was added while TrackVia investigated the issue and full functionality was restored by 8:26 AM MT. 

Root Cause Analysis

An investigation of the event concluded optimization efforts conducted on the previous day left the database with a corrupted table cache leading to suboptimal performance. Application processing slowed eventually leading to a small outage. Trackvia Operations flush the table cache restoring database performance.

Corrective Actions

  1. TrackVia Operations initiated an application autoscaling event adding two additional app servers in each Availability Zone.
  2. TrackVia Operations forced a database failover
  3. TrackVia Operations executed a restart of existing application servers.
  4. TrackVia Operations placed account 21227 in maintenance mode to safe guard data integrity.
  5. TrackVia Operations executed a flush tables against each database reader which ultimately returned database performance back to normal
Posted Nov 09, 2021 - 16:38 MST

This incident has been resolved.
Posted May 26, 2021 - 08:40 MDT
TrackVia is currently investigating a performance issue in our Commercial cloud environment. No data has been lost and the problem should be resolved shortly.
Posted May 26, 2021 - 07:57 MDT
This incident affected: Commercial Cloud.