TrackVia Incident Report -

Service Disruption/ Elongated Maintenance Window

November 7 2021

Summary

TrackVia entered a planned 60 minute maintenance window on November 11,2021 at 9:00 AM MT to deliver new application software. Upon completion of the maintenance activities, TrackVia Operations identified unusual application performance issues causing some client sessions to eventually timeout. A decision was made to revert the software changes restoring full performance functionality by 12:34 PM MT.

Root Cause Analysis

An investigation of the event showed that user authentication changes in the new version software contained a query with suboptimal performance only seen in high volume environments.

Corrective Actions

TackVia extracted several application dump files for debugging purposes.
TrackVia placed the site into maintenance mode and restored the authentication database from a backup taken prior to the software release.
TrackVia reverted all customer accounts to the previous software version.
TrackVia Development Team is working on a correction to the new release.

Posted Nov 09, 2021 - 14:53 MST

Completed

The maintenance for the Commercial environment is complete.

Posted Nov 07, 2021 - 12:59 MST

Update

We are continuing to verify the maintenance items.

Posted Nov 07, 2021 - 12:52 MST

Update

We are continuing to verify the maintenance items.

Posted Nov 07, 2021 - 12:39 MST

Update

TrackVia is going into emergency maintenance to back out changes made this morning. We expect down time to be no more than 30 minutes. We will continue to post updates as information becomes available.

Posted Nov 07, 2021 - 11:59 MST

Update

TrackVia is investigating application performance issues. We will provide additional updates as more information becomes available.

Posted Nov 07, 2021 - 10:41 MST

Verifying

TrackVia has completed changes and is verifying results.

Posted Nov 07, 2021 - 09:39 MST

In progress

Scheduled maintenance is currently in progress. We will provide updates as necessary.

Posted Nov 07, 2021 - 09:00 MST

Scheduled

The scheduled maintenance will take place on Sunday, November 7th starting at 9am MST and lasting for approximately 60 minutes. During this maintenance window, your account will be placed into Maintenance Mode and will be unavailable for one-hour to you or your users. We selected this time frame in hopes of minimizing any disruption. The maintenance occurring is necessary for us to provide new, flexible ways for your users to log into the system in the near future.

For system performance reasons we recommend not syncing offline Android and iOS devices during this one hour window.

Should you have any questions or concerns, please reach out to TrackVia Support.

Posted Oct 25, 2021 - 17:01 MDT

This scheduled maintenance affected: Commercial Cloud.