Our background processing queue is caught up and our search cluster has properly rebalanced. This incident has been resolved.
Posted Aug 30, 2019 - 11:40 GMT-05:00
A fix has been implemented and our search cluster and background processing jobs are recovering. We will continue to monitor the situation as the recovery progresses.
Posted Aug 30, 2019 - 09:52 GMT-05:00
At 12:30 UTC our background processing queue began backing up jobs. At 14:00 UTC we received a notification of degraded performance in the web app, caused by the growing backlog of unprocessed jobs. We have identified an issue with our search cluster's rebalancing behavior as the source of the backlog and are currently implementing a solution.