Delays with notifications

Customers may have experienced up to 15 minute delays with notifications between 4:00 UTC and 4:55 UTC on Thursday, the 19th of January, 2023. 

We were alerted of an increase in queues on our notification workers at 4:12 UTC, and in an attempt to reduce these queues, more worker queues were added at 4:19 UTC. The additional worker queues did not mitigate the notification queues.

We continued to investigate and, at 4:38 UTC, identified many jobs attempting to retry due to unsuccessful delivery to many email subscribers.

At 4:55 UTC, the worker queues were dramatically reduced, and notifications were delivered as expected.

We resolved the incident at 5:13 UTC, and identified the root cause. 

We apologise for the inconvenience this has caused. As part of our post-incident review, we will make further improvements to the notification mechanism to prevent this type of incident from recurring.