Postman Test notice appeared on some Status Pages

Today, we saw a notice with the subject line "Postman Test Notice" appear on several customer status pages, but no notifications were sent to subscribers. This resulted from a bug that slipped through our usual QA process.

That bug has been patched, and we've improved our automated test coverage to ensure it doesn't happen again.

We're sorry for the embarrassment this caused.

Here is how events unfolded:

Wednesday, 24th July

11:30: A deployment was made containing a bug that, under certain circumstances, allowed a test notice to be posted to several customer status pages.

Our automated testing on this code was not comprehensive enough to catch the bug before it was deployed.

The bug lay dormant and undetected for several days.

Friday, 26th July

05:02: An automated process in our QA system ran, triggering the dormant bug. A test notice was posted to several customer status pages, but no notifications were sent to subscribers.

09:13: The Sorry team spotted the suspicious test notice on a customer status page.

10:02: The test notice is removed from customer status pages.

10:30: Work began reproducing the bug and determining the root cause.

14:02: A patch is deployed to production to prevent further notices from being posted to pages other than their intended target.

This patch also included some additional automated tests to ensure the code continues to behave as expected.