- Status Pages
- Message Distribution
- Website Plugin
- Wed, 21 Jun 2017
- 10:04:57 UTC
We're seeing delays akin to the ones we had yesterday, where changes to pages may take longer than expected to appear, and message delivery to your audience may be delayed.
Our monitoring was quicker to highlight this after the changes we made, however, we appear to still have a ghost in the machine which is causing the problem.
We're taking a closer look now, I'll update you as soon as we know more.
- 10:29:21 UTC
Things look to have returned to normal now folks. Thank you so much for understanding - letting you down twice in as many days is embarrassing.
On a positive note, I believe we now know where the issue lay, and we'll be patching this today.
I'll come back to you all once we've got something worked out.
- 13:46:51 UTC
Following up on this morning's message delays, it seems the main cause of the problem was a bug which meant that reording the components on a page flooded our job queue with lots of unnecessary jobs - causing a bottleneck which led to the delays.
We've just made some changes which should stop this from happening again.
If anyone has any technical questions about this then please let me know, I'll be only too happy to talk you through it.
Once again, thanks for all being so understanding.
[Resolved] Delays to Page Updates and Message DistributionThis incident lasted 24 minutes.
- Tue, 20 Jun 2017
- 09:26:09 UTC
Oh-oh, looks like we may have a few issues causing delays folks, we’re going to take a closer look, I’ll update you again shortly.
- 09:43:56 UTC
We've beefed up the infrastructure responsible for updating the status pages and distributing messages, and things seem to be looking much better now.
We're going to watch things closely for a little while and try to unpick what went wrong so we can prevent it from happening again.
I'll be in touch again shortly when we have some more useful insights.
- 09:58:43 UTC
After watching for the past 15 minutes I'm happy that things have returned to normal; page updates and message distribution are now as speedy as usual.
I've also made some headway on understanding the cause of the problem and putting together a strategy to prevent it in future. More on that to follow.
Thanks so much to everyone for being so understanding.
[Resolved] DelaysThis incident lasted 32 minutes.
- Past notices
- No further notices from the past 7 days.