Service Degradation on January 30, 2026

What Happened

On January 30, we deployed a network configuration change designed to block aggressive web crawlers from China that were placing unnecessary load on our infrastructure. Unfortunately, this change contained a bug that caused some requests to our service to be handled incorrectly.

During the affected period, some customers may have experienced:

• Status notice updates not being saved when submitted
• General performance degradation across the platform

Timeline

• 12:48 PM UTC: Configuration deployed to production
• 3:30 PM UTC: Issue discovered by our engineering team
• 4:32 PM UTC: Configuration rolled back, normal service resumed

Why This Happened

The bug slipped through our standard quality assurance process because it affected network infrastructure that wasn't covered by our existing automated testing. While our performance monitoring detected the degradation, the threshold wasn't set to trigger an immediate alert.

What We've Done

We've taken the following steps to prevent similar issues:

1. Improved monitoring thresholds to alert our team immediately when performance degrades
2. Expanded automated testing to include network-level infrastructure changes that were previously outside our QA scope
3. Conducted a full review of the change to understand what went wrong

Impact Assessment

We're grateful that no customers reported issues during this incident, suggesting the actual impact was minimal. Importantly, no data was lost - any failed form submissions would have been immediately apparent to users, who could simply retry.

Moving Forward

We've fixed the issue and improved our safeguards to prevent this from happening again. If you have any questions about this incident, feel free to reach out.