Production API and Dashboard Latency Increase

Incident Report for Alloy

Resolved

From 12:05 to 12:19 ET today, additional load on our production database caused a significant increase in processing time for dashboard and API actions. Once the load was targeted and addressed, latency went down immediately and is now back to normal. Our infrastructure team is currently working on addressing the root cause of the problem and making sure we have no reoccurrences of similar issues.
Posted Mar 31, 2022 - 13:04 EDT

Update

We are continuing to monitor for any further issues.
Posted Mar 31, 2022 - 12:23 EDT

Monitoring

We have identified the latency issue and are now monitoring the fix to make sure there are no further degradations.
Posted Mar 31, 2022 - 12:23 EDT

Update

We're seeing increased latency on certain queries from our core database. We are currently working on identifying the issue and mitigating the latency
Posted Mar 31, 2022 - 12:17 EDT

Investigating

Our API integration tests have encountered an increase in errors. We are currently investigating. Stay tuned for updates.
Posted Mar 31, 2022 - 12:09 EDT
This incident affected: Production API, Customer Dashboard, and Sandbox API.