The issue was resolved since 10:40am, here is our investigation and the team is working on the follow ups to make sure it won't happen again.
Root cause: A metadata service that our query engine depended on got into a bad state because of spammy users. Restarting the service did not help because queries involving these spammy users still ran intermittently before 11:30am PT.
Actions taken: At 11:30am PT, we surgically blocked these problematic queries from running.
Actions taking now: We are implementing more monitoring in the metadata service to detect similar load issues more proactively. We are also improving our failure recovery process so that we can minimize the impact of this service’s downtime.
Posted May 01, 2023 - 16:02 PDT
Monitoring
We identified the issue and made a fix, the system was back starting 10:40am. We are still monitoring the system to make sure it's working correctly.
Posted May 01, 2023 - 11:31 PDT
Update
We are continuing to investigate this issue.
Posted May 01, 2023 - 11:01 PDT
Investigating
Our query service is experiencing intermittent partial outage, customer's charts might load slowly or time out. The issue started around 9:40am PT.
a) Our ingestion systems are working as expected and we are not losing any data. b) All of our customers are impacted. c) Impacted customers will experience slow loading times or errors when loading Amplitude web-pages.
We are investigating the issue and will post an update after 30 minutes or earlier if we identify the issue.
Posted May 01, 2023 - 11:01 PDT
This incident affected: Analytics (Web Reporting).