From 9:24 AM PT the Amplitude websites were not available to all our customers impacting their ability to load charts or create new charts. Additionally, our customers also experienced latency in our data processing component. The website issue was resolved by 10:05 AM PT and the real-time data processing component was mostly recovered by 12:50 PM PT. There was no data loss during this incident and further customer action is not required.
The root cause for this incident was a database replication problem shared by our systems causing a number of services to fail. We have recovered our databases and they are working as expected.
Posted Apr 28, 2023 - 12:53 PDT
Update
We are monitoring recovery of the delay in our ingestion system which continues to recover. No data has been lost.
Posted Apr 28, 2023 - 12:43 PDT
Update
We are monitoring recovery of the delay in our ingestion system. No data has been lost. More than 90% of data is delayed less than 10 minutes. The remaining data ingestion delay is continuing to recover.
Posted Apr 28, 2023 - 12:05 PDT
Update
Data ingestion continues to recover, however we expect full recovery closer Noon PST. No data has been lost.
Posted Apr 28, 2023 - 11:26 PDT
Monitoring
We are monitoring the recovery of our data ingestion. No data has been lost. Recovery has been progressing as expected. We expect data ingestion to be fully recovered by 11:25AM PST.
Posted Apr 28, 2023 - 11:10 PDT
Update
Our data ingestion delay is recovering. No data has been lost and charts are loading in the web application. We expect data ingestion to be fully recovered within 1 hour (11:25AM PST).
Posted Apr 28, 2023 - 10:25 PDT
Update
Our data ingestion delay is recovering. No data has been lost. We will update again as the ingestion continues to recover.
Posted Apr 28, 2023 - 10:18 PDT
Update
Since 9:24AM PST, Amplitude began to experience degraded performance on the Amplitude Website.
We identified the root cause and resolved the core underlying issue. Since 10:05 PST, charts are loading and we are actively monitoring the performance of the web application.
Data ingestion is still delayed, no data been has lost. We are monitoring the recovery of the data ingestion and web application will provide another update soon.
Posted Apr 28, 2023 - 10:11 PDT
Update
Since 9:24AM PST, Amplitude began to experience degraded performance on the Amplitude Website. Charts are not loading and data ingestion is delayed. There has been no data loss due to this incident. We are actively investigating the issue and will provide an update once the root cause is identified.
Posted Apr 28, 2023 - 10:02 PDT
Update
We are continuing to investigate this issue.
Posted Apr 28, 2023 - 09:55 PDT
Investigating
Since 9:24AM PST, we started experiencing degraded performance on the Amplitude Website. Charts are not loading and data ingestion is delayed. We are actively investigating the issue and will provide an update once the root cause is identified.
Posted Apr 28, 2023 - 09:55 PDT
This incident affected: Analytics (Web Reporting, Web Application), Data (Data Processing), and Experiment (Web Application).