On March 27, 2018 we experienced an outage. We apologize for any disruption this may have caused and want to be transparent about the situation. In most cases, data was not affected. If you suspect any irregularities in your data, please reach out to your CSM or email support@amplitude.com
At 16:35 PDT we deployed a code change to perform retry DB logic for one of our services. This retry logic was also picked up by our event ingestion service which caused excessive load on a critical database. The resulting slowdown in this database caused an outage in data collection and customers being able to access web reporting.
Data collection degraded performance started at 16:40 PDT and continued at varying levels until 19:05 PDT when event collection was back at 100% success rate.
During the period that data ingestion was down, events that were being sent to Amplitude were being rejected. This means that we did not accept any events during that time frame and data could have been lost. In most cases, data won’t be affected. However, there is a possibility of data loss, which we have outlined below.
There is a possibility of data loss whether you are sending data via our SDKs or HTTP API, depending on certain factors outlined below.
SDKs - there is default logic built into our SDK that stores up to 1000 events if a device goes offline; the device will then send the stored events once the device reconnects. Any events logged after 1000 will begin to displace the oldest event stored. All data has been preserved for users who performed 1000 or fewer events during the outage time frame and have reopened the software. Any user who performed more than 1000 events during the outage time frame will experience partial data loss. Users who have not come back online yet will also be missing data.
HTTP API - if you are using our HTTP API and have built a retry logic that continued to retry until after data collection recovered, there should be no impact on data. If you do NOT have retry logic built in, you can re-send the events that occurred during the time frame. PLEASE NOTE: because a subset of the events were still being accepted and processed, there will be a possibility of duplicated events if you were NOT sending an insertid to Amplitude during the time of the outage. If you were using an insertid, then we will deduplicate all events that have the same insertid. Unfortunately, there is no way for us to fix duplicates if you choose to resend events without an insertid.
Please check your data to see if there are any irregularities in your data during the outage time-frame. Because of differences in user usage patterns across customers, it’s difficult for us to accurately measure impact for any one customer.
If you think your data was impacted by the outage, please reach out to us and include your findings so we can verify and look into possible solutions.
The following is a detailed timeline of when systems were unstable:
Web reporting outages totalled 17 minutes and were during the following time frames:
Event collection outages were spread over 2 hours during the following time frames:
A project is underway to migrate the impacted DB to more fault-tolerant and scalable infrastructure with additional read replicas to handle unexpected load. We expect to complete this project by the end of this week.
We understand that our customers rely on us to reliably collect data about their product, and we apologize for the incident and any complications it has caused. We are actively working to address the specific causes of this outage to prevent it from happening again in the future.
Thank you for your support.