Data Ingestion Incident Postmortem

On Friday July 28, 2017 from 5:52 PM PDT to July 29th 1:38 AM PDT we experienced an outage that prevented data from being received by Amplitude servers. Beginning at 8:40PM PDT on July 28th we began accepting data at a reduced rate. By July 29th 1:38 AM PDT we started to accept all events. Due to this outage, some data was lost. Data processing remained delayed until July 29th 7:30 AM PDT.

What happened?

On Friday, July 28 at 5:30 PM PDT, an engineer began performing upgrades to our ingestion system. At 5:52 PM PDT, we detected issues in data collection and initiated a rollback. During the rollback, a critical component of the ingestion system failed. In the following 3 hours, we were able to fix and manually restart the component, restoring partial data collection at 8:40 PM PDT on July 28. Due to system load, full recovery of data collection was delayed until Saturday, July 29 at 12:07 AM PDT.

At this point, data processing was delayed by several hours. While catching up, a shared dependency between processing and collection became overloaded. This caused a second data collection outage between 1:25 AM PDT and 1:38 AM PDT on Saturday, July 29. At this point Amplitude began accepting and processing ALL data that we received. Data processing caught up at 7:30 AM PDT on July 29th.

Scope of impact

During the period that ingestion was down (July 28th 5:52 PM PDT - 8:40 PM PDT) all events that were sent to Amplitude were rejected. Beginning from July 28th 8:40 PM PDT until July 29th 1:38 AM PDT, we started accepting a percentage of the events that we received. From 1:38 AM onwards, all events were being accepted.

What is the extent of data loss?

There is a possibility of data loss whether you are sending data via our SDKs or HTTP API, depending on certain factors outlined below.

SDKs - There is default logic built into our SDK that stores up to 1000 events if a device goes offline; the device will then send the stored events once the device reconnects. Any events logged after 1000 will begin to displace the oldest event stored. All data has been preserved for users who performed 1000 or fewer events during the outage time frame (July 28, 2017 from 5:52 PM PDT to July 29th 1:38AM PDT) and have reopened the software. Any user who performed more than 1000 events during the outage time frame will experience partial data loss. Users who have not come back online yet will also be missing data.

HTTP API - If you are using our HTTP API and have built a retry logic that continued to retry until after data collection recovered (July 29th 1:38 AM PDT), there should be no impact on data.

If you do NOT have retry logic built in, you can re-send the events that occurred during the time frame. PLEASE NOTE: because a subset of the events were still being accepted and processed, there will be a possibility of duplicated events if you were NOT sending an insertid to Amplitude during the time of the outage. If you were using an insertid, then we will deduplicate all events that have the same insertid. Unfortunately, there is no way for us to fix duplicates if you choose to resend events without an insertid.

How can I tell if my data was impacted?

We encourage you to check to see if there were any noticeable dips in event volume during the outage time-frame. Because of differences in user usage patterns across customers, it is difficult for us to accurately measure impact for any one customer. For most customers, there will be on average a 2 to 3% decrease in event totals during the outage window.

If your investigation reveals that your data was impacted by the outage, please reach out to us and include your findings so we can verify and look into possible solutions.

Why did it happen?

This incident and the subsequent length of recovery were a result of a combination of factors.

• We were not able to accurately simulate production level load during development, causing unexpected behavior while performing upgrades on our production environment.
• We were unable to identify issues as quickly as we could have due to limitations in our instrumentation.
• We discovered a problematic default configuration that prevented us from using an automated restart. This forced us to perform a much slower manual process.
• We were not prepared for the impact of heavy load on the recovery of data collection servers after a prolonged outage, delaying full recovery of data collection.

What are we doing to prevent it from happening again?

There’s a lot to learn and improve on from this incident, and we’d like to share a few of the specific actions we’re taking to avoid future incidents like it.

• We are planning to build a load testing sandbox environment that can simulate production load on ingestion servers.
• We are aggressively improving metrics related to the data collection system to enable more rapid triaging in the future.
• We have fixed the problematic default configuration, and will be reevaluating our other default configuration settings
• We have added mechanisms to mitigate the impact of load on data collection servers during recovery.
• We have taken steps to fix the dependency between data collection and data processing. We will be performing a comprehensive audit of the dependencies of our data collection system.

We understand that our customers rely on us to reliably collect data about their product, and we apologize for the incident and any complications it has caused. We will do everything we can to improve our processes to ensure that you can rely on Amplitude in the future.

Thank you for your support.

Alan Ibrahim

Director of Customer Success