On Friday, July 22 we experienced some platform instability which caused some queries from some customers to error. We apologize for any inconvenience this caused and want to thank you for your understanding as we work to improve our internal alerting to prevent a similar issue from occurring.

What happened?

On Friday, July 22, at approximately 3:15 PM PST, some dashboard query nodes died and some queries from customers began to fail. Amplitude engineers became aware of the issue at 4:55 PM PST and restarted the failing nodes at 4:58 PM PST. One node required a full reboot, and did not came back online until 5:26 PM PST. All errors stopped and the issue was resolved at 5:38 PM PST.

Why did it happen?

The incident was caused by a query node becoming unreachable and several other query nodes running out of memory. Processes on nodes that died did not start back up as they are expected to.

What are we doing to prevent it from happening again?

We have identified several steps to take to mitigate the impact that a multiple node failure has. We have already instrumented an alert to page the on call engineer when our error rate is high. We have also implemented automated email alerts for nodes becoming unreachable. We will expand our Pingdom check coverage. We are investigating why the processes running on query nodes did not restart as expected.

Maintaining a high level of platform stability critical for us to keep your business. We sincerely apologize for any inconvenience this has caused. Thanks for your support.