A number of customers were unable to access the Amplitude web application between 7:23am PST and 8:13 am PST on 2017-05-10 due to issues with authentication.

WHAT HAPPENED?

Around 7:23 am PST our account database which is used to authenticate users to our web application became overloaded. This caused all login requests to our web servers to timeout when connecting to the database and return server errors (500). We identified the queries that were causing the issue around 7:50 am PST and optimized those queries to not overload the database. The issue was fully resolved by 8:13am PST. Data collection was not impacted and no customer data was lost during this incident.

WHAT ARE WE DOING TO PREVENT IT FROM HAPPENING AGAIN?

We have added additional monitors on the accounts database that will page the on-call member if the CPU usage becomes too high. We are also going to also add more monitors around query latencies on this database. We will also be improving our automated downtime detection systems to add additional checks for authentication and emulate the login flow that our customers go through.

Maintaining a high level of platform stability is critical for us to keep your business. We sincerely apologize for any inconvenience this has caused. Thanks for your understanding and support.