500s on Dashboard

Incident Report for Amplitude

Postmortem

Recently our dashboards have experienced some downtime where queries would talk a long time or never complete. This was due to a significant increase in volume of queries sent to our servers (many via our REST API), that caused request queuing in our system.

A fix that was meant to redistribute query load more effectively had a bug that caused a 500 error instead when heavy load occurred, and affected dashboards intermittently on April 10, 10AM-4PM UTC. A new fix has been pushed that should properly redistribute the query load, and we are closely monitoring the situation for other issues.

We sincerely apologize for any inconvenience this may have caused. Our engineers are working hard to scale our systems to support the increased query load we have been experiencing. We're working on improving the system in two ways: first, by reducing the impact others' queries have on your query response time, and second by speeding up the time it takes for any one particular query to run.

Posted Apr 10, 2015 - 09:50 PDT

Resolved

A fix has been pushed to resolve the 500 errors.
Posted Apr 10, 2015 - 09:49 PDT

Identified

We're aware of recent slowness and 500s on the dashboard and are working on a fix. More information shortly.
Posted Apr 10, 2015 - 08:55 PDT