Amplitude Experiment Website Outage

Incident Report for Amplitude

Postmortem

Incident summary:

partial outage 23 mins (11:32AM - 11:55AM): experiment/flag pages showed empty pages (only rendering side nav)
site down 15 mins (11:55AM - 12:10AM): Showed “Something went wrong”

Detailed timeline:
11:32AM deployed #574 sklyab-ui : 400 errors for flag/experiment pages
11:55AM rolled back to #573 skylab-ui: site down (cors error)
12:10AM redeployed #573 skylab-ui (with correct image name): back up online

‌

Analysis:

Paritial outage:

#574 has dependency to skylab-backend for graphql query for new field which hasn’t been deployed prior to

Site outage:

Rollback on “Production Release” pipeline in spinnaker incorrectly sets image name and caused CORS error
Manually input skylab image name and reploying resolved the issue

Posted Apr 27, 2023 - 13:21 PDT

Resolved

Our website was PARTIALLY down. The issue started around 11:32AM and has been resolved by 12:10PM on 4/27/2023.

a) Our API endpoints are working as expected and no data loss is detected.
b) MOST of our customers are impacted.
c) Impacted customers will experience slow loading times or errors when loading Experiment UI.

Posted Apr 27, 2023 - 11:30 PDT