What happened: At 17:57 PDT June 6th, a regression was introduced to the percentile formula computation logic. Between 17:57 PDT June 6th and 01:33 PDT June 7th, charts measured with the percentile formula were all returning 0s incorrectly. This affected all customers who were using the percentile formula during the timeframe.
What was the resolution: - Our on-call engineering team rolled back the bad change after discovering the bug. - We have fixed the issue and added additional tests in the following day. - We ask all customers to refresh their percentile formula charts to purge potentially cached results.
What is amplitude going to do in the future to prevent this from happening again? - Short term: we added test for this specific feature. - Medium term: we are reviewing our test cases and adding more tests to ensure high test coverages around critical features. - Medium/Long-term: We are working on an auto regression detection framework. This framework will help us identify and address long-tail corner cases, ensuring that we can prevent similar issues from occurring in the future.