What happened: - On Friday, June 9, 2023 at 6:40pm PST, code with a critical bug in Funnel charts was pushed to production. Any users that visited a Funnels chart from June 9, 2023 at 6:40pm PST - June 11, 2023 at 7:10am PST saw the “Something went wrong” error on their funnel charts.
What was the resolution: - On Sunday, June 11, 2023 at 7:05am PST, our on-call engineering team discovered the bug with funnel charts and rolled back the change within 5 minutes, at 7:10am PST. This fixed the issue for all customers at that point in time. - A subsequent fix was created and tested and finally deployed 1h30m later to production with all the changes added in the release on Friday. - Since this put the customer’s UI in a bad state, we ask all customers to refresh Amplitude to restore a clean state for all charts.
What is amplitude going to do in the future to prevent this from happening again? - Short-term: Set up additional monitoring to alert on-call teams of extended front-end sentry errors for funnels and any other uncovered charts - Medium-term: Address the root cause of the issue relating to some typing gaps in our current codebase. - Medium/Long-term: Additional e2e test coverage to catch this specific regression + more broad coverage to catch similar regressions in the future.