From 11:25 PM PDT Sept 18, 2023 until 4:45 PM PDT Sept 19, 2023, our customers experienced high latency of up to 12 hours in all our products and services. This issue started by an AWS networking incident that disabled all our services except our data reception and lasted until 11:05 PM on Sept 18th. From that time, we worked on processing our 12 hours of backlog which were fully cleared by 4:35 PM Sept 19th. All our services are working normally and there was no data loss during this incident. Further actions are not required from our customers.
Posted Sep 19, 2023 - 17:12 PDT
Update
As of now, about 99% of realtime data are showing up in charts and dashboards. Full recovery of data is expected to be before 4:30 PM PDT including data warehouse exports.
Posted Sep 19, 2023 - 15:23 PDT
Update
As of now, about 85% of realtime data are showing up in charts and dashboards. Full recovery of data is expected to be before 4:30 PM PDT, and data warehouse exports will catch up before 5:30 PM PDT.
Posted Sep 19, 2023 - 13:26 PDT
Update
As of now, about 70% of realtime data are showing up in charts and dashboards. Full recovery of data is expected to be before 7:30 PM where data warehouse exports will catch up before 8:30 PM.
Posted Sep 19, 2023 - 12:51 PDT
Update
We are still working on speeding up the rate of recovery. However, our current estimated time will be 18 PDT for all services. We will update the estimated times every other hour.
Posted Sep 19, 2023 - 10:02 PDT
Update
The underlying problem has been fixed, and our systems are working through the backlog. Customers should not see any data loss from this issue - We expect to be fully caught up with ingestion by 16:00 PT - Export systems will recover shortly after ingestion as the system catches up - Reporting systems are expected to be fully caught up by 18:00 PT. in the meantime, customers may not see recent data in their charts/dashboards
We will give updates throughout the day as our systems make progress
Posted Sep 19, 2023 - 08:40 PDT
Update
All our components and services are working as expected. We are keeping this incident open and our on-call team are monitoring the systems to make sure there will be no surprises during the night. We will resolve this incident after 6-8 hours of normal execution.
Posted Sep 19, 2023 - 00:11 PDT
Monitoring
As of 11:25 PM PDT Sept 18, 2023, the AWS incident is resolved and we are resuming our components and services. There will be a large latency for the data to get processed and we expect to catch up in the next 10 hours. Most other services are recovered. Please see the updated list of affected services below.
Posted Sep 18, 2023 - 23:49 PDT
Update
AWS issue is not resolved. We are still monitoring the status and trying to find a workaround using a few AWS services that have recovered.
Posted Sep 18, 2023 - 23:02 PDT
Update
AWS Services started to recover but not completely. We are still monitoring and waiting for AWS to resolve their outage.
Posted Sep 18, 2023 - 22:34 PDT
Update
We are still waiting for AWS to resolve their networking issue.
Posted Sep 18, 2023 - 21:15 PDT
Update
We are still waiting for AWS to resolve their networking issue.
Posted Sep 18, 2023 - 19:01 PDT
Update
We are still waiting for AWS to resolve their networking issue. We will update here as soon as AWS problem is resolved.
Posted Sep 18, 2023 - 16:32 PDT
Update
On September 18 at 11:43am PDT, AWS identified a networking issue in their cloud and are working to mitigate the issue.
At this time, we do not see any impact to data reception. However, most other services are affected. We do not expect any data loss. No action is required from our customers at this time.
Posted Sep 18, 2023 - 12:35 PDT
Update
We are continuing to work on a fix for this issue.
Posted Sep 18, 2023 - 12:18 PDT
Identified
On September 18 at 11:43am PDT, AWS identified a networking issue in their cloud and are working to mitigate the issue.
At this time, we do not see any impact to data reception. However, most other services are affected. We do not expect any data loss. No action is required from our customers at this time.
Posted Sep 18, 2023 - 12:15 PDT
This incident affected: Analytics (Web Reporting, Web Application), Experiment (Evaluation API, Management API, Web Application), Audiences (Profile API, Web Application), and Data (Web Application, Data Processing, Cohort Export, Event Streaming, Data Export).