Since 06:25PM PT, 2023-07-22, our data processing has fully caught up. All downstream Analytics services are working with 100% of the data. There was no data loss during this incident and no further actions are required from our customers.
Posted Jul 22, 2023 - 18:37 PDT
Monitoring
We deployed a fix at 5:30PM PT and the data processing delay for the %5 partitions has been recovering since then. We will continue monitoring the recovery.
Posted Jul 22, 2023 - 17:48 PDT
Update
We are continuing to work on a fix for this issue. We will provide another update in 30 minutes.
Posted Jul 22, 2023 - 16:20 PDT
Identified
Starting at 2:00PM PT, 2023-07-22, we started experiencing data processing delay in our ingestion pipeline for ~5% of our partitions. This means 5% of the data across all customers since 2:00PM could be missing for all downstream systems.
We are actively mitigating the issue and will provide another update in 30 minutes. We expect no data loss and further actions are not required from our customers.