Our data processing systems have completely caught up. Due to this incident, between 6:30AM PST and 2:30PM PST data processing was delayed by up to 5hrs minutes. No data was lost during the incident and no further action is required from our customers.
Posted Aug 14, 2020 - 14:53 PDT
Update
Data processing system is still recovering. Data received through '/batch' and S3 is experiencing processing delay of up to 2hrs, data received from other sources (ex: SDK, httpapi) is completely caught up.
Posted Aug 14, 2020 - 13:45 PDT
Update
Data processing system is recovering. Data received through '/batch' and S3 is experiencing processing delay of up to 5hrs, data received from other sources (ex: SDK, httpapi) is completely caught up.
Posted Aug 14, 2020 - 12:16 PDT
Monitoring
Data processing system is recovering. Data received through '/batch' and S3 is experiencing processing delay of up to 4hrs, data received from other sources (ex: SDK, httpapi) is experiencing processing delay of up to 30min.
Posted Aug 14, 2020 - 10:59 PDT
Identified
Issue was caused by underlying hardware. We are replacing it.
Posted Aug 14, 2020 - 10:10 PDT
Investigating
Our data processing systems are delayed because of a database issue. This incident started at 6:30AM PST.
Current status: a) Our ingestion systems are working as expected and we are not losing data. Our processing systems are not processing newly ingested data. b) All of our customers are impacted. Customers will see delayed metrics up to 2 hours.
We are investigating the issue and will post an update in an hour or earlier if we identify the issue.