Temporary Data Ingestion Delay and Degraded Export & Query API Performance
Incident Report for Mixpanel
Resolved
This incident has been resolved.
Posted Nov 20, 2024 - 21:30 PST
Monitoring
Updating the status to monitoring.
Posted Nov 20, 2024 - 11:21 PST
Update
The data ingestion backlog has been processed. Events should now be up to date. We will continue to monitor the situation. If you are still experiencing a data delay or have any questions, please contact support.
Posted Nov 20, 2024 - 11:08 PST
Update
our engineering team continues to work on the backlog of events and profile updates. We'll provide an additional update within an hour on the progress and status.
Posted Nov 20, 2024 - 08:55 PST
Update
the team is seeing a much improved rate to catch up with events and profiles in the queue. The current rate should indicate catching up with real time in about an hour. We will keep monitoring and update the page accordingly.
Posted Nov 20, 2024 - 06:23 PST
Update
Engineering continues to work on the backlog; we'll provide an update in the timeline as soon as we have additional news
Posted Nov 20, 2024 - 05:17 PST
Identified
The new solution implemented looks to be working correctly. As we have a queue of pending event and profile updates to process, the current estimation is about 2 hours before it catches up. We'll keep updating as we work through it.
Posted Nov 20, 2024 - 02:56 PST
Investigating
The solution implemented has helped on the storage side but we're still investigating another issue causing the delays. The team doesn't have an updated at this time but we'll keep working on the issue and update the status as soon as possible
Posted Nov 20, 2024 - 01:44 PST
Identified
A solution is being implemented and the current estimated time for events to catch up to real time is about an hour. The engineering team is actively working and monitoring this so we'll keep this page updated with updated timelines. Thanks for your patience.
Posted Nov 20, 2024 - 00:07 PST
Update
The current incident is caused by an issue with one of our service providers. The engineering team has identified that the Export and Query APIs are affected as well. Data collection is not affected, so no data is being lost. The engineering team is currently implementing a workaround while they wait for the service provider to resolve the issue on their end. We really appreciate your patience in the meantime. If you have any questions, please contact support.
Posted Nov 19, 2024 - 21:43 PST
Investigating
We are experiencing delays with our data ingestion pipeline for US and EU Projects. As a result, real-time data for a subset of projects is delayed. No data is being lost. We appreciate your patience while our engineers work to restore real-time functionality. If you have any questions, please contact support.
Posted Nov 19, 2024 - 18:26 PST
This incident affected: Website, Query API, and Data Export API.