On February 4th, from approximately 7:56 am to 2:38 pm UTC, a temporary issue in our system caused a failure to load reports for projects hosted in our EU data center. During this period, all Mixpanel reports failed to load intermittently for these projects. Projects hosted in US and India were not affected.
The disruption resulted from a code change that broke the proper queuing of background queries in our system. This overwhelmed our compute cluster, and queries from projects in the EU cluster failed to complete. Once we identified the issue, the change was quickly reverted and reports resumed loading normally.
To mitigate future issues, we have added more alerts to our system to catch overload conditions sooner. Additionally, we are adding protection mechanisms to prevent this type of overload from reoccurring.
We are committed to delivering accurate, timely analysis to support your critical decision-making, and we prioritize fast, high-quality customer support. If you have any questions or concerns, please feel free to reach out.