Track API Endpoint Partially Unavailable
Incident Report for Mixpanel
Postmortem

On January 8th, from approximately 3:14 pm UTC to 6:00 pm UTC, a temporary service disruption caused degraded performance for our Ingestion APIs. During this period, Mixpanel was unable to accept certain data sent through the Ingestion API endpoints, including event data (via /track and /import), user profile data (via /engage), and group profile data (via /groups).

This disruption resulted from an outage with one of our upstream service providers. Following their recovery, our Ingestion APIs resumed normal functionality. The disruption primarily affected projects hosted in our US data center. Projects hosted in EU and India through our Data Residency Programs were unaffected.

Potential Action Required

Data sent during this period of time that resulted in an error was not stored on our end and may require resubmission.

  • If you have already implemented the recommended retry behavior, your requests were likely ingested once the services were restored and no additional action is necessary.
  • If you have not implemented the recommended retry behavior, you will need to resend data for the affected period. The data you are resending includes any events already seen in Mixpanel, so please ensure those events maintain the same $insert_id, time, distinct_id, and event name so that Mixpanel properly deduplicates the data. If you have not already implemented automatic retry behavior for your ingestion pipeline we would highly recommend doing so.

We are committed to delivering accurate, timely analysis to support your critical decision-making, and we prioritize fast, high-quality customer support. If you have any questions or concerns, please feel free to reach out.

Posted Jan 10, 2025 - 15:51 PST

Resolved
This incident has been resolved.
Posted Jan 08, 2025 - 11:37 PST
Update
This incident has been resolved.
Posted Jan 08, 2025 - 11:35 PST
Update
A fix has been deployed to restore the /import, /track, and /engage ingestion API endpoints. A small backlog in the ingestion queue is being processed. We are actively monitoring the situation and will provide ongoing updates as progress continues.
Posted Jan 08, 2025 - 10:52 PST
Update
We deployed a partial fix to enhance the availability of the /import endpoint. Our engineering team is actively working to fully restore service for the /track and /engage endpoints.
Posted Jan 08, 2025 - 10:06 PST
Update
We are continuing to work on a fix for this issue.
Posted Jan 08, 2025 - 09:36 PST
Identified
We are currently experiencing a service disruption affecting our ingestion endpoints. The root cause of this issue has been resolved by our upstream provider but our systems are still in the process of recovering.

Our engineering team is currently actively working to restore full functionality as quickly as possible. We will keep on providing updates every 30 minutes or sooner if significant progress is made.

We understand the critical nature of the issue and sincerely apologize for the inconvenience it is causing. Thank you for your patience and understanding as we work to resolve this.
Posted Jan 08, 2025 - 08:53 PST
Update
We have identified an issue with an upstream provider and our Engineering team is currently reviewing it. We will share further updates soon.
Posted Jan 08, 2025 - 08:14 PST
Update
We are continuing to investigate this issue.
Posted Jan 08, 2025 - 07:38 PST
Update
We are continuing to investigate this issue.
Posted Jan 08, 2025 - 07:37 PST
Investigating
From approximately 3:30PM UTC a subset of servers that serve the ingestion API endpoint served 502 errors.

Updates to this incident will be made shortly.

We sincerely appreciate your patience. If you have any questions, please contact support@mixpanel.com.
Posted Jan 08, 2025 - 07:36 PST
This incident affected: Data Collection.