Write errors during unplanned database failover
- Started
- Duration
- 1 hr 36 min
- Last updated
- Resolved
Affected: Data Services, Merchant Gateway API, Collections, Payouts and Settlement, Merchant Portal
Resolved
The platform has been fully operational for over an hour and replication is healthy. Resolved. Total write-unavailability was approximately 21 minutes. Payments initiated during the window were rejected rather than accepted, so no transaction was recorded in an inconsistent state.
Monitoring
Failover completed and the new primary is accepting writes. Error rates have dropped sharply. We are monitoring replication lag on the remaining replicas.
Identified
The primary database node became unresponsive and an automatic failover is in progress. Writes are rejected while the new primary is promoted.
Investigating
We are seeing widespread errors across the platform. Payments, the API and the merchant portal are all affected. We are investigating with urgency.