We experienced a major issue in our platform as a result of our main database failing unpredictably. Our failover mechanism was activated but unfortunately for reasons we are still studying also failed. We will provide further information.
The first alarm was received at 2am and we got a few stores communicating to us at 3am. Our team started to work on the issue, however, due to the main database failing and the failover not working as expected it affected our ability to recover the system having an issue in the add to cart function. This had a partial issue (not affecting all stores) for 1 hour and additional 6 hours of outage to recover the system.
We are further investigating on the issue to ensure this is not happening again due to the same problem and put all mechanisms in place to avoid this issues.