What happened?

On Monday March 4 at 9:45 CEST network issues occurred at our cloud provider as a result transaction creation and signing was not possible.

At 13:27 our cloud provider migrated our database instances which initially resulted in yet another short period of outage. Within a few minutes this migration resolved our issues, creating transactions and signing was possible again.

What did we do?

As soon as we noticed the issue, we started putting parts of our platform in maintenance. We quickly discovered though that our whole platform was suffering from this issue, so we decided to put the whole platform in maintenance mode to prevent the message queues filling up and to have the database catching up again. The moment the database caught up we enabled our services one by one, starting with the API and moving on from there. Around 13:55 we were fully operational again.

We also re-acknowledged with our cloud provider that they should not do any migrations without our explicit permission.

What will we do better?

• Improve our communication with our cloud provider. We need to verify our agreements with them periodically to ensure these agreements are being followed up correctly.
• We’re constantly improving our database setup, i.e. hunting down expensive queries, setting up indexes, cleaning up unused data, etc. We will continue doing so.
• We were already in the process of improving the design of the database setup as a whole and this issue gave us a better insight in what parts need to be improved.
• Further improve monitoring of our current database setup.