On Tuesday October 15th starting around 14:00 CEST users were unable to create transactions caused by a configuration change by our hosting party.
After having noticed the issue occurring, we directly tried to analyze what the root cause may be. Once we had determined that the issue must be with our hosting provider, we contacted them immediately.
Simultaneously, we put our platform in maintenance mode to prevent overloading while continuing investigating the issue.
Our hosting provider notified us that the error was caused by a release earlier that morning on their side. We escalated the issue to the hosting provider to ensure it received high priority.
While the hosting provider worked on a fix, our internal team also started working on a solution to solve the issue on our end. This parallel approach allowed us to compare progress and determine which solution could restore the platform faster.
Unfortunately, after this issue was resolved a second issue was identified. For this issue we used the same approach as before where both teams worked on a solution to resolve it as soon as possible.
Around 20:00 CEST our platform was back online and fully operational.
Our hosting party released a new version of the authentication file, which was not communicated beforehand. With this change, some of the values in our file were no longer accepted.
This change was the cause of the authorization errors on our end and not being able to connect to our file storage.
After this issue was fixed, we still received an authorization error on our requests. This was caused by another change in the release, where the HTTPS header was found to be case-sensitive.
In order to guarantee our uptime and provide a high-availability service, we are dependent on available and proactive hosting. In the upcoming period we will evaluate and make agreements with our hosting party to prevent these unannounced downtimes moving forward.
On the longer term, we aim to add a backup environment under the umbrella of Entrust EU and US hosting, so we can leverage the high availability and scale of our parent company to provide our services moving forward when our primary hosting is unavailable for any reason, whilst making sure that customer data always resides in the right region.
Both these measures will guarantee our high availability on short and long term, and our teams will focus on scaling and building out our hosting moving forward. We are committed to provide our signing service 24/7, and the availability of our platform has our highest priority.