Issues creating transactions.
Incident Report for Signhost Verified Signing
Postmortem

What happened?

On Tuesday October 15th starting around 14:00 CEST users were unable to create transactions caused by a configuration change by our hosting party.

What did we do?

After having noticed the issue occurring, we directly tried to analyze what the root cause may be. Once we had determined that the issue must be with our hosting provider, we contacted them immediately.

Simultaneously, we put our platform in maintenance mode to prevent overloading while continuing investigating the issue.

Our hosting provider notified us that the error was caused by a release earlier that morning on their side. We escalated the issue to the hosting provider to ensure it received high priority.

While the hosting provider worked on a fix, our internal team also started working on a solution to solve the issue on our end. This parallel approach allowed us to compare progress and determine which solution could restore the platform faster.

Unfortunately, after this issue was resolved a second issue was identified. For this issue we used the same approach as before where both teams worked on a solution to resolve it as soon as possible.

Around 20:00 CEST our platform was back online and fully operational.

What was the cause of the downtime?

Our hosting party released a new version of the authentication file, which was not communicated beforehand. With this change, some of the values in our file were no longer accepted.
This change was the cause of the authorization errors on our end and not being able to connect to our file storage.
After this issue was fixed, we still received an authorization error on our requests. This was caused by another change in the release, where the HTTPS header was found to be case-sensitive.

Next Steps

In order to guarantee our uptime and provide a high-availability service, we are dependent on available and proactive hosting. In the upcoming period we will evaluate and make agreements with our hosting party to prevent these unannounced downtimes moving forward.

On the longer term, we aim to add a backup environment under the umbrella of Entrust EU and US hosting, so we can leverage the high availability and scale of our parent company to provide our services moving forward when our primary hosting is unavailable for any reason, whilst making sure that customer data always resides in the right region.

Both these measures will guarantee our high availability on short and long term, and our teams will focus on scaling and building out our hosting moving forward. We are committed to provide our signing service 24/7, and the availability of our platform has our highest priority.

Posted Oct 17, 2024 - 11:09 CEST

Resolved
The incident has been resolved. We are working on getting all information and evaluating, a post mortem will be posted later.
Posted Oct 15, 2024 - 20:17 CEST
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Oct 15, 2024 - 20:10 CEST
Update
Unfortunately, we have not yet been able to resolve the issue. The issue is caused by a change in configuration by our hosting party, which was not communicated beforehand. We are now working on a fix on our side and will continue that this evening, to have our services back online as soon as possible.
Posted Oct 15, 2024 - 18:50 CEST
Update
We are actively working on a fix. This issue is being treated with the highest priority.
Posted Oct 15, 2024 - 16:59 CEST
Identified
The issue is caused by a change in configuration by our hosting party. We are in close contact with our provider and working together to fix this issue as soon as possible.
Posted Oct 15, 2024 - 16:09 CEST
Update
Unfortunately, we have not yet been able to identify the problem. We have put the platform in maintenance while we are investigating the issue. Our services are unavailable for the time being.
Posted Oct 15, 2024 - 15:02 CEST
Update
We are continuing to investigate this issue.
Posted Oct 15, 2024 - 14:22 CEST
Investigating
We are currently experiencing problems with creating new transactions. New transaction creation results in 'In progress' transactions. We are currently investigating the issue.
Posted Oct 15, 2024 - 14:20 CEST
This incident affected: API, UI / View, and Portal.