Difficulties creating transactions
Incident Report for Evidos trust services

On March 29th we encountered an incident where after server downtime at night some of our services did not reboot correctly .https://status.signhost.com/incidents/mhc18lwx58cz

After solving this issue, during the afternoon a day later on March 30th, we still saw some unexpected behaviour by some services. We found the underlying issue and rebooted our services. We reboot our redundant services one by one so this has no impact on ongoing transactions. After this reboot 17:00 CET we sudddenly saw errors arising . Transaction creation was still possible but in a small percentage of cases resulted in retryable error 500s. After 30 minutes we decided to block transaction creation to prevent further errors queueing up and to be able to fully diagnose the cause of the issue.

Around 18:15 we found the cause of the errors, a client broke down because of the reboot and in this specific scenario escaped our automatic monitoring and logging in place. Full functionality was immediately restored and we added more logging to identify this problem if it would further occur in the future. Futhermore we improved our reboot behaviour so clients keep behaving as expected if rebooting again after downtime.

Between 17:30 and 18:30 transaction creation was not possible. We have taken steps to prevent such downtime in the future.

Posted Apr 01, 2022 - 16:31 CEST

This incident has been resolved. We will followup with a post mortem and will update this space and notify our subscribers.
Posted Mar 30, 2022 - 18:50 CEST
A fix has been implemented and we are monitoring the results.
Posted Mar 30, 2022 - 18:35 CEST
We found a broken process, and are slowly getting our services operational again. Transaction creation works but may be intermittend while we reactivate more services.
Posted Mar 30, 2022 - 18:32 CEST
We are still investigating this issue. We are working hard to restore full functionality and this issue has our full attention.
Posted Mar 30, 2022 - 18:00 CEST
We are continuing to investigate this issue.
Posted Mar 30, 2022 - 17:14 CEST
Some users are experiencing an error 500 code whilst sending out a transaction. We are currently investigating this matter and will keep you updated.
Posted Mar 30, 2022 - 17:02 CEST
This incident affected: API, Portal, ID Proof, and Webforms.