SMS are being sent in a loop
Incident Report for Evidos trust services
Postmortem

Our webhosting service CloudVPS encountered downtime during the night of March 29th.
https://cloudvps-status.com/incidents/bbqppjfn4665

After our hosting party restored functionality, our systems automatically rebooted and operations continued as usual.
When higher load occured during the day however, we saw that our SMS service did not reboot fully as expected, and delivered up to five sms messages where one was expected as mentioned here. https://status.signhost.com/incidents/mhc18lwx58cz

Because of the high amount of sms messages our system had to queue, we saw some slowing behaviour in our API, where sometimes it took longer than expected to create a transaction because of the other traffic our queue had to work with. After diagnosing this issue and restoring the queue nodes we were fully operational again. During emergency reboot of the queue node to prevent further slowing of transaction creation, we saw a few errors where transactions could not be created, and this was resolved as soon as reboot was complete.

Posted Mar 29, 2022 - 12:17 CEST

Resolved
This incident has been resolved.
Posted Mar 29, 2022 - 10:59 CEST
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Mar 29, 2022 - 10:24 CEST
Investigating
We are investigating an issue where the sms for the SMS verification is being sent out multiple times.
Signing by SMS stilll works, and other methods are all operational, but signers might receive more sms messages than expected. We are looking to fix this as soon as possible.
Posted Mar 29, 2022 - 09:57 CEST
This incident affected: ⮑ External verifications (Phonenumber / SMS).