Postback messages delayed
Incident Report for Evidos trust services
Postmortem

Summary

On Friday the 30th of December, we saw our postback delivery become slower and slower. To mitigate this, we parked all pending postbacks so ongoing processes could proceed. This resulted in postbacks being sent out slower out from approximately 13:30 CET until 18:30, and some postbacks not at all after we parked the existing queue because processing times became too slow. Transaction status for these postbacks had to be updated by performing a GET call.

Incident

Due to a high amount of postbacks to send out our postback service environment became less responsive and slowed down considerably. This resulted in postbacks being sometimes too slow to send out and forming a queue. After notifying these postback delays we parked the existing queue. Notification was impacted by less than optimal availability of colleagues due to the holidays. This resulted in new postbacks being sent out in real time again. In the following days we closely monitored the postback performance and we saw problems reoccuring. We looked at re-introducing the parked postback queue, but as this would introduce considerable risk into our ongoing transactions and operations, we decided not to.

Mitigation

We introduced multiple postback efficiency fixes in the past few days, to improve postback handling and speed. We also took a look at database optimization to improve postback related queries. Furthermore, we improved our monitoring to alert us sooner if postback slowdowns occur. Lastly, we investigated re-introducing parked postbacks into our system so if this occurs in the future we are able to introduce these without risk.

Posted Jan 09, 2023 - 10:02 CET

Resolved
We released some postback related fixes and saw no new problems arise after friday the 30ths downtime. We kept a close monitoring on our service but are now comfortable to say the problems have been resolved and that apart from Friday the 30th during the incident all postbacks were sent succesfully. A post mortem will be posted on this page to further explain what we saw happening and our mitigation.
Be advised to perform a GET on any transactions with an unresolved status, as missing postbacks from Friday cannot be sent again. We are sorry for inconvenience caused.
Posted Jan 04, 2023 - 18:01 CET
Monitoring
As of now the postbacks are being sent out in real time. We are monitoring performance and will update this space when a definite fix is set live.
Posted Jan 02, 2023 - 11:02 CET
Update
The postback issues we saw on friday occur again, postbacks are delayed and we are focussing on bringing back this service with the highest priority. We will keep you updated via this page, and advise you to pull information via a GET to have the latest transaction status in your appliation.
Posted Jan 02, 2023 - 10:43 CET
Identified
You may experience a delay in receiving the postback messages. We're working on a solution to solve the problem.
Posted Jan 02, 2023 - 10:23 CET
Monitoring
A fix is found, there is no delay anymore in the postbacks. We are monitoring the results
Posted Dec 30, 2022 - 18:35 CET
Identified
You may experience a delay in receiving the postback messages. We're working on a solution to solve the problem.
Posted Dec 30, 2022 - 17:22 CET
This incident affected: API, Portal, ID Proof, Webforms, and Evidos App.