Issues with signing transactions and downloading documents.

What happened?

On Thursday, February 12th, around 10:30 CEST, we observed an increase in errors affecting the file download and file preview functionalities.
As a result, customers experienced issues downloading documents or receipts and viewing documents during signing. In many cases, the action succeeded after retrying.

After resolving the incident, we retried open transactions on our end resulting in under 1% of transactions ending in an error state. In some cases, this may have been caused by transactions that were deleted in the meantime.

What we did

Once the increase in errors was detected, our team immediately started investigating the issue.
Initial analysis focused on our application and infrastructure, but after ruling out internal causes, we escalated the issue to our hosting provider.

What caused the issue?

The incident was caused by a misconfiguration at our hosting provider during a migration.

Because of this incorrect setting:

• Larger data packets within the data center were dropped.
• This led to failures on our side when retrieving or previewing files.

What are we doing next?

To resolve the incident, the hosting provider corrected the configuration on the affected switches, fully mitigating the issue.

To reduce the likelihood of recurrence and improve detection going forward, the following improvements are being implemented:

• Improved monitoring: Thresholds in the monitoring system of our hosting provider that tracks specific error rates will be adjusted so similar issues can be detected earlier.
• Faster escalation: We are reviewing escalation procedures with our hosting provider to shorten investigation time for infrastructure related incidents.
• Hosting migration: We are migrating our file services to another hosting provider this quarter to have more control over configuration.