This afternoon we released new functionality on our platform. This release coincidentally took place during a regular maintenance update on one web server. Normally the release process on that one web server is paused until it is finished updating, and in the meantime the traffic is sent to the other active web services. We have a continuous release policy so releases can be performed at any time without user impact. We release changes multiple times per week.
Now, however, the web server was not automatically taken out of transaction processing, and traffic was routed to it that could not be handled any further, as this web server was still updating. This resulted in signers receiving an error 504 when opening their signing link, and impacted some active signers who were routed to that individual web server between 12:26 and 13:00 Transaction creation, emailing and other services were not impacted. Signers routed to the other web servers where not impacted.
After the discovery at 12:26 we immediately started the investigation. At 12:50 we took the affected server out of roulation. From that moment on we saw the problem diminish. Around 13:00 we were fully operational again.
We immediately made a change in our web server and deploy process to prevent this scenario from happening again, and releases during server updates will proceed with no impact as they did before. In detail, we improved our web server health check during update and release processes.
All sign links can be completed, including those created before or during the incident.
The release consisted of an update to our postback component, which makes the delivery of messages from our service to customers more robust and faster. This enhancement is now live.