Evidos connectivity incident
Incident Report for Signhost Verified Signing
Resolved
Connectivity issue description 02/02/2022 16:00 > 03/02/2022 11:00

Evidos experienced a heavy DDOS on our DNS host via Transip on January 27.
https://status.signhost.com/incidents/ymsfnflpdrl4

To mitigate this, we switched to another DNS hosting environment via Azure in an emergency procedure, where we were already running several internal applications and could therefore move quickly. We were then running stably on Azure.

In consultation with Transip, we started looking at solutions for DNS and the DDOS. Returning to Transip, however, accelerated on February 2 after the discovery that certain DNS technologies we deem important for security were not possible on Azure.

We went back to Transip on February 2 at the end of the day to reactivate the more secure DNS technologies. Usually a DNS migration happens without notice. However, we immediately saw problems the stability of connections, and in consultation with Transip we aborted the migration and went back to Azure.

Unfortunately, during and directly after this migration, some DNS cached environments still thought of us being hosted on Transip. No message was available that the DNS should be automatically renewed and followed to our current host Azure.

This meant that systems with our old DNS entry in their cache, were still trying to find us there, instead of looking for us at the new host. This could mean portal users, some signers who sign more often in our systems, and API connecting servers. We were online for anyone with no existing cache, but people with existing cache might have experienced issues connecting between 2/2/2022 16:00 CET and 3/2/2022 11:00 CET.
During this time, we advised flushing of DNS cache as an instant solution. Waiting 10 minutes for an auto renewal worked as well for many people.

We are in consultation with Transip to resolve the problems with the malfunctioning connections, the reason we had to move away to Azure in the first place. This will reenable the DNS technologies.

At the same time, we are looking for more fallback options, something that we had already started after January 27, but has now been accelerated.

By noon of 3 februari all local DNS caching should be automatically renewed, if you have problems reaching our environments still, a manual cache flush will do the trick.
Posted Feb 02, 2022 - 16:00 CET