23rd February 2019

Silverfin Post mortem 18th February

On 18th February Silverfin had two consecutive downtimes before the planned maintenance window later that evening. The downtime was 1 minute at 15:32 UTC, and 45 minutes starting from 18:45 UTC. These two downtimes were caused by an attempt to gain access to a key-value store server used for temporary information like background jobs and caching. The attack did not cause any data loss and there are no signs any data was compromised. For transparency, a short timeline of the events is listed below:

15:32 UTC: the attacker succeeds in changing a config setting in the key-value store. There's a short downtime caused by this change in config. The temporary data reset points to something going wrong with the key-value store, nothing suggests this is an attack yet. After the recovery, we're monitoring the key-value store while preparing for the planned migration in the evening.

18:45 UTC: the attacker makes another config change, trying to gain administrator access to the server. The config change tries to write to a location the key-value store user does not have access to. This causes downtime, and it's immediately clear we're facing an attack. We choose first to investigate the situation before bringing Silverfin back online. Investigation shows that we can harden the key-value store for this attack in multiple ways. We make a few configuration changes to the key-value store and application code to work with a new hardened configuration.

19:40 UTC: After changing these settings, we bring the key-value store back online, and Silverfin with it.

22:20 UTC: The planned migration starts. This includes moving the attacked key-value store server to a new stack

00:38 UTC: The planned migration is finished. The attacked key-value store is deactivated.

We spent the last two days further investigating logs to make sure nothing was missed. We're also consulting with an external independent security forensics team which will do their own separate investigation to confirm the proper conclusions and measures were taken. The external investigation will be concluded next week. Any additional measures that would enable Silverfin to further protect its stack from similar attacks will be implemented based on the findings of the investigation and a summary will be communicated through this page.

Although there are no signs any data was compromised, we take this event, and security in general very seriously. You can email security@silverfin.com if you have any question regarding this event or security in general.