Elevated errors in Silverfin

Incident Report for Silverfin

Postmortem

On 2019-09-03 we had scheduled database maintenance that included a change in collation (which specifies how data is sorted and compared). Unfortunately, this resulted in various failures such as syncs or logins not working. This caused us to have to rebuild various indexes on our (very big) database, which took quite some time.

We take these types of maintenances very seriously and always thoroughly test things, but this one slipped through the cracks. We will revise and review our upgrade procedure to try to avoid this sort of thing from happening again.

We apologise for the inconvenience. We’d also like to stress that customer data was safe at all times. If you have any questions, do not hesitate to contact us.

Posted Sep 04, 2019 - 18:32 CEST

Resolved

This incident has been resolved.

Posted Sep 04, 2019 - 18:31 CEST

Monitoring

We mitigated most of the impact. We still might have some degraded performance on particular syncs, but we've fully identified the issues and are putting the necessary fixes in place. We will update with a full disclosure when it's fully resolved. We apologize for the inconvenience and thank you for bearing with us.

Posted Sep 04, 2019 - 15:59 CEST

Identified

We are currently facing some instability issues within the Silverfin platform. These can result in behavior such as not being able to log in, or syncs not working. Our team is aware of this and is working on resolving this with utmost priority. We apologize for the inconvenience this is causing.

Posted Sep 04, 2019 - 10:33 CEST

This incident affected: Silverfin (Silverfin, Syncing, Exports).