9th August 2017

Silverfin - Syncing Syncing post mortem

What happened

Between 16:50 - 17:35 Augustus 9, we had very slow synchronisation for accountancy data. Loading new data was slow at best, or not working in some cases.

A faulty synchronisation with a web accountancy tool generated lots of import files. Silverfin can normally handle the amount of files and load this generates, so no monitors were triggered at that moment. Due to a bug in that specific synchronisation, the import files were not queued serial, but parallel, something that should never happen. This caused this specific company to sync slowly, occupying all other sync processes and blocking the other companies to sync.

What we're doing about it

We will improve the code to queue imports with a locking mechanism that ensures only one sync job can be run for one specific company. So the sync of one company cannot accumulate sync processes. We're investigating if we can put better metrics in place to monitor the synchronisation processes more closely.