17/10/2025
It’s been a challenging week at STC, it all started off with a call on Monday morning just after 9am from a customer, “We’ve got a problem, we can’t access any of our files and nobody can work”
Their first thought was we’ve been hacked, the problem was the NAS box which is used for file storage had failed, all the lights on solid and not responding. Our priority was to get the data recovered from the backup onto a PC that would be a temporary fileshare. It takes time to recover over 750GB of data, but we got them back up and running as soon as we could.
So what went wrong? It’s easy to just buy a new NAS box, restore the data and carry on. But what would have been learnt? Could we have recovered the situation any quicker?
We found out that this brand of NAS suffered from a bug in the processor, Intel discovered this back in 2016 and made their customers aware of this issue, there is no permanent fix for the bug it’s just a matter of time before the processor fails. However a temporary fix was possible with a modification to the mainboard inside the NAS by soldering a 100ohm resistor across pins 1 and 8 of the LPC connector.
Although we had already recovered the customers data from backup, we wanted to find out if this was the reason the NAS had failed, otherwise there would have been no way of knowing what happened. After the modification the NAS box powered on fully and we could access all the data and were able to confirm to the customer it wasn’t down to any hack or ransomware.
Having a backup saved the day, I can’t stress enough, have multiple backup methods, Cloud / onsite using a couple of external drives and rotate them daily. More importantly TEST the backups don’t leave it to chance.