ReFS vs. newest Ransomware

[mkretzer]

For the last week i have been helping out in an evironment that got hit hard by a new kind of ransomware.
Instead of encrypting the files on the servers/clients it encrypted part (the first few GB?) of the disks directly - which caused the filesystems to appear as raw. Only windows was affected.

It also hit the Veeam server and its ReFS volumes. Because of block cloning we knew that it is very unlikely that we could salvage alot - even if only the first 50 GB have been overwritten some block-cloned block will most likely be in the overwritten area. Gladly, tape backups were in place but kind of out of date because it was written by a non-GFS job with non-synthetic tape fulls (it just took the synthetics that were on disk). Interestingly, this "classic" backup mode helped us later...
But how do you "fix" a ReFS with such a severe corruption?

Interestingly, Windows 2019 delivers a tool for that (it has been mentioned here before but i do not believe in this context): ReFSutil (with parameter -FS) can in fact scan a whole disk even if the filesystem is mostly destroyed/not even recognized. The best thing is that thanks to integrity streams you know exactly what can be rescued and what (partly) corrupted. And even the directory structure gets recovered!
Sadly, in this case most of the VBKs were badly corrupted. Some VIBs were fine though and in a few cases could be combined with the VBKs from the tape.

This whole thing shows again how important the 3-2-1-0 rule and following best practices is.
- Do backup copies to a different kind of system (linux immutable, cloud, even another non-domain server would have helped here).
- Tape can safe you for a very low price. I can sleep alot better knowing that we and all the companies we talk to use them as a last line of defence.
- The simple, safe-contained Veeam backup file format might have its downsides but it allows for creative restores (combining tape backups with VIBs even after loosing all VBMs/database).
- Do backup testing and get to know the features Veeam has avaiable for recovery! Don't depend on only one way to get your data back - in this case Veeam had some kind of bug after the config restore causing the proxy component to fail with every restore - so we did instant recoveries + VMotion until support had sorted it out.

BTW we did involve a professional data recovery company who did try to rescue files from the NTFS filesystems of the VMs - the result was even worse since the directory structure was nearly completely lost and it is not even clear if the files (SQL dumps and such) are corrupted or not. So even with the problems block cloning brings in such a situation ReFS has for the first time shown to be more "Resilient" than NTFS.

[soncscy]

This is a very interesting read, thank you very much mkretzer!

If I can add to this, the policy I introduce to all my clients is:

"assume your primary backups will be compromised"

The on-site local backups are the most useful because you get quick and convenient restores, but they're also the most liable in my opinion. It's the biggest and easiest target, and the trade-off is always between the convenience of restores from your primary backups and the security of them. You must must must have a secondary strategy and understand what the recoverability looks like.

Hardened repositories are great, and a properly secured one will undoubtedly be great, but for my mind, I sleep a lot easier knowing there are multiple immutable options (software with S3 or hardware with tape) to rely on, even if there's "some" delay.

I can only hope there's a future with Veeam soon where we can easily get things like transaction logs to an immutable medium soon. I know it __can__ be done to tape, but the restore narrative is a bit challenging for my taste.

[mkretzer]

Wow - i just got modified Agents from Veeam Support and with them i was able to restore alot of files even from the most badly corrupted vbk/vib.

Veeam support rescues the day

[kentsu]

Hi, Thank you mrkretzer for this interesting read. Really hard to find anything about corrupted or ReFS in RAW state outside of the scope of the january microsoft patch that impacted ReFS volumes.

Any other lessons learned on recovering ReFS or any new cases you have been involved with same situation?

[mkretzer]

No, "gladly" we had no other problems like this.
It seems like the options are quite limited with windows 2016 (without ReFSutil which saved us) from what i read