CSV volume out of diskspace during backup

[remko.de.koning]

Last weekend a periodic active full started (1 per month) of a large fileserver. (1.2 TB)
The .VHDX file was the only file on the CSV and the CSV had 230 GB of free space when the backup began.
In the middle of the night, Veeam One started sending emails that the free diskspace of the CSV was decreasing rapidly and at approx 4.00 AM the CSV ran out of diskspace.
Not much later I received a Veeam One email that the background disk merge failed.

It made me wonder how this diskspace could decrease so rapidly. This could only be if there are many disk alterations on the disk during the backup window.
The disk of the FileServer has Microsoft 2012 Dedup enabled on the volume so perhaps a nightly maintenance tasks caused the differencing disk to grow rapidly.

Question:
Altough the disk is back online without any user intervation I wonder if I lost every change that happened between the start of the backup and the time the CSV ran out of diskspace.
Is my assumption correct?

[Mike Resseler]

Remko,

What I assume happened. Backup started, VHDX got locked and AVHDX was created to temporarily store the data (so a snapshot). While the backup was being performed on the locked VHDX, all new data is being written in that AVHDX. Why it was decreasing so rapidly is a good question, but it does mean that during that backup window the fileserver was making (a lot of) changes. And since you are talking about 230 GB free, it must be a lot. So I would certainly look into the fileserver what was causing that.


Did you lose data? When the disk cannot write anymore (full), the VM should have gone into pause instantly. But then the merge still needs to happen. So you need to figure out if the disk is merged. Veeam ONE told you that it failed. Have a look at all logs of Hyper-V and see if it didn't succeed in the retry? Since the disk is back online without user intervention, then changes are that it succeeded after all, but certainly worth to investigate. One thing should normally never happen. The VM (or Hyper-V) won't automatically throw away the AVHDX to start the VM again. So either a manual operation has been done and the disk has been thrown away (let's hope not) or you should be good.

My 2 cents
Mike

[remko.de.koning]

Thanks Mike for your detailed answer.
I had a closer look at the Hyper-V-VMMS Admin logs today and indeed.....
The machine was put on pause at 04:45 AM and failed twice to merge the disk. The third attempt was successfull at 5:30 AM.
So if I understand all correctly, the fileserver was down for approx 45 min.

Thanks for your 2 cents

Much appreciated!

Remko

[Mike Resseler]

No worries! And that also means that you have no data loss, but you did have downtime...