Snapshots and low disk space

[duhej]

Good afternoon. There is MS Server 2016 (2 pieces: VM source and VM receiver) and VBR U2 and configured in it backup and VM replication tasks from source Hyper-V to receiver Hyper-V. Periodically, randomly, on the HDD (on the VM source) with the VM, snapshots of the VM are created and then the free space on the disk with the VM ends and then the VMs do not work. This is totaly not necessary and creates a lot of problem problems for me. How to completely disable the creation of snapshots on the VM source, so that all backups and replicas are created ONLY to receiver Hyper-V?

[Mike Resseler]

Hi @duhej,

First: Welcome to the forums!

I am not sure if I completely follow. Our replication technology is based on snapshot creation so we need to create a snapshot or we can't replicate the changes. That being said. If you are running out of disk space, that means that either it takes a long time for the replication which means there is a lot of changes that need to be processed or you are really low on disk space.

How much space do you have left? Are the checkpoints (new MSFT name for snapshots) placed on the same disk or redirected to another location?

Thanks
Mike

[duhej]

First: thanks for answer.
Second: i have source hyper-v and reciever hyper-v. I totaly do not need any snapshots or another changes to original VM vhdx files placed at source hyper-v. For snapshots or another changes i have reciever hyper-v. I have 1000 gb and I am running out of disk space, because there are a lot of snapshots of VM (at source hyper-v) placed on the same disk and this is the real problem.

[Mike Resseler]

You do need a temporary snapshot at the source side when the replication starts. Otherwise it is not possible to do the replication.

When you look here in our manual on how it works: https://helpcenter.veeam.com/docs/backu ... tml?ver=95

in point 8 you see: Veeam Backup & Replication requests Microsoft Hyper-V VSS to create a cohesive point-in-time copy of a VM. Microsoft Hyper-V VSS creates a volume snapshot or VM checkpoint, depending on the hypervisor version.

But that snapshot should be removed after the replication job is done
Mike

[duhej]

Ok, so if there are a lot snapshots (chain of snapshots) at the same disk - something is wrong?

[Mike Resseler]

Hi,

Yeah, I think something is wrong at that point in time. This could be for many reasons, and one of it could be Veeam (although when it is a chain, I doubt it but still...).

Now the issue with a chain of snapshots is that it could be dangerous to "remove" or "apply" them. So at this point in time, I would advise you to create a support call and let our engineers see if the issue is related to us or not, and at the same time they most probably will advise you what the best next step is to do (if it is not us that creates the chain at least...)

Please post the support ID here and the follow-up after working with support

Thanks
Mike

[duhej]

thx

[Calby]

Hi,
I have notice that if the free space on the datastore disk goes under xxx then I can't backup with Veeam.
I have googled but I can't find out what the limit is?
How much free space do I need on the datastore?

Is it over 5% of the total harddrive space that needs to bee free?

[Mike Resseler]

Robin,

The reason of that free space is because we need to take a snapshot of the Hyper-V VM. At that moment, what happens is that the original VHD(X) gets "frozen" (read-only) and all writes are going to a differential file (AVHD(X)). This is the way the backup of VM's work in Hyper-V (and for that matter, more or less the same in VMware). If the datastore is low on space that file can't be placed there and also the merge (after the backup) won't be possible as it needs additional space also (working space, it gets free again after that)

You say 5%, but how much is that in GB? And what is your host OS? And lastly, what is the size off the VM in total?

[Calby]

Robin,

The reason of that free space is because we need to take a snapshot of the Hyper-V VM. At that moment, what happens is that the original VHD(X) gets "frozen" (read-only) and all writes are going to a differential file (AVHD(X)). This is the way the backup of VM's work in Hyper-V (and for that matter, more or less the same in VMware). If the datastore is low on space that file can't be placed there and also the merge (after the backup) won't be possible as it needs additional space also (working space, it gets free again after that)

You say 5%, but how much is that in GB? And what is your host OS? And lastly, what is the size off the VM in total?

Hi,
I'm running Hyper-V in Win2016.
The guest OS are running on a seperated SSD the system of the Guest OS. No issue there, but my storage disks that are attached to the VM are almost full and aldo they are not included in the backup (I have choosen to only backup system disk 0:0) it seem that they are making some issue.
I have excluded them from the backup in Veeam but they are still making some kind of issue.

First disk are 2,72GB with 94,6GB free (I can adjust this so I get more free space but then I need to know the limit, how much free space do I need)
The second disk 2,72GB with 125GB Free (Same here I can adjust it, but then I need to know how much free space I need)
Third disk is total 209GB (Don't know how much free space is have, but it's making some alerats)

[Mike Resseler]

Hmm. I would rather say you have space enough for those sizes. But, I might be wrong here but your VMs OS disk is only 2.72 GB? that seems low if it is a windows VM?

The disk free space I cannot calculate because it depends on how much churn there is (and the growth of the AVHDX because of that). If it is a very busy VM during the backup, it will be a big AVHDX file (could even be bigger than the original one if many changes occur).

How much is the total size of those datastores?

[Calby]

Hmm. I would rather say you have space enough for those sizes. But, I might be wrong here but your VMs OS disk is only 2.72 GB? that seems low if it is a windows VM?

The disk free space I cannot calculate because it depends on how much churn there is (and the growth of the AVHDX because of that). If it is a very busy VM during the backup, it will be a big AVHDX file (could even be bigger than the original one if many changes occur).

How much is the total size of those datastores?

Sorry I do mean TB not GB sorry for that, the total space is in TB and the space left is in GB.
Beside the 209GB disk that one is 209GB in total.

The AVHDX is big on this diskes, they are over 2TB in size.

Production drive E:\ is getting low on free space (125,3 GB left), and may run out of free disk space completely due to open snapshots.
Production drive K:\ is getting low on free space (96,5 GB left), and may run out of free disk space completely due to open snapshots.


AVHDX sizes:
K: 2,61 TB
E: 2,59 TB

[Mike Resseler]

That means you are already running on snapshots for some time I think. Can you check on Hyper-V manager to see if there are open snapshots? Potentially multiple?

[Calby]

That means you are already running on snapshots for some time I think. Can you check on Hyper-V manager to see if there are open snapshots? Potentially multiple?

No, no other snapshots.
And I have tried to run Veeam again.
Then I get: 2018-02-19 10:30:43 :: Processing Media Error: Skipping VM processing due to insufficient free disk space on drive E:\.

I guess that I get this error now because I have increased the "Skip VM processing when free disk space are below" to 5% again as I did try to turn it down to 1% before.
If I have it on 1% I do get this kind of warnings as I did write about above.

But is the free diskspace limit 5% on the datastore disks?

Edit:
My guess is that I did get the first warning because when Veeam are opening a snapshot it get's out of free space - this is based on that when I do increase the "Skip VM processing when free disk space are below" to 5% I do get a error because of low diskspace and if I lower or turn the "Skip VM processing when free disk space are below" of then I get a warning.
Because I don't have any other kind of snapshot opened.

So if I do increse the free diskspace this should be solved, but as I don't know the free space limit it's kind of hard to do =/
Can you point me to the right direction?

[Mike Resseler]

From what I read here, I think it is indeed on the disks that hold the VMs.

I would suggest at this point in time to open a support call so our engineers can help you with (at least I believe this will be the way to go if I read everything correctly) merging the AVHDX file (unless that AVHDX is from a cluster disk, which unfortunately also uses AVHDX as extension) and then work further from there. As long as we don't have the space, we can't take that backup. If you run out of it... Well, you want to avoid that

I think it is better that the engineers look at it with the logs and also maybe with a support session because if I would give you advice to do a merge here, without actually seeing what is going on, I might give you the wrong advice and you could loose production data which I don't want to happen

Please post the support case ID here and the outcome after investigation/ working with the engineers

Thanks
Mike

[Calby]

From what I read here, I think it is indeed on the disks that hold the VMs.

I would suggest at this point in time to open a support call so our engineers can help you with (at least I believe this will be the way to go if I read everything correctly) merging the AVHDX file (unless that AVHDX is from a cluster disk, which unfortunately also uses AVHDX as extension) and then work further from there. As long as we don't have the space, we can't take that backup. If you run out of it... Well, you want to avoid that

I think it is better that the engineers look at it with the logs and also maybe with a support session because if I would give you advice to do a merge here, without actually seeing what is going on, I might give you the wrong advice and you could loose production data which I don't want to happen

Please post the support case ID here and the outcome after investigation/ working with the engineers

Thanks
Mike

I did fix the issue by my self, it was like I did think.

The total space for one of my disk was 2785GB (2,72TB) so I did take -6% (167,1gb) of that and that is 2617,9GB so I did round it down so what I did was that I did format the VHDX to only use max 2600GB and now everything works like a charm
Now I know that I never can get that kind of issue again with this disk I guess (hope).

[Mike Resseler]

That is actually a great tip! I'm sure others will benefit from this also when having the same problem.

Thanks for letting us know, and I am glad that you have solved it!

Cheers
Mike

[ssdn108]

Hi, i am running into same issue. When you say in the last response you formatted the vhdx, what actually you mean by that? You cannot format the original drive that is itself one of the drives for the VM that you are backing up that has data. So, i am confused with this format. Pls explain. Thx