Targeted Datastore job no free space warning, crashing VM

[Barker]

I'm on Veeam 9.5 (9.5.0.823) and have a job where I'm only backing up a single data store on the VM (ISCSI 0:2) and most of the time it works, but 2 times it has run out of space and crashed the VM.

The most recent occurrence, it actually cause a different datastore (that is also attached to the VM, but not where the VM definition is stored) to run out of space. Once I increased the space for this excluded datastore, the job continued.

How does Veeam select the datastore to store the snapshots in, and why does it not warn me and cancel the job if it does not have enough space? I have all of the defaults set in regards to space notifications, and VMWare (5.5) has not been modified to change the default snapshot storage location.

Any suggestions as to why this may be occurring?

Thank you.

[PTide]

Hi and welcome to the community!

In ESXi 5.0 and later, virtual disk delta.vmdk files for snapshots are placed in the same directory as the parent virtual disk (.vmdk) file. That is, Veeam does not select the datastpre to stor the snapshot in. I'd recommend either to move VM to another datastore, or to edit workingDir parameter in VM's .vmx configuration file. For additional details please check this KB.

Regarding your question why Veeam does not warn you - VeeamONE provides reports on datastores capacity.

Thank you.

[blithespirit]

You should also check out what else you have in use in your datastores. You should never have over 80% used in a datastore so I would be looking at maybe a little housework.

Remember you memory balloon swap file is stored with your VM location regardless of if it's used or not, if you have high memory VMs then they will eat the same space in you data store as the allotted memory, for example a 64GB memory allocation on a SQL server will use 64GB of space for swap on disk (only when VM is powered on). Worth checking and try to get under that 80% marker.

[Barker]

Sorry about the delay in response.
Thanks for the info. I've had this occur 2 separate times, and each time it was a different datastore that ran out of space, and neither of them was the datastore that the .vmdk was stored in. That's why I'm confused. Any of my normal (full vm) backups work fine, it's only this backup where I choose a specific datastore that is failing. I'll reach out to VMware support. If you have any suggestions, or can point me to any additional log files that I can look at that may point to what the issue is that would be great. I'm just curious why different datastores are filling up during this backup.

This is a dfs fileserver with 7 different datastores assigned to it (all iscsi connections) and I'm only looking to back up a single datastore. None of the datastores are in any kind of usage warning levels. I noticed that if the datastore is over 80% full it will throw an alert in VCenter, so I always make sure it's about 30% larger than it needs to be to avoid that.

[Barker]

I just noticed, even though I have this job disabled and nothing from Veeam is using it currently, there is still a snapshot that is on that server labeled "VEEAM BACKUP TEMPORARY SNAPSHOT".

If Veeam isn't currently performing any backups, is it safe to delete this?

Another question...when you choose a single targeted datastore, does the snapshot for that VM still include ALL connected datastores, or just the targeted one? Would it make more sense for me to connect datastores I'm not planning on backing up with the VMware option of "Independent"? It appears that independent datastores are not included in snapshots. Because I'm only backing up 200GB, but the combined connected datastores equal 7TB, I'm wondering if maybe I'm not following best practice on this.

[foggy]

I just noticed, even though I have this job disabled and nothing from Veeam is using it currently, there is still a snapshot that is on that server labeled "VEEAM BACKUP TEMPORARY SNAPSHOT".

If Veeam isn't currently performing any backups, is it safe to delete this?

Yes, you can remove this snapshot via vSphere Client manually, if it is not removed by the job automatically.

Another question...when you choose a single targeted datastore, does the snapshot for that VM still include ALL connected datastores, or just the targeted one? Would it make more sense for me to connect datastores I'm not planning on backing up with the VMware option of "Independent"? It appears that independent datastores are not included in snapshots. Because I'm only backing up 200GB, but the combined connected datastores equal 7TB, I'm wondering if maybe I'm not following best practice on this.

Correct, the snapshot is triggered for the entire VM, unless you switch disks to independent.

[Barker]

Thank you Foggy.

I think that's what happened then...I noticed that all of the datastores had snapshots growing in them. I'm guessing that Veeam Snapshot that was not properly removed was growing in size. One of the "unexplained" datastores that had run out of space did so on a day when we propagated ACL changes to all of the files on it. So that would make sense that the snapshot for that particular datastore would blow up.

I'll be sure to keep a closer eye open for orphaned snapshots that veeam leaves behind.

[foggy]

You can ask support for assistance in investigating the reasons of these snapshot leftovers, should they continue to occur.