Veeam B&R - Vmware backup job creates unwanted snapshot

[GennX]

Hello,

I have an issue with a Vmware backup job, the job starts without an issue, the backup is being copied to a network backup repository and after about 70-80% progress a "VEEAM BACKUP TEMPORARY SNAPSHOT" is being created and makes the Virtual Machine become unresponsible because the storage reamains out of space. Using the Vmware vSphere client I delete the "VEEAM BACKUP TEMPORARY SNAPSHOT" from the snapshot manager to release the space and power on the Virtual Machine. My questions is, why does Veeam B&R creates a temporary snapshot? I just want to backup the VM to a network repository, I don't want any snapshot to be created.

Veeam details:

Removing VM snapshot... (0% done)
Removing VM snapshot... (0% done) Details: The operation cannot be allowed at the current time because the virtual machine has a question pending: 'msg.hbacommon.outofspace:There is no more space for virtual disk /vmfs/volumes/5860e5b7-f6a49be4-5e7b-000af79c0cd8/NGI/NGI-000002.vmdk. You might be able to continue this session by freeing disk space on the relevant volume, and clicking Retry. Click Cancel to terminate this session. '.

[veremin]

To back Virtual Machine up a backup server always creates a snapshot (this's how the process works):

Veeam Backup & Replication does not install agent software inside the VM guest OS to retrieve VM data. To back up VMs, it leverages VMware vSphere snapshot capabilities. When you back up a VM, Veeam Backup & Replication requests VMware vSphere to create a VM snapshot. The VM snapshot can be thought of as a point-in-time copy of a VM that includes virtual disks, system state, configuration and so on. Veeam Backup & Replication uses this point-in-time copy as a source of data for backup.

If you create a snapshot of a VM outside of a backup server (using the vSphere client, for instance), will it also become unresponsive? Is there enough free space available?

Thanks!

[GennX]

If you create a snapshot of a VM outside of a backup server (using the vSphere client, for instance), will it also become unresponsive? Is there enough free space available?

Thanks!

I cannot create a snapshot using vSphere client as I know there is not enouth storage, the whole storage was provisioned for the VM (only 6GB of free space being left).
1. Why does the backup job starts and transfers 1TB of data (about 70%) and then it wants to make a snapshot ?
2. One thing I don't understand, I have a backup job plan for exactly the same VM created aprox. 2 years ago (recurent) being saved to another repository, the job finishes without an issue. I just wanted to clone the backup job to be able to save the backup to another repository and this is what happenes, transfers a lot of data to the new repository, then the VM storage remains wihout space because of the veeam temp snapshot beein created and then the VM goes offline.
Something is not right...

[veremin]

So the lack of free space seems to be the main issue here.

1) Doesn't seem right as a snapshot should be created at the beginning - was it indeed created in the middle of the job or did it just grow in size, take all free space available, and make the job fail?
2) Hard to tell the differences between the two without seeing debug logs and virtual and backup infrastructures.

I recommend reaching our support team for further investigation.

Thanks!

[GennX]

The lack of free space shouldn't be a problem, as first backup job on a different repository goes without a problem and the second one doesn't.

I've already contacted the support, but didn't get a reply yet so I could come back with more details as I wrote above.

Here is the task log (uploaded), where I can see that the snapshot is being created at the begining but without any error and the backup process goes on, strange...

Log: https://file.io/vDT2iEq2veIC

[veremin]

The lack of free space shouldn't be a problem, as first backup job on a different repository goes without a problem and the second one doesn't.

The lack of free space is a problem. The fact that one of the job finishes successfully does not necessarily help here - it might run faster to a different repository, so a snapshot is kept for a shorter period of time, thus it occupies less space and does not fill the datastore completely.

I've already contacted the support, but didn't get a reply yet so I could come back with more details as I wrote above.

What is your support ticket number, so we can follow the investigation?

Here is the task log (uploaded), where I can see that the snapshot is being created at the begining but without any error and the backup process goes on, strange...

This is expected - a snapshot is created at the job session start.

Thanks!

[GennX]

I understood one thing, I need to fix the issue with the free space, thank you.

[GennX]

BTW, till I solve the issue with the storage, what backup type can I use so it wouln't make a snapshot during the job? There is Replication Job, VM Copy, File Copy? What will do in my case? Thank you.

[Origin 2000]

Veeam will always needs to create a Snapshot when try to create a backup on Image level. Same is true with a Replicationjob. You can install a Agent wihin the GuestOS and backup on OS level.

BUT... you can tell the ESXi to create the Snap into another location (DataStore) so you will avoid that the Datastore runs out of free Storage during the backup.
https://kb.vmware.com/s/article/1002929

Regards,
Joerg

[veremin]

Correct, either the usage of an in-guest agent or the addition of free space will be required in your case. Thanks!

[JannieH]

I believe this has already been sufficiently answered, but I'll comment on this part of the question:

My questions is, why does Veeam B&R creates a temporary snapshot? I just want to backup the VM to a network repository, I don't want any snapshot to be created.

The short answer is: for data consistency (aka "to prevent data corruption.")

Longer version: When backup/replication software read VMDKs, they do so from outside the guest and limited visibility of what the guest is doing. Example timeline:
* 18:00: you kick off your backup or replication
* 18:10: your job reaches block 1,000 (arbitrary number)
* 18:11: the application running inside the VM modifies block 999 and block 2,000 using data that is interdependent
(say, a database file and a transaction log entry, or a filesystem table and an associated file.)
* 18:20: your backup/replication job reaches block 2,000, which is now inconsistent with block 999, making the resulting backup potentially unusable

To avoid the situation above, all image-based backups must take a snapshot of a VM to get a consistent view of the VM throughout the backup process. In practice, the underlying VMDK then effectively becomes read-only and can be backed up more or less like a regular file. All writes get redirected to the snapshot ("delta") disk.

There are all sorts of clever tricks with change block tracking etc. that can help with this in certain circumstances like Storage VMotion, but for the most part the above applies.

Although you could manipulate where snapshots are written to, IMO the best course of action would be to fix the underlying space issue. A general rule of thumb - unrelated to Veeam - is to always have "enough" (10-15%) free space on a VMFS volume. In the past this added some additional cost, but in this day of thin provisioning and fairly effective space reclamation, it is not usually a problem provided that not all your volumes "spike" at the same time.

Rather than free space, the biggest challenge with larger snapshots tend to be the impact of committing them back to the base disk, due to the fact that VMware snapshots are not copy-on-write. (If you will, the performance impact is after-the-fact, rather than during the snapshot's lifespan.)

From a general interest point of view: the amount of time the snapshot is open on the VMware side can be reduced in several ways, of which the most efficient tends to be to use storage arrays that can offload the snapshot for you at hardware level. In that case the snapshot (at the VMware end) is open only for a few seconds while the underlying hardware creates its own snapshot (which remains open for the remainder of the backup window.) Other ways of reducing the time are basically just to make sure the backup completes faster - make the VM smaller, use faster storage/network, etc.