B&R is not smart enough?

[MAA]

I recently migrated a big fileserver VM to another host (change both compute resource and storage, and change virtual disk format from thin to thick).
After that B&R started making a full backup (after 3 days only 27% done) instead of continuing with an fast incremental one.
Why wasn't B&R smart enough to keep doing fast incremental backups?

[Andreas Neufert]

My guess is that you have done this while the server was in shut down mode. That way a Storage vMotion will destroy VMware Change Block Tracking information.
Other option could be that VMware has reported to us a disk size change, which leads as well to Change Block Tracking reset as CBT do not work reliable with changed disk sizes (VMware issue).

[foggy]

Another reason could be the VM ID change that might occur during the migration depending on how you've performed it.

[MAA]

Do you plan to improve this "dumb" behavior?
B&R should send over network only changed information, in any possible situation.

[Egor Yakovlev]

Behavior is quite logical.
1. If VM MoRefID changes, it is a new VM from Vmware standpoint and as a result new VM for Veeam to backup. New VM means start with Full backup and start tracking changed blocks for fast incremental backups afterwards.
2. If VM MoRefID is the same, we ask Vmware to give us information about changed blocks since last backup. If VMware gives us full disk to read, we read full disk. It is not our logic to determine what was changed, only ESXi host itself can track changes as they are made.
/Thanks!

[MAA]

I am looking at the situation from the point of view of the end user - unchanged information should not be transmitted over the network.
If the Vmvare logic is too "dumb", I expect "smart" B&R logic to be applied.

[Gostev]

It is unclear what is your scenario.

For scenario 1, it's impossible to avoid transferring all blocks since from our perspective there's the new VM which needs a full backup created. However, you should never end up in this scenario if you have a vCenter Server registered with Veeam (and not just a bunch of standalone ESXi hosts).

For scenario 2, unchanged information will never be transmitted over the network from a backup proxy to a backup repository. Because even if "dumb" VMware logic will return us all blocks as changed in some cases, Veeam will only transfer actually changed blocks over the network. This is achieved through comparing a hash of each block that VMware returned to us as "changed" with what is already stored in the backup repository. So, the "smart" B&R logic you want is already there (since v1 actually).

[MAA]

foggy

you didn't mention the bottleneck stats for this job - even an active full shouldn't normally take that long, so maybe we can give some advices regarding your setup optimization.

Вottleneck is network - 3.5ТВ VM over 40Mbps WAN

Gostev

unchanged information will never be transmitted over the network from a backup proxy to a backup repository. Because even if "dumb" VMware logic will return us all blocks as changed in some cases, Veeam will only transfer actually changed blocks over the network. This is achieved through comparing a hash of each block that VMware returned to us as "changed" with what is already stored in the backup repository. So, the "smart" B&R logic you want is already there

Then how can I investigate why all the data is being sent over the network?
"Statistics" and "Report" does not have the necessary information.

[Andreas Neufert]

Do you perform direct backup over WAN links?

Or is it a backup copy job?

[MAA]

This is a "Backup Job"

[foggy]

How do you tell it is sending the entire VM data? Could you also describe your setup in a bit more details - what kind of repository are you using, how is it added to Veeam B&R? What are the Read and Transferred counter values reported in the job stats?

[MAA]

Default Local Repository (Windows).

Cannot attach screenshot with job stats. Forum do not support it?

Processed 502.1GB (14%)
Read 476.2GB
Transferred 446.7GB (1.1x)

[wishr]

You may use any external image hosting resource you like to embed images right into your forum posts. Imgur, OneDrive, Sharepoint, anything you are comfortable with.

However, since there are still lots of unclear details about your environment and the whole situation in general, instead of fortune-telling, I would suggest raising a support case, let our engineers check your logs, determine what caused the full backup (if it's really a full backup ) and slow data transfer speeds, and suggest the best ways to address these issues. Plus we will also be able to review the technical details collected by the engineers and come up with suggestions when needed.

[foggy]

Looks like it is the first scenario with the changed VM ID due to improper VM migration with VM re-registration. As mentioned above, in this case Veeam B&R treats this VM as a completely new one and there's no way to tell it it is actually the same VM (what if it is not?).

Anyway, with 3.5ТВ VM and a 40Mbps link you should avoid active fulls by all means, this is a killer. Consider switching to local backups with remote backup copy jobs using WAN acceleration.

[ksimon]

I am not sure why this is a Veeam issue.

You made major changes to your infrastructure. Including data and hardware and software. Changed block information was lost during the process. Either that or Vmware or Veeam is now seeing the disks as new disks.

The expected result would be that, for the next backup, CBT would be unavailable, and the entire disk would need to be backed up.

If you were doing a local backup + copy, this likely would have been fine. Since you are backing up over a slow link, yes, it will take a long time.
Perhaps you could make a local backup and use it to seed a remote backup?

It sucks when your infrastructure is not up to your needs. I know this from experience.

[MAA]

Yesterday, with a progress of ~ 90%, there was a network failure at the ISP, and the backup job ended with an error.
After I manually started the backup job, the backup of this VM started not from 90%, but from scratch!
I can not believe that B&R has simply lost ten days of backup process, and I have to start all over again.

[Egor Yakovlev]

Hi Maa,

Backup Job relies on the VMware snapshot - that means we will trigger VM snapshot, read and transfer VM data to the target storage and delete snapshot afterwards. If job is interrupted and triggered again, we will yet again initiate a VMware snapshot, that will contain a stable VM state at that moment. There is no way for backup job to turn back time and ask VMware to snapshot your VM in the state from 10 days ago if job fails - that is to process from where we was interrupted.
Having a backup job running over internet for 10 days does not seem to be very efficient approach - consider a fast site-local backup, followed by Backup Copy Job afterwards - out of all benefits for that approach I can note your production storage will love you for it - as VM snapshots will not be held for the entire offsite-copy cycle, as well as Backup Copy does support network interruptions, and in case your VM has multiple disks, we will start from the disk we were interrupted on previously.

/Thanks!

[Andreas Neufert]

As we shared your concept do not follow the best practices at all and you will run into issues as described.
Backup always locally for fast backup and restore. Then use our Backup Copy Jobs to copy data to other places.
In case of small or unreliable WAN links you can use our WAN accelerators that enable resume on disconnect together with optimized data transport for small WAN links.

The reason for the retry is that the VMware API do not allow the continuation in many cases. As after a failed backup we have to remove the snapshot (with the preserved state) from the source VM so that the datastore will not run full over time. Usually this is not at all an issue with the above mentioned concept.