-
- Expert
- Posts: 101
- Liked: 3 times
- Joined: Apr 27, 2013 12:10 pm
- Contact:
B&R is not smart enough?
I recently migrated a big fileserver VM to another host (change both compute resource and storage, and change virtual disk format from thin to thick).
After that B&R started making a full backup (after 3 days only 27% done) instead of continuing with an fast incremental one.
Why wasn't B&R smart enough to keep doing fast incremental backups?
After that B&R started making a full backup (after 3 days only 27% done) instead of continuing with an fast incremental one.
Why wasn't B&R smart enough to keep doing fast incremental backups?
-
- VP, Product Management
- Posts: 7081
- Liked: 1511 times
- Joined: May 04, 2011 8:36 am
- Full Name: Andreas Neufert
- Location: Germany
- Contact:
Re: B&R is not smart enough?
My guess is that you have done this while the server was in shut down mode. That way a Storage vMotion will destroy VMware Change Block Tracking information.
Other option could be that VMware has reported to us a disk size change, which leads as well to Change Block Tracking reset as CBT do not work reliable with changed disk sizes (VMware issue).
Other option could be that VMware has reported to us a disk size change, which leads as well to Change Block Tracking reset as CBT do not work reliable with changed disk sizes (VMware issue).
-
- Veeam Software
- Posts: 21139
- Liked: 2141 times
- Joined: Jul 11, 2011 10:22 am
- Full Name: Alexander Fogelson
- Contact:
Re: B&R is not smart enough?
Another reason could be the VM ID change that might occur during the migration depending on how you've performed it.
-
- Expert
- Posts: 101
- Liked: 3 times
- Joined: Apr 27, 2013 12:10 pm
- Contact:
Re: B&R is not smart enough?
Do you plan to improve this "dumb" behavior?
B&R should send over network only changed information, in any possible situation.
B&R should send over network only changed information, in any possible situation.
-
- Product Manager
- Posts: 2581
- Liked: 708 times
- Joined: Jun 14, 2013 9:30 am
- Full Name: Egor Yakovlev
- Location: Prague, Czech Republic
- Contact:
Re: B&R is not smart enough?
Behavior is quite logical.
1. If VM MoRefID changes, it is a new VM from Vmware standpoint and as a result new VM for Veeam to backup. New VM means start with Full backup and start tracking changed blocks for fast incremental backups afterwards.
2. If VM MoRefID is the same, we ask Vmware to give us information about changed blocks since last backup. If VMware gives us full disk to read, we read full disk. It is not our logic to determine what was changed, only ESXi host itself can track changes as they are made.
/Thanks!
1. If VM MoRefID changes, it is a new VM from Vmware standpoint and as a result new VM for Veeam to backup. New VM means start with Full backup and start tracking changed blocks for fast incremental backups afterwards.
2. If VM MoRefID is the same, we ask Vmware to give us information about changed blocks since last backup. If VMware gives us full disk to read, we read full disk. It is not our logic to determine what was changed, only ESXi host itself can track changes as they are made.
/Thanks!
-
- Expert
- Posts: 101
- Liked: 3 times
- Joined: Apr 27, 2013 12:10 pm
- Contact:
Re: B&R is not smart enough?
I am looking at the situation from the point of view of the end user - unchanged information should not be transmitted over the network.
If the Vmvare logic is too "dumb", I expect "smart" B&R logic to be applied.
If the Vmvare logic is too "dumb", I expect "smart" B&R logic to be applied.
-
- Chief Product Officer
- Posts: 31814
- Liked: 7302 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: B&R is not smart enough?
It is unclear what is your scenario.
For scenario 1, it's impossible to avoid transferring all blocks since from our perspective there's the new VM which needs a full backup created. However, you should never end up in this scenario if you have a vCenter Server registered with Veeam (and not just a bunch of standalone ESXi hosts).
For scenario 2, unchanged information will never be transmitted over the network from a backup proxy to a backup repository. Because even if "dumb" VMware logic will return us all blocks as changed in some cases, Veeam will only transfer actually changed blocks over the network. This is achieved through comparing a hash of each block that VMware returned to us as "changed" with what is already stored in the backup repository. So, the "smart" B&R logic you want is already there (since v1 actually).
For scenario 1, it's impossible to avoid transferring all blocks since from our perspective there's the new VM which needs a full backup created. However, you should never end up in this scenario if you have a vCenter Server registered with Veeam (and not just a bunch of standalone ESXi hosts).
For scenario 2, unchanged information will never be transmitted over the network from a backup proxy to a backup repository. Because even if "dumb" VMware logic will return us all blocks as changed in some cases, Veeam will only transfer actually changed blocks over the network. This is achieved through comparing a hash of each block that VMware returned to us as "changed" with what is already stored in the backup repository. So, the "smart" B&R logic you want is already there (since v1 actually).
-
- Expert
- Posts: 101
- Liked: 3 times
- Joined: Apr 27, 2013 12:10 pm
- Contact:
Re: B&R is not smart enough?
foggy
Gostev
"Statistics" and "Report" does not have the necessary information.
Вottleneck is network - 3.5ТВ VM over 40Mbps WANyou didn't mention the bottleneck stats for this job - even an active full shouldn't normally take that long, so maybe we can give some advices regarding your setup optimization.
Gostev
Then how can I investigate why all the data is being sent over the network?unchanged information will never be transmitted over the network from a backup proxy to a backup repository. Because even if "dumb" VMware logic will return us all blocks as changed in some cases, Veeam will only transfer actually changed blocks over the network. This is achieved through comparing a hash of each block that VMware returned to us as "changed" with what is already stored in the backup repository. So, the "smart" B&R logic you want is already there
"Statistics" and "Report" does not have the necessary information.
-
- VP, Product Management
- Posts: 7081
- Liked: 1511 times
- Joined: May 04, 2011 8:36 am
- Full Name: Andreas Neufert
- Location: Germany
- Contact:
Re: B&R is not smart enough?
Do you perform direct backup over WAN links?
Or is it a backup copy job?
Or is it a backup copy job?
-
- Expert
- Posts: 101
- Liked: 3 times
- Joined: Apr 27, 2013 12:10 pm
- Contact:
Re: B&R is not smart enough?
This is a "Backup Job"
-
- Veeam Software
- Posts: 21139
- Liked: 2141 times
- Joined: Jul 11, 2011 10:22 am
- Full Name: Alexander Fogelson
- Contact:
Re: B&R is not smart enough?
How do you tell it is sending the entire VM data? Could you also describe your setup in a bit more details - what kind of repository are you using, how is it added to Veeam B&R? What are the Read and Transferred counter values reported in the job stats?
-
- Expert
- Posts: 101
- Liked: 3 times
- Joined: Apr 27, 2013 12:10 pm
- Contact:
Re: B&R is not smart enough?
Default Local Repository (Windows).
Cannot attach screenshot with job stats. Forum do not support it?
Processed 502.1GB (14%)
Read 476.2GB
Transferred 446.7GB (1.1x)
Cannot attach screenshot with job stats. Forum do not support it?
Processed 502.1GB (14%)
Read 476.2GB
Transferred 446.7GB (1.1x)
-
- Veteran
- Posts: 3077
- Liked: 455 times
- Joined: Aug 07, 2018 3:11 pm
- Full Name: Fedor Maslov
- Contact:
Re: B&R is not smart enough?
You may use any external image hosting resource you like to embed images right into your forum posts. Imgur, OneDrive, Sharepoint, anything you are comfortable with.
However, since there are still lots of unclear details about your environment and the whole situation in general, instead of fortune-telling, I would suggest raising a support case, let our engineers check your logs, determine what caused the full backup (if it's really a full backup ) and slow data transfer speeds, and suggest the best ways to address these issues. Plus we will also be able to review the technical details collected by the engineers and come up with suggestions when needed.
However, since there are still lots of unclear details about your environment and the whole situation in general, instead of fortune-telling, I would suggest raising a support case, let our engineers check your logs, determine what caused the full backup (if it's really a full backup ) and slow data transfer speeds, and suggest the best ways to address these issues. Plus we will also be able to review the technical details collected by the engineers and come up with suggestions when needed.
-
- Veeam Software
- Posts: 21139
- Liked: 2141 times
- Joined: Jul 11, 2011 10:22 am
- Full Name: Alexander Fogelson
- Contact:
Re: B&R is not smart enough?
Looks like it is the first scenario with the changed VM ID due to improper VM migration with VM re-registration. As mentioned above, in this case Veeam B&R treats this VM as a completely new one and there's no way to tell it it is actually the same VM (what if it is not?).
Anyway, with 3.5ТВ VM and a 40Mbps link you should avoid active fulls by all means, this is a killer. Consider switching to local backups with remote backup copy jobs using WAN acceleration.
Anyway, with 3.5ТВ VM and a 40Mbps link you should avoid active fulls by all means, this is a killer. Consider switching to local backups with remote backup copy jobs using WAN acceleration.
-
- Service Provider
- Posts: 6
- Liked: 8 times
- Joined: Feb 18, 2020 4:24 pm
- Full Name: Ken Simon
- Contact:
Re: B&R is not smart enough?
I am not sure why this is a Veeam issue.
You made major changes to your infrastructure. Including data and hardware and software. Changed block information was lost during the process. Either that or Vmware or Veeam is now seeing the disks as new disks.
The expected result would be that, for the next backup, CBT would be unavailable, and the entire disk would need to be backed up.
If you were doing a local backup + copy, this likely would have been fine. Since you are backing up over a slow link, yes, it will take a long time.
Perhaps you could make a local backup and use it to seed a remote backup?
It sucks when your infrastructure is not up to your needs. I know this from experience.
You made major changes to your infrastructure. Including data and hardware and software. Changed block information was lost during the process. Either that or Vmware or Veeam is now seeing the disks as new disks.
The expected result would be that, for the next backup, CBT would be unavailable, and the entire disk would need to be backed up.
If you were doing a local backup + copy, this likely would have been fine. Since you are backing up over a slow link, yes, it will take a long time.
Perhaps you could make a local backup and use it to seed a remote backup?
It sucks when your infrastructure is not up to your needs. I know this from experience.
-
- Expert
- Posts: 101
- Liked: 3 times
- Joined: Apr 27, 2013 12:10 pm
- Contact:
Re: B&R is not smart enough?
Yesterday, with a progress of ~ 90%, there was a network failure at the ISP, and the backup job ended with an error.
After I manually started the backup job, the backup of this VM started not from 90%, but from scratch!
I can not believe that B&R has simply lost ten days of backup process, and I have to start all over again.
After I manually started the backup job, the backup of this VM started not from 90%, but from scratch!
I can not believe that B&R has simply lost ten days of backup process, and I have to start all over again.
-
- Product Manager
- Posts: 2581
- Liked: 708 times
- Joined: Jun 14, 2013 9:30 am
- Full Name: Egor Yakovlev
- Location: Prague, Czech Republic
- Contact:
Re: B&R is not smart enough?
Hi Maa,
Backup Job relies on the VMware snapshot - that means we will trigger VM snapshot, read and transfer VM data to the target storage and delete snapshot afterwards. If job is interrupted and triggered again, we will yet again initiate a VMware snapshot, that will contain a stable VM state at that moment. There is no way for backup job to turn back time and ask VMware to snapshot your VM in the state from 10 days ago if job fails - that is to process from where we was interrupted.
Having a backup job running over internet for 10 days does not seem to be very efficient approach - consider a fast site-local backup, followed by Backup Copy Job afterwards - out of all benefits for that approach I can note your production storage will love you for it - as VM snapshots will not be held for the entire offsite-copy cycle, as well as Backup Copy does support network interruptions, and in case your VM has multiple disks, we will start from the disk we were interrupted on previously.
/Thanks!
Backup Job relies on the VMware snapshot - that means we will trigger VM snapshot, read and transfer VM data to the target storage and delete snapshot afterwards. If job is interrupted and triggered again, we will yet again initiate a VMware snapshot, that will contain a stable VM state at that moment. There is no way for backup job to turn back time and ask VMware to snapshot your VM in the state from 10 days ago if job fails - that is to process from where we was interrupted.
Having a backup job running over internet for 10 days does not seem to be very efficient approach - consider a fast site-local backup, followed by Backup Copy Job afterwards - out of all benefits for that approach I can note your production storage will love you for it - as VM snapshots will not be held for the entire offsite-copy cycle, as well as Backup Copy does support network interruptions, and in case your VM has multiple disks, we will start from the disk we were interrupted on previously.
/Thanks!
-
- VP, Product Management
- Posts: 7081
- Liked: 1511 times
- Joined: May 04, 2011 8:36 am
- Full Name: Andreas Neufert
- Location: Germany
- Contact:
Re: B&R is not smart enough?
As we shared your concept do not follow the best practices at all and you will run into issues as described.
Backup always locally for fast backup and restore. Then use our Backup Copy Jobs to copy data to other places.
In case of small or unreliable WAN links you can use our WAN accelerators that enable resume on disconnect together with optimized data transport for small WAN links.
The reason for the retry is that the VMware API do not allow the continuation in many cases. As after a failed backup we have to remove the snapshot (with the preserved state) from the source VM so that the datastore will not run full over time. Usually this is not at all an issue with the above mentioned concept.
Backup always locally for fast backup and restore. Then use our Backup Copy Jobs to copy data to other places.
In case of small or unreliable WAN links you can use our WAN accelerators that enable resume on disconnect together with optimized data transport for small WAN links.
The reason for the retry is that the VMware API do not allow the continuation in many cases. As after a failed backup we have to remove the snapshot (with the preserved state) from the source VM so that the datastore will not run full over time. Usually this is not at all an issue with the above mentioned concept.
Who is online
Users browsing this forum: No registered users and 32 guests