-
- Veeam Software
- Posts: 889
- Liked: 160 times
- Joined: Feb 16, 2012 7:35 am
- Full Name: Rasmus Haslund
- Location: Denmark
- Contact:
Cluster Change Block Tracking
Please correct me if I am wrong, but from what I have understood Veeams Hyper-V CBT saves CBT information locally on each individual node in a Hyper-V cluster.
This means if a host is offline for whatever reason, CBT will not be used for any VM in that cluster as soon as any host is offline.
Would you please consider saving CBT information together with the VM files, in a similar way as how ESXi handles it so that even if a host is offline (maybe being service or hardware fault), CBT can still be used for backups.
Thank you in advance.
This means if a host is offline for whatever reason, CBT will not be used for any VM in that cluster as soon as any host is offline.
Would you please consider saving CBT information together with the VM files, in a similar way as how ESXi handles it so that even if a host is offline (maybe being service or hardware fault), CBT can still be used for backups.
Thank you in advance.
Rasmus Haslund | Twitter: @haslund | Blog: https://rasmushaslund.com
-
- VP, Product Management
- Posts: 27562
- Liked: 2858 times
- Joined: Mar 30, 2009 9:13 am
- Full Name: Vitaliy Safarov
- Contact:
Re: Cluster Change Block Tracking
Hi Rasmus,
You're correct that CBT data is stored on the host, since CBT driver is installed on the hosts directly after you add them to the Veeam B&R console and is activated on the first job run against them.
Thanks for your feedback!
You're correct that CBT data is stored on the host, since CBT driver is installed on the hosts directly after you add them to the Veeam B&R console and is activated on the first job run against them.
Thanks for your feedback!
-
- Chief Product Officer
- Posts: 32230
- Liked: 7591 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: Cluster Change Block Tracking
In fact, this was our original implementation (when we released "v1" of our Hyper-V support). Something did not work well with storing CBT files on CSV though (don't remember what exactly, but major issues) - so we have re-architected this in the following release, resulting in the current implementation.rhaslund wrote:Would you please consider saving CBT information together with the VM files, in a similar way as how ESXi handles it so that even if a host is offline (maybe being service or hardware fault), CBT can still be used for backups.
-
- Influencer
- Posts: 11
- Liked: 2 times
- Joined: Mar 05, 2015 1:06 pm
- Contact:
Re: Cluster Change Block Tracking
Are you telling me that it's working now, Vitaly???
We are using CBT and we can see that the backed-up virtual machine is read completely. But only a few MB / GB were written to the repository.
Maybe I missunderstand something, but shouldn't CBT prevent that the whole source is read?
We are running a Windows 2012 R2 Cluster with 8 hosts and one huge CSV
We are using CBT and we can see that the backed-up virtual machine is read completely. But only a few MB / GB were written to the repository.
Maybe I missunderstand something, but shouldn't CBT prevent that the whole source is read?
We are running a Windows 2012 R2 Cluster with 8 hosts and one huge CSV
-
- Veeam Software
- Posts: 21167
- Liked: 2153 times
- Joined: Jul 11, 2011 10:22 am
- Full Name: Alexander Fogelson
- Contact:
Re: Cluster Change Block Tracking
CBT should work fine, you'd better contact technical support directly and investigate why it is not used.
-
- VP, Product Management
- Posts: 27562
- Liked: 2858 times
- Joined: Mar 30, 2009 9:13 am
- Full Name: Vitaliy Safarov
- Contact:
Re: Cluster Change Block Tracking
If you're using CBT, then entire VM image should not be read. Do you see any warning messages in the job session?
-
- Novice
- Posts: 7
- Liked: never
- Joined: Aug 21, 2013 10:00 pm
- Full Name: Bob Digby
- Contact:
[Merged] Failed to create Change Tracking Time Stamp
We use Veeam to backup our Hyper-V VM's on four nodes in a cluster on Server 2012 Datacenter.
We have had no problems with backing up or changing tracking messages recently.
Last night I turned one of our nodes within our cluster off to do some diagnostics outside of windows. When Veeam went to backup any VM on the cluster it came up with the following message:
"Failed to create change tracking time stamp for virtual disk file C:\ClusterStorage\Volume1\VMname\etc.vhdx"
It then reread the whole VM taking hours instead of minutes to backup each VM.
As soon as I turned the node back on (not actually hosting any VM's) the next backup worked fine without that message.
Does anyone know why this happened? I found this KB http://www.veeam.com/kb1934 which mentions if a node is offline it may cause problems but surely if the files are kept on the cluster and not the node why does every node have to be online for Veeam to work properly?
Cheers
Bob
We have had no problems with backing up or changing tracking messages recently.
Last night I turned one of our nodes within our cluster off to do some diagnostics outside of windows. When Veeam went to backup any VM on the cluster it came up with the following message:
"Failed to create change tracking time stamp for virtual disk file C:\ClusterStorage\Volume1\VMname\etc.vhdx"
It then reread the whole VM taking hours instead of minutes to backup each VM.
As soon as I turned the node back on (not actually hosting any VM's) the next backup worked fine without that message.
Does anyone know why this happened? I found this KB http://www.veeam.com/kb1934 which mentions if a node is offline it may cause problems but surely if the files are kept on the cluster and not the node why does every node have to be online for Veeam to work properly?
Cheers
Bob
-
- Veteran
- Posts: 7328
- Liked: 781 times
- Joined: May 21, 2014 11:03 am
- Full Name: Nikita Shestakov
- Location: Prague
- Contact:
Re: Cluster Change Block Tracking
Hello Bob, since CBT for cluster is node-based, when the host was powered off CBT was not used, that`s why it took longer to produce the backup.
So, the behavior is expected. Please review the thread for more info and ask additional questions if you have any. Thanks!
So, the behavior is expected. Please review the thread for more info and ask additional questions if you have any. Thanks!
-
- Novice
- Posts: 7
- Liked: never
- Joined: Aug 21, 2013 10:00 pm
- Full Name: Bob Digby
- Contact:
Re: Cluster Change Block Tracking
Hi Shestakov, why is this the case do you know?
Surely if you use a cluster the CBT should be stored on the cluster. The whole point of a cluster is to have redundancy and some people even power down nodes during the evenings when things are quiet in a large environment to save power. Having to have every node operational in the cluster for CBT to work seems like a bad design?
Thanks
Bob
Surely if you use a cluster the CBT should be stored on the cluster. The whole point of a cluster is to have redundancy and some people even power down nodes during the evenings when things are quiet in a large environment to save power. Having to have every node operational in the cluster for CBT to work seems like a bad design?
Thanks
Bob
-
- VP, Product Management
- Posts: 27562
- Liked: 2858 times
- Joined: Mar 30, 2009 9:13 am
- Full Name: Vitaliy Safarov
- Contact:
Re: Cluster Change Block Tracking
Hi Bob, see Anton's response above for this decision made:
Thanks!Gostev wrote:Something did not work well with storing CBT files on CSV though (don't remember what exactly, but major issues) - so we have re-architected this in the following release, resulting in the current implementation.
-
- Novice
- Posts: 7
- Liked: never
- Joined: Aug 21, 2013 10:00 pm
- Full Name: Bob Digby
- Contact:
Re: Cluster Change Block Tracking
Ok fair enough I see this as a major down point though will this be re looked into at any stage do you know?
-
- Novice
- Posts: 7
- Liked: never
- Joined: Aug 21, 2013 10:00 pm
- Full Name: Bob Digby
- Contact:
Re: Cluster Change Block Tracking
In fact re reading Gostev comment I quote "so we have re-architected this in the following release, resulting in the current implementation." does this mean CBT should work now if a host in a cluster is offline or not? Can you confirm
Thanks
Thanks
-
- Veeam Software
- Posts: 21167
- Liked: 2153 times
- Joined: Jul 11, 2011 10:22 am
- Full Name: Alexander Fogelson
- Contact:
Re: Cluster Change Block Tracking
No, this means that initially it was designed to work so, but then was re-architected to keep CBT data on each host, as you have observed it.
-
- Novice
- Posts: 7
- Liked: never
- Joined: Aug 21, 2013 10:00 pm
- Full Name: Bob Digby
- Contact:
Re: Cluster Change Block Tracking
Ok fair enough I see this as a major down point though will this be re looked into at any stage do you know?
-
- VP, Product Management
- Posts: 27562
- Liked: 2858 times
- Joined: Mar 30, 2009 9:13 am
- Full Name: Vitaliy Safarov
- Contact:
Re: Cluster Change Block Tracking
Not in the short term future, however if read some bloggers, for example mentioned in this thread, you can see that alternative ways of addressing this behavior might arise.
-
- Enthusiast
- Posts: 33
- Liked: never
- Joined: Mar 15, 2015 6:47 am
- Full Name: Krishnakuamr
- Contact:
[MERGED] CSV and CBT
standalone Hyper-V VM's backup gives different result if the VM resides in the local disk and in the CSV volume.
For VM running from the local disk, the backup go through well with out any errors (Full or Incremental). But for the VM' s from CSV volume gives warning like " Failed to flush change tracking data. Using full scan incremental". The backup for the VM's in the CSV even it is full or incremental taking same time and amount of data to backed up.
This is same for CSV from iSCSI or FC storage.
Any clue?
For VM running from the local disk, the backup go through well with out any errors (Full or Incremental). But for the VM' s from CSV volume gives warning like " Failed to flush change tracking data. Using full scan incremental". The backup for the VM's in the CSV even it is full or incremental taking same time and amount of data to backed up.
This is same for CSV from iSCSI or FC storage.
Any clue?
-
- Veteran
- Posts: 7328
- Liked: 781 times
- Joined: May 21, 2014 11:03 am
- Full Name: Nikita Shestakov
- Location: Prague
- Contact:
Re: Cluster Change Block Tracking
Hello,
Since CBT for cluster is node-based results for clusters and standalone hosts can be different.
Please read the thread for more info and ask additional questions if you have any.
Thanks!
Since CBT for cluster is node-based results for clusters and standalone hosts can be different.
Please read the thread for more info and ask additional questions if you have any.
Thanks!
-
- Enthusiast
- Posts: 33
- Liked: never
- Joined: Mar 15, 2015 6:47 am
- Full Name: Krishnakuamr
- Contact:
Re: Cluster Change Block Tracking
Even we tested the CBT with clustered nodes still receiving same result "failed to flush change tracking data"
-
- Veeam Software
- Posts: 21167
- Liked: 2153 times
- Joined: Jul 11, 2011 10:22 am
- Full Name: Alexander Fogelson
- Contact:
Re: Cluster Change Block Tracking
Is there any reason for configuring CSV without a cluster? In case of CSV, changes are tracked on all cluster nodes, to avoid possibility of data loss. Since there're no more nodes, Veeam B&R lacks CBT information from them and performs full scan of VM disks to calculate changes to be transferred.
-
- Lurker
- Posts: 2
- Liked: never
- Joined: Aug 25, 2014 10:09 am
- Full Name: Danny Casier
- Contact:
[MERGED] Failed to create change tracking time stamp
I have for all the backup jobs this waring
10/06/2015 21:36:38 :: Failed to create change tracking time stamp for virtual disk file C:\ClusterStorage\Volume1\Hyper-V\xxx\xxx_disk_1.vhdx
This error occures every time.
What can causes the errors and how to solved the problem?
Tnx
Dan
Support ID 00314642
10/06/2015 21:36:38 :: Failed to create change tracking time stamp for virtual disk file C:\ClusterStorage\Volume1\Hyper-V\xxx\xxx_disk_1.vhdx
This error occures every time.
What can causes the errors and how to solved the problem?
Tnx
Dan
Support ID 00314642
-
- VP, Product Management
- Posts: 27562
- Liked: 2858 times
- Joined: Mar 30, 2009 9:13 am
- Full Name: Vitaliy Safarov
- Contact:
Re: Failed to create change tracking time stamp
Hi Dan,
It's hard to say much about this error without seeing full debug log. What did our support engineer advise to do? Did you shutdown any nodes from the Hyper-v cluster by any chance?
Thanks!
It's hard to say much about this error without seeing full debug log. What did our support engineer advise to do? Did you shutdown any nodes from the Hyper-v cluster by any chance?
Thanks!
-
- Lurker
- Posts: 2
- Liked: never
- Joined: Aug 25, 2014 10:09 am
- Full Name: Danny Casier
- Contact:
Re: Cluster Change Block Tracking
Yes we have a node down in the cluster.
But this should not result in problems with making backups. Even with virtual machines that were not on that node.
But this should not result in problems with making backups. Even with virtual machines that were not on that node.
-
- Veeam Software
- Posts: 21167
- Liked: 2153 times
- Joined: Jul 11, 2011 10:22 am
- Full Name: Alexander Fogelson
- Contact:
Re: Cluster Change Block Tracking
Actually, it does result in inability to use CBT, since, as mentioned above, CBT information is stored on nodes themselves.
-
- Lurker
- Posts: 2
- Liked: never
- Joined: Jul 07, 2015 3:42 pm
- Full Name: Timo
- Contact:
Re: Cluster Change Block Tracking
Just to let Veeam know: We're also affected by this issue.
Running a five node Hyper-V Cluster we let System Center Virtual Machine Manager decide whether to turn off individual hosts based upon utilization. This Feature is called Power Optimization. Talking about buzzwords like private cloud and green IT it seems quite reasonable to activate such a feature.
Unfortunately at this time it seems we have to choose between wasting energy or exceeding the backup window and increasing system load by disabling CBT. This is not satisfying.
I already noticed that Windows Server 2016 will bring its own native implementation of CBT. Yet it will take some time until release and afterwards updating hosts. So if it's possible for Veeam to store CBT on the CSV, this might be a solution?
Running a five node Hyper-V Cluster we let System Center Virtual Machine Manager decide whether to turn off individual hosts based upon utilization. This Feature is called Power Optimization. Talking about buzzwords like private cloud and green IT it seems quite reasonable to activate such a feature.
Unfortunately at this time it seems we have to choose between wasting energy or exceeding the backup window and increasing system load by disabling CBT. This is not satisfying.
I already noticed that Windows Server 2016 will bring its own native implementation of CBT. Yet it will take some time until release and afterwards updating hosts. So if it's possible for Veeam to store CBT on the CSV, this might be a solution?
-
- VP, Product Management
- Posts: 27562
- Liked: 2858 times
- Joined: Mar 30, 2009 9:13 am
- Full Name: Vitaliy Safarov
- Contact:
Re: Cluster Change Block Tracking
There were some issues with storing CBT data on the CSV storage, but, yes, that was our original implementation idea.IT_Timo wrote:I already noticed that Windows Server 2016 will bring its own native implementation of CBT. Yet it will take some time until release and afterwards updating hosts. So if it's possible for Veeam to store CBT on the CSV, this might be a solution?
-
- Veeam Software
- Posts: 631
- Liked: 251 times
- Joined: Sep 27, 2011 12:17 pm
- Full Name: Craig Dalrymple
- Location: Scotland
- Contact:
[MERGED]: CBT data - new host
Can anyone offer an advice on how to handle the following situation.
When we add a new Hyper-V host to our cluster, and rebalance the cluster , existing VMs are moved onto the new host, however when the next backup is run a full backup is taken. I'm guessing it's because the CBT information is stored per host in HyperV , and as the VMs are moved onto the new host, no existing CBt information exists. Is there any way we can update the Veeam job to use the CBT data existing on other hosts, or am I thinking about this all wrong ?
When we add a new Hyper-V host to our cluster, and rebalance the cluster , existing VMs are moved onto the new host, however when the next backup is run a full backup is taken. I'm guessing it's because the CBT information is stored per host in HyperV , and as the VMs are moved onto the new host, no existing CBt information exists. Is there any way we can update the Veeam job to use the CBT data existing on other hosts, or am I thinking about this all wrong ?
-
- Product Manager
- Posts: 6576
- Liked: 773 times
- Joined: May 19, 2015 1:46 pm
- Contact:
Re: Cluster Change Block Tracking
Hi,
Thank you.
Does that happen to all VMs or just to VMs that were moved? Also, if full read occurs that's an expected behaviour.when the next backup is run a full backup is taken
Each host is aware of changes happened to other host's VMs. CBT data has to be cleared after VM migration thus causing full read (not full backup) during next backup run. The fact that you have full backup means that something went go wrong with VM ID. Normally Hyper-V cluster does not change VM's ID, so I suggest you to open a case with support team and post your case ID in this topic.because the CBT information is stored per host in HyperV , and as the VMs are moved onto the new host, no existing CBT information exists.
Thank you.
-
- Veeam Software
- Posts: 631
- Liked: 251 times
- Joined: Sep 27, 2011 12:17 pm
- Full Name: Craig Dalrymple
- Location: Scotland
- Contact:
Re: Cluster Change Block Tracking
aah ..gotcha, the full read was indeed what was happening. Thanks for that
-
- Veeam ProPartner
- Posts: 3
- Liked: never
- Joined: Nov 10, 2015 4:20 pm
- Full Name: Kevin Doel
- Location: Orlando
- Contact:
[MERGED] Backup repository slow after addition of new Host S
Good Morning
On a prior evening we added an HV Host Server to a cluster. That cluster is currently being backed up in Veeam. Following the addition of the new Host Server I rescanned the Cluster to discover the new Server and install agents on it. When my backups started during the next cycle some of the backups have taken an unusually long time to complete the backups/merges. All of the slow backups are being backed up to the same repository. The Repository itself is quite healthy and shows no sign of degradation or bad disks. Do I need to rescan each repository after adding new hosts to the VEEAM B&R? Everything appears to work fine but I exceeded my SLA this morning and do not want that to occur again. Any ideas?
On a prior evening we added an HV Host Server to a cluster. That cluster is currently being backed up in Veeam. Following the addition of the new Host Server I rescanned the Cluster to discover the new Server and install agents on it. When my backups started during the next cycle some of the backups have taken an unusually long time to complete the backups/merges. All of the slow backups are being backed up to the same repository. The Repository itself is quite healthy and shows no sign of degradation or bad disks. Do I need to rescan each repository after adding new hosts to the VEEAM B&R? Everything appears to work fine but I exceeded my SLA this morning and do not want that to occur again. Any ideas?

-
- Veeam Software
- Posts: 21167
- Liked: 2153 times
- Joined: Jul 11, 2011 10:22 am
- Full Name: Alexander Fogelson
- Contact:
Re: Cluster Change Block Tracking
Kevin, please review the thread above for the answer. This thread also has some further details. Thanks.
Who is online
Users browsing this forum: No registered users and 23 guests