Full backup when doing a reverse incremental

[pwinterbourne]

Hi,

I have noticed recently a couple of times when Veeam has decided to do a full back up of one of the vm's in my replication job. This has happened when we either added a new vmdk to the virtual server or when we have had a lot of data change on the drive alongside extending the drive (for example our SQL server admin did a backup of 27GB then deleted 75GB of old files and I extended the drive in windows 2008 R2 by 100GB). In the second example i expected the data read to be roughly 100GB but it ended up reading the entire drive which pushed the backup window in to the production hours.

Can someone explain to me what variables veeam looks at to make this decision?

My other question is a follow on from this, we will shortly be using replication to a DR site and it worries me greatly that if i do any admin jobs like the above then it will force a full replication of the entire vm (900GB) across our 50Mbps wan pipe.

I supposed its not that the replication worries me, its the fact that my production SQL server ends up being used in production hours with a snapshot in place whilst veeam is still running a back up of it. I get slightly concerned when a snapshot is left running on a vm in production. We use an equallogic SAN with 22 x 10K drives which is fairly unutilized but even so, back in the day you could pretty much bet your money on seeing major issues when a snaphot was cmommited during working hours on a production vm. Maybe this is not the case anymore (I am going back to vmware 3.0 days and using fairly rubbish sans).

What are others thoughts on replication during the day, snapshots and performance on the virtual servers they are backing up and replicating? are you still nervous? or is that old hat and I need to man up!

thanks,

Paul

[homerjnick]

We run a Netapp N Series 3600 with a mixture of SATA and FC disk with 2gig fibre cards...fairly low spec SAN and we have 3 ESXi 5.0 Blade servers running about 150 VM's and we have running backups during production hours as well as replication to meet our RPO's and RTO's and very rarely do any users notice when snapshots are being taken/committed.

We have SQL servers, Exchange, DC's, web servers all virtual and all backed up during working hours fine...

I think with the enhancements of ESXi5 and Veeam 6.5 users will not notice anything...

[pwinterbourne]

thanks for the reply,

thats good to hear!. If you dont mind me asking ....out of interest how many spindles are you using on your SAN, Raid level and spindle speed?

My snapshot commited ok this morning so that was good. I always wondered how snaphosts commit in vmware because if you have one snapshot, when you commit a snapshot another snapshot file is created (which of course grows and then needs to be commited once the original snapshot has been commited). At some point vmware needs to commit the final snapshot so there must be some algortihm that says "hey, I am only a 5 MB snapshot so its safe to commit me without creating yet another snapshot file" or something!!.

Anyway i digree, thats more a vmware thing rather than veeam.

Still owuld be interested to know why veeam does a full read on the disk when either a new disk is added/extended. Is this the reason why? or is it a combination of the fact that a lot of data changes have occured and this or is it just because a lot of data changes have occured so Veeam decides to do a full read of the disk again. It might not even be a veeam thing, perhaps vmware resets the CBT or something I dont know?

[foggy]

Still owuld be interested to know why veeam does a full read on the disk when either a new disk is added/extended. Is this the reason why?

Paul, it is fully expected that in case of adding a new disk to the source VM it is being replicated to the target location in its entirety (as it is a new disk and should be added to the replica VM along with all its content).

Regarding the changing disk size of the original VM, the entire drive is read in this case also. Moreover, all replica restore points except the most recent are deleted. This is explained by the fact that replication restore points are stored as VMware snapshots, which do not support the base disk size change. Do you observe this behavior on your replicas?

[homerjnick]

If you dont mind me asking ....out of interest how many spindles are you using on your SAN, Raid level and spindle speed?

FC disks are 15,000rpm and SATA 7200rpm..raid is raid-dp which is Netapp's own raid and is basically a raid 6 with double parity and a funky optimisation built in...62 spindles in our SAN...

We are upgrading to 8gig fibre shortly but as I say even with a measly 2 gig fibre connection to the SAN and having 30 servers, 120 user desktops, they never notice when backup/replication/committing of a snapshot is completed.

The only thing we used to suffer was boot storms but that was related to Windows XP VM's not being aligned with a Netapp SAN and 1 block of data on a XP disk writing on two blocks of SAN...this was fixed once we moved users to Windows 7...but all that is Netapp related!

The fact is, we have a low-end SAN using an old backplane of an IBM Bladecenter running our entire state on 3 ESXi servers and we backup, replicate and snapshot all the time and never have issues...but that is because we have planned our LUNs well in ESX and they are never pushed to their limits...

[Vitaliy S.]

I have noticed recently a couple of times when Veeam has decided to do a full back up of one of the vm's in my replication job. This has happened when we either added a new vmdk to the virtual server or when we have had a lot of data change on the drive alongside extending the drive (for example our SQL server admin did a backup of 27GB then deleted 75GB of old files and I extended the drive in windows 2008 R2 by 100GB).

Adding new virtual disk to VM configuration cannot not trigger full run of the replication job, while re-sizing virtual disks does cause this behavior (foggy has correctly explained the reason for this behavior in the post above).

In the second example i expected the data read to be roughly 100GB but it ended up reading the entire drive which pushed the backup window in to the production hours.

Be aware that any block modification inside the guest OS (file relocation, deleting files, antivirus and file system scans etc.) changes virtual disk blocks, so VMware CBT marks these blocks as new ones.

I would recommend to look through this topic, since you're planning to replicate over the WAN link: Large VIB file on a small static server

[pwinterbourne]

Hi thanks for all your replies they are very insightful.

At the moment I am only backing up not using replication so all the things that have happened only apply to back ups.

I think the posts have helped but I will explain one specific situation where the full read was done....

1.) I have a vm that has three separate jobs to back up each of its vmdk's
2.) I created a new VMDK and logged in to the associated windows vm and copied 600GB of data from disk2 to the new disk. This *should* have meant that the backup asscoaied with this drive would only read 600GB not the entire drive. I appreciate the thing about defragemtnation, antivirus scanning etc but these do not apply as we do not do full scans and a defrag is not scheduled.
3.) the resulting backup reread the entire drive and no extension of the drive was done (it reread 1.9TB). Even taking in to account any file changes during the day and the fact I moved 600GB off that drive I don't understand why a full read of the drive was kicked off.

In a nutshell I guess what I want to know is:

are there any circumstances under which veeam will perform a full reread of a vmdk when being used in a reverse incremental situation? (apart from extending the drive)

If there are no circumstance then I wont worry too much anymore. however, if there is an internal algorithm in veeam to do a full reread of the drive if say 30% or more changes are incurred on the drive since the last back up then it affects how I design things in the future.

thanks!

p.s. a SAN with 62 drives running over 2GB with a mixture of 15k and Sata on Raid 6 (equiv) to me is pretty good!. I have 26 virtual servers and 10 virtual pc's running on a 22 disk Raid 10 SAS 10K set up with 2Gbps bandwidth. IMHO I would say your set up is on the higher end of spectrum for a medium sized business and I probably wouldn't worry about snapshots as much if I have your SAN (you have a lot of spindles to get that data recommitted and cope with overhead). Having said that your comments have made me feel far more at ease now and I think we may even look at replication at lunch time on some of the vm's.

[chrisdearden]

Hi Paul, I'm a little curious as to why you are splitting a VM into 3 jobs - I'm guessing each of those Disks is a 2TB volume ?

I assume you have changed block tracking working correctly ? something is causing a higher than expected amount of changed blocks - I have seen Volume shadow copies cause some of that. Are you using VSS copies ?

[foggy]

Paul, what Veeam B&R version are you running?

[Vitaliy S.]

are there any circumstances under which veeam will perform a full reread of a vmdk when being used in a reverse incremental situation? (apart from extending the drive)

Veeam B&R will read the entire drive only if VMware CBT data is not available for this disk. In this case Veeam will read the entire drive to compare all the blocks to start transferring only changed blocks to the destination target.

[pwinterbourne]

Hi,

it has 3 jobs because it has three 2TB drives and we copy the vbk to USB drive every night for offsite backup from the veeam server. As the USB drives are only 3TB I cant copy the vbk if it was one job.

CBT seems to be working correctly and we are using veeam 6.1

I may be having a major memory problem here because now I am thinking that I might have increased that drive by 40GB to ease the space issue I was having which by the sounds of it would have kicked off a backup. I am going to read through the VMware logs and see if I can see any info to confirm. I apologise if this is the case!