Backups SLOWED WAY DOWN after sdelete on drives

[ColtsFanMN]

VBR 6.5
VMware environment, 12 servers backed up.

Previous to running sdelete on drives, total backups ran 9 hours..
total data read (snapshots) is 2.3TB
total data copied to backup was 825GB

Cleaned all my VMs with sdelete. Servers are running fine, no problems.

but now my backups are running about 13 hours..
total data read (snapshots) is 2.3TB
total data copied to backup was 612GB

this happened after running sdelete to clear 'dirty' data blocks on VMs.

the 2 servers that were 'cleaned' of the most dirty blocks, are the ones taking the most additional time.

this is consistent over all backups since using the sdelete utility. 'Read' data is consistent with the files on the servers but everything is slowed way down now.

What could be causing less backup data to take way more time?

[Vitaliy S.]

Hello,

Can you please tell me what sdelete parameters (flags) you used against your virtual disks? Based on your previous posts I see that you're using thick disks and active fulls every day, right?

Thanks!

[Gostev]

sdeleting thin disks will blow them up, which will in turn slow down backups... that's the explanation I have.

[tsightler]

sdeleteing thick disks will cause VMware to report via CBT that the entire disk is used whereas before it likely reported only blocks that had been written by normal operations, this means that on a full backup Veeam won't skip the competely unused blocks automatically, rather it now has to read those blocks to determine they contain only zeros. This is my guess as to the reason.

[ColtsFanMN]

Vitaliy,

Yes, thick disks.. I used the switch -z


Gostev,

My disks are thick.. what do you mean by 'blow them up'?


tsightler,

Users in THIS forum told me to use sdelete to clean the 'dirty' blocks... even after I said that the disks were 'thick'.. so now all my backups going forward will take hours longer than needed? Is this a 'guess' or is this factual information?

[Vitaliy S.]

My disks are thick.. what do you mean by 'blow them up'?

If you had used thin-provisioned disks, then running sdelete against these disks would have expanded them to full provisioned size and you would be forced to do storage vMotion to reclaim virtual disks space.

[ColtsFanMN]

OK.. so that isn't the case.

So what is going on then? Why is it slowed way down then?

[tsightler]

It is factual that the way sdelete works is by writing zeros to all "unused" areas on disk. It is factual that Veeam queries CBT, even during full backups, to attempt determine which areas of the disk have been actually used, no need to read data from areas that have never been written. It is factual that using sdelete to write zeros to all unused areas will cause VMware CBT to always report that all areas of the disk are used. Is it factual that this is causing the issue you are reporting? I don't know since I don't have any details from your system logs, but based on the facts I do know, I believe that this is very likely.

One of the warnings against using sdelete is that it "blows up" thin disk, which is basically the thin version of the same behavior you are seeing with thick disk. If you have a 40GB thin disk that is only using 20GB on disk, but the guest only has 10GB of data, and you then run sdelete, it will zero out the unused data in the guest, but the thin disk will grow (blow up) to the full 40GB. In this scenario it will take longer to backup up, but will backup less data, basically the same behavior you are seeing.

I believe your previous posts were asking about the fact that your backups were larger than your guest data, because data had been deleted from the guest but Veeam still backs up this data as it backs up all blocks containing any data, even if it has been deleted in the guest (that's pretty much the concept of an image based VM backup). Running sdelete can be an effective way to overcome this issue and, as you noted, your backup sizes are now smaller, but it does have this side effect that Veeam itself has to read more blocks, which could make backups take longer. This is true whether the disks are thin or thick.

[tsightler]

BTW, can you tell me your source storage type (i.e. local, SAN, NFS)?

Also, I see you are on 6.5. I suspect moving to v7 and enabling parallel processing might offset a large percentage of the negative performance impact from this change, perhaps all of it.

[ColtsFanMN]

I was not trying to be a smart a## when asking about 'factual vs. guess'... it is a legitimate question in this context, as in 'is this a known issue, or an educated assumption' basically, so I really didn't appreciate the tone of your reply. The 'it is factual...' shoved back at me several times was unnecessary.

and Yes, I am somewhat agitated.. but put yourself in my shoes. I tried to fix an issue with direction from this forum, now my problems are compounded. Running backups at 13 hours is not acceptable for me/us.. so what am I going to do now?

Is there any 'fix' for this problem then?

To answer your question, my VMs run off of two (2) hosts, connected to a 3TB SAN.

[tsightler]

My apologies that the tone of my previous response was offensive, it was not written with that intent at all, although, reading it back, I guess I can see your point. I was only trying to make it clear the parts that I knew as fact vs the parts where I was making an educated guess based on those facts. I can only say, when I wrote that response, I had no thoughts that you were being a "smart a##" nor was I trying to be one, just trying to explain some of the factors involved with the goal of eventually getting to the bottom of your issue.

I'm running some tests in my lab to attempt to reproduce/confirm these assumptions and determine if this issue can be "fixed". As I mentioned above, I suspect simply using v7 and parallel processing would likely help, just because time spent reading zero blocks won't be completely wasted as other VMs can be processed concurrently. Could your share the typical bottleneck statistics from one of your jobs?

[ColtsFanMN]

Apology accepted, and thank you.

Bottleneck statistics have always been source (going back as far as 3 months), before and after the sdelete. One item of note is that my backups before the sdelete were running about 60-65 Mbs transfer rate, and now they are running in the 50-55 Mbs rate. Nothing from a hardware or software stance has changed other than zeroing the blocks with sdelete. Not sure why the throughput would have dropped, but it appears to have anyway. it's not enough to account for the 4 hours difference of course, and the VMs that had changed very little as far as size of backup, still are close in time-to-backup compared to before the sdelete.

[tsightler]

Just to update with the results of my testing. To test I took a small Windows 2K8R2 VM and added a 40GB thick disk, then copied approximately 10GB of ISO files to it, then deleted those files, then ran a backup, results:

Backup time: 10min 31sec
Backup size: 15.8GB

The 40GB "thick" disk backed up in 2min 37sec, and only read 10.8GB total (roughly the size of the data I had copied to that disk and deleted).

I then ran sdelete on that disk to "zero out" the space and ran a new backup,

Backup time: 15min 59sec
Backup size: 8.6GB

So the backup was significantly smaller, as expected, but it took much longer because it had to read the entire 40GB of zeros from the second disk exactly as I described previously. It took 8min 40sec to backup this now zero'd disk. At this point I'm nearly 100% sure this is what occurred in your case as well.

Regarding fixing it, you can probably use procedures similar to the issue with thin disk, basically, use svMotion to convert the disk to thin, then inflate back to thick, but I've never actually performed this procedure. Or, with newer versions you could power off the VM and use the vmkfstools command line tool with "punchzero" option to deallocate the zero blocks, and then convert back to thick.

[sbbots]

I am experiencing the same phenomenon.

It is one particular VM which has a 1TB secondary drive with only 45GB used. After sdelete backups tripled in time; Most likely because the backup is now reading the entire 1TB instead of just 45GB (as tsightler alludes to). But a few months ago I ran sdelete on a similar server with a 1TB drive and ~150GB used and the backup time actually went down after sdelete. That is why this is puzzling.

[Gostev]

Hi, Matt. Basically, this means it was reading the entire 1TB in the first case both before and after sdelete, however most of that 1TB contained "dirty" blocks, which is why sdelete helped to speed up the processing performance (zeroed blocks are much faster to process). Thanks!

[tsightler]

You can get the speed back by either manually converting the VMDK to thin, and then back to thick or by using vMotion, although that method is less reliable than it used to be because of changes in the way vMotion works with ESXi 5.x and newer. Now it will only reclaim unused blocks if the two storage systems use different block sizes, which is much less likely with modern VMFS 5 that uses a fixed block size.

There is another option, although it sounds a little strange at first, but you can actually use Veeam to restore the VM disk in question, although the VM must be powered off to do this. When Veeam performs the VM disk restore task it deletes the old VMDK and creates a brand new VMDK and then writes only blocks that contain actual data, skipping the zero'd blocks. VMware will then know once again that these disks are not used and thus will simply skip them during backup. I've tested this in my lab setup and it seems to work well.

I'd also guess you could do an instant restore of the VM and then use the migrate option to get the same result since the vMotion process would be from the vPower NFS datastore to a VMFS datastore it should reclaim the zero'd space, but I have not tried that yet.

[VladV]

If downtime isn't an issue (or maybe a scheduled downtime is a possibility) you can run --punchzero on the vmdk in question. That will delete all zeroed out blocks from the disk and converts it to thin.

More details here http://pubs.vmware.com/vsphere-51/index ... 43902.html

[dellock6]

Tom, another "trick" to workaround the problem of not having different block sizes in VMFS5 when doing storage vmotion is to have a NFS volume around and use it as a temporary target for the storage vmotion.

[tsightler]

Sure, did I not mention NFS in my post (I guess not, only mentioned vPower NFS, but the point is the same).

[sbbots]

I am still seeing very long backup times even during incrementals (which take almost as long as an active full). Using Tom's advice I am going to restore the VM disk in question after our daily backup completes tonight.

[tsightler]

I wouldn't anticipate incremental runs being impacted by this. Also, make sure your backup is good first, perhaps to a test restore or something first just to be 100% sure. Not saying I don't trust the backup to be good, but I'm a very conservative person when it comes to protecting my (or anyone's) data, and there is obvious something going on that is unexplained, so I would at least clone the existing VM first assuming you have enough space to do so. That way, if for some reason something goes wrong with the restore you still have the clone to fall back.

[sbbots]

Unfortunately, restoring the entire VM from scratch (which recreates the VMDK files) did nothing to solve the issue. An incremental backup which used to take ~10 minutes now takes over 2 hours. I'll work on this later today and figure it out.

[VladV]

Are you sure that indeed it's running an incremental job using changed block tracking? As tom stated, the incremental job shouldn't be impacted by zeroed out blocks or dirty blocks as they are not changed between jobs.

Also, have you tried punching zeroes in the vmdk as I've written above?

Cheers

[sbbots]

Are you sure that indeed it's running an incremental job using changed block tracking? As tom stated, the incremental job shouldn't be impacted by zeroed out blocks or dirty blocks as they are not changed between jobs.

Also, have you tried punching zeroes in the vmdk as I've written above?

The job is definitely using CBT. It is one particular VM that is having the issue. I zeroed out the blocks as suggested and the disks, which were thick provisioned before, are now thin provisioned after a restore; One drive is 250GB using only ~60GB, the other drive is 1TB using only ~6GB.

Yet the long incremental backups continue. Last night's backup took over 4 hours, of which almost all of the time was spent on that one troublesome VM. I will go ahead and submit a support ticket. Thanks for the help.

[sbbots]

I stand corrected. The backups for that server were no longer using CBT after the day I ran sdelete. The proxy for host #2 is also the problematic VM (GMELSW01). That server is lightly used so it was a temporary proxy until I could create a dedicated proxy. Looking at the logs; When I first setup the proxy on that server it was using CBT, but for whatever unexplainable reason it stopped using CBT after I ran sdelete. Once I removed the proxy responsibility from that server the problem went away. Yet it worked fine before I ran sdelete? Weird.

Before



Then I ran sdelete...

After


All subsequent incremental backups ran this slow.

Then I removed the proxy duty from that server (so only the proxy on host #1 is used) and everything is back to normal...

[dellock6]

Sounds like two different issues to me. CBT is always disabled for hotadd proxies, and you just said you second proxy is a VM.
Any chance the temporary proxy was added at the same time you run the sdelete?

[sbbots]

The hotadd proxy was added weeks before and the log shows CBT running even before I ran sdelete. Look at the before and after screens above with red arrows pointed at both the proxy being used and the mode. Am I reading the log file incorrectly?

[sbbots]

CBT is always disabled for hotadd proxies

That basically says everything. The lesson is; Always use a dedicated server for the proxy. At other remote locations I use a dedicated VM for Veeam (and thus the proxy), which I cannot backup anyway since I use DirectPath to attach the tape drive to the Veeam VM = no VMware snapshot. The proxy I was using here was temporary while I setup SolarWinds, but I created a new VM just to use as a proxy so I get the benefit of CBT on this server.

Sorry for the long-winded posts and thanks for your patience with someone still relatively new to IT

[veremin]

At other remote locations I use a dedicated VM for Veeam (and thus the proxy), which I cannot backup anyway since I use DirectPath to attach the tape drive to the Veeam VM = no VMware snapshot.

Please be aware, that by using VMDirectPath you lock corresponding VM to one physical server, and it won't be possible to many of VMware features, such as FT, vMotion , Storage vMotion , HA , DRS, etc. In such cases, the recommended approach includes usage of redirecting tools.

Additionally, the upcoming release should that problem once and for all.

Thanks.

[fcambot]

What is an "hotadd" proxy ? i have several proxies (which are VM by the way), they are reported with "[hotadd]" in the activity window and CBT is working. I put them in the right place to benefit from hotadd.