Inconsistent Processing Rates and CBT

[brupnick]

I've been running VBR 5 for a few weeks now and have noticed that my processing rates vary significantly from day to day and job to job. I have one VM that is 775 GB and will process at over 300 Mb/s. This particular VM shows the backup mode as "HOTADD/NBD without changed block tracking." I have another VM that is 15 GB, but processes at 13 MB/s, showing the backup mode as "HOTADD/NBD with changed block tracking." Both VMs are part of the same job that consists of 42 VMs. This job is being run in Virtual Appliance mode, writes to the same LAN target using reversed incremental mode, "best" compression and "LAN target" storage. This job has inline deduplication enabled as well as changed block tracking.

With all that being said, how is it possible that the giant server that isn't using CBT processes so many times faster than the tiny server that supposedly is using CBT? And for the 775 GB server that doesn't seem to be using CBT, the following lines do exist in the .vmx file:

scsi0:0.ctkEnabled = "TRUE"
scsi0:1.ctkEnabled = "TRUE"
scsi0:2.ctkEnabled = "TRUE"
ctkEnabled = "TRUE"

So why isn't CBT being used for this VM?

Thanks in advance and if there's any more details that I can provide, please let me know.

[Vitaliy S.]

Hello Brian,

Please make sure you meet all the requirements for Change Block Tracking to work. If all the requirements are met, please refer to our support team for assistance.

Thank you!

[brupnick]

Hello Vitaliy-

Based on your link, yes, I do meet those requirements. I just ran a test with a VM that during my normal nightly backups, ran "without changed block tracking." This was a 100 GB VM that took 1 hour and 40 minutes at 17 MB/s. This was the first time backing up this VM with this job, so this would have been a full which I expected. When that job finished, I ran it again to see how the times would compare. For this second run, an incremental, I noticed that it was now running "with changed block tracking." Nothing was changed on the VM, so what caused it to run "without" the first time, but "with" the second (as well as all subsequent)? Nothing was changed on the VM or in the job between the runs, so is this the normal behavior for fulls and incrementals? Will fulls always run "without" because they are just that - fulls?

I apologize if these are very elementary questions, but I'm getting frustrated by the erratic processing rates across my VMs. My nightly jobs back up about 4 TB of data across two jobs, but with CBT and nightly runs, I wouldn't think that these jobs should take the 10-14 hours that they are. I do have tickets open with support regarding these issues, but I thought I would also try here to see if anyone in the community had similar experiences and could give me some advice or direction.

Thank you,
Brian

[Gostev]

Will fulls always run "without" because they are just that - fulls?

Yep, sure. CBT is only used for incremental passes.

The difference in processing rate is usually caused by different amount of changed blocks. More blocks to process means more time it takes, which in turn affects "processing rate". If your VMs are not on the same LUN, then this could also be caused by storage issues - I've seen that before, and more than once.

By the way, with reversed incremental backup mode 4TB and 42 VMs you are getting close to limit where a single backup server stops fitting typical backup window, so I would consider deploying a second server, or changing backup mode to incremental (which provides up to 3 times better incremental backup performance).

[brupnick]

That make sense Gostev; thanks for clearing that up.

Could you elaborate on what type of storage issues I might be encountering and how I might be able to identify them?

As for the jobs, I'm using reversed incrementals because we still have a requirement to go to tape. Our tape systems are starting to show their age, so we're really not comfortable using traditional incrementals because of their reliance on previous tapes. For now, reversed incrementals seemed to give us the best of both because I would be running incremental jobs, but still have the most recent copy of my data in a full that I could write to tape.

I understand what you're saying about processing rate, however, I'm still getting what I deem to be horrible performance from a majority of my VMs. I say that because as I mentioned, some of my VMs will get 250-300 MB/s, while others will get 9 MB/s. Even if that 9 MB/s is 20 MB/s, it's still more than 10x slower than my best VMs. Even if it's unreasonable to expect all VMs to hit 300 MB/s, I was really hoping that each one would be at least 100 MB/s for incrementals. Based on what I've heard about VBR and CBT, I don't think this is entirely unreasonable, but if it is, please let me know.

As I mentioned, I still have VMs that show "without changed block tracking" when running incrementals. As far as I can tell, everything required is and has been in place: v7 tools, ctkEnabled = TRUE and scsi#:#.ctkEnabled = "TRUE" for all the disks. So given that, I'm still frustrated as to why they don't seem to be utilizing this and why they take so long to run. To your point about backup windows, I actually have two separate appliances that back up 33 and 42 VMs each. A bad night of incrementals for the 42 VMs takes about 14 hours, so you are correct that it is starting to approach a 24 hour backup window. That being said, I really believe that this window can and should be significantly smaller if CBT and VBR were working together as I thought they would. Again, I'm still new to this, so maybe my expectations are too high.

[Gostev]

By storage issues I mean when some LUNs provide much slower I/O than other LUNs. Different controller or controller cache settings, different hard drives, etc. Ironically, from a few top threads on backup performance problems that I still remember (because users there were screaming out loud how our product sucks, and it was pretty ugly) all ended up with users coming back and confirming the issue was nailed down to either source LUN or target storage performance, and has nothing to deal with Veeam.

It is perfectly fine to have 10x slower processing rate for some VMs, if these VMs have 10x more changed blocks. Some VMs are very special in a way that they generate tons of disk changes daily, like 30% or more of disk blocks are changed (for example, Exchange or SQL do this because of transaction logs). While for other VMs with "regular" apps (over 90% of your VMs) daily changes are typically in 2% ball park. Here is your 15x difference in processing rate right there.

Make sure you caught "without changed block tracking" displayed when processing VMDK, and not configuration files (the latter is fine). If this is still for disks, then our support should be able to easily figure out from logs why CBT is not being used. Generally speaking, CBT is pretty reliable - it is more of an exception when you need to "babysit" and fix it. Usually it works fine for all VMs right away.

The key to faster reverse incremental backup is very fast target storage, because this backup mode does 3x more I/O than regular incremental mode. Target storage is almost always a primary bottleneck. Also, make sure CPU on backup servers is not sitting at 100% when the job is running, I mentioned you are using Best compression in a VM, while this compression level was really designed for 2 CPU 8+ core physical servers.