Serious performance decrease since 6.x!!!

[dkiernan]

I've talked to support and they have told me I "do not need to worry about" the performance metrics in my backup job.

I am seeing performance in the KBytes/s range on several VM's, and my overall backup window has ballooned to over 12 hours for a reverse incremental job. Most VM's process at 3 MB/s or less.

I need to worry about the performance, as the backup window is now extending into production hours.

Every 2-3 months, I delete the entire backup chain and force an active full backup on a Saturday. This job takes over 35 hours.

Does anyone have any performance tuning or other diagnostics I can look at to see if I'm getting the best performance I can expect?

[Vitaliy S.]

Can you please tell us what bottleneck stats you have for the job in question? Also what is your destination target storage?

[veremin]

Also, it might be worth providing previous bottleneck statistics, so that, we can catch the difference between those? By the way, have there been any changes introduced recently in your environment? Thanks.

[dkiernan]

Bottleneck says "Target" - which I seriously doubt.

I am able to copy the VBK file from the target to a USB 3.0 disk at nearly 6GBytes/min. This suggests throughput to the repository is *not* an issue.

The backup repository is a bare-metal Windows share with a 10-disk RAID-5 array over a 1gbit LAN link.

Nothing "substantial" has changed in the network architecture - other than upgrading Veeam versions.

I have begged Veeam support to help me identify what might lead to these drop-offs in performance - that's when I was told that I "don't need to worry" that the Read rate is less than 1MB/s. Well...I *DO* need to worry about it because my backup window is unacceptable.

An Exchange 2010 server with several disks which take over 1.5 hours each - total backup time (this *ONE* vm): 6.5 hours!!!

disk 1: 80gb, 12gb read, 2MB/s
disk 2: 280gb, 7.0gb read, 1MB/s
disk 3: 200gb, 2.9gb read, 609KB/s <--- WHAT??!?!?!
disk 4: 150gb, 7gb read, 2MB/s

Comparing to other customers (yes, yes, I understand - each installation is different, etc.) I have running Veeam, I have one in particular with a similar Exchange 2010 server (not quite as big, but 4 v-disks, and pretty "busy"), which backs up in 1/3rd the time. The backup repository is also over a 1gb LAN link, but a 4-disk RAID-5 Netgear NAS (abysmal units in their own right - but this still outperforms the aforementioned setup).

disk 1: 80gb, 9gb read, 5MB/s
disk 2: 220gb, 9.5gb read, 5MB/s
disk 3: 200gb, 6gb read, 5MB/s
disk 4: 50gb, 7gb read, 5MB/s
total: 2.5 hours.

Clearly the transfer rate of 5MB/s is the difference - my question is: WHAT DO I NEED TO CHANGE on the other job to achieve 5MB/s??!?!?!/

My point is - I can't find anything to *account for the difference*. Meaning, I am attempting to reconcile the various installations, and can't identify items to add/remove in order to make different customers an apples-to-apples comparison.

[dellock6]

You said in the opening post you are using reversed incremental, and with this method target is absolutely expected to be the bottleneck.
Reversed incremental has a high I/O impact on the target, and requires 3 I/O per saved byte, and even more this activity is random. A simple copy of a file cannot be at all compared to the I/O pattern created by reversed incremental.
If you have enough space to hold two full backup files, the quickest solution is to switch to forward mode.

[dkiernan]

If you also read the OP, you'll understand that the performance has degraded significantly over time - running reverse incremental jobs since installing Veeam 5.0.

Incidentally, I use reverse incrmental so I can copy only the .vbk and .vbm files to USB. Using this method, a single USB disk is a self-contained DR point.

The Reverse Incremental backup is one of the selling points - and great features - of Veeam.

I have been less satisfied with Veeam support - essentially brushing off our concerns and issues as not important or worth investigating.

The point is - my backup window has increased from 8 hours to 12...13..15... hours.

Again, I can account for some of this - increased VM size, new VM's etc. I cannot, however, reconcile the FULL increase in time window - or more specifically, the *SHARP DECREASE* in the READ RATE.

I am looking for assistance in identifying this issue.

[dellock6]

I read the OP and it also states that a new full backup is taking 35 hours. A full backup is always using the same forward mode even if it is inside a reversed chain, reverse method comes into play from the second day. So even the performances of the full backup job are low.

Also you are talking about the performances of a copy towards an USB drive, but this one again is sequential read from repository to the usb drive. What about the performances of the active full backup you run periodically? Can you give us more details, like overall size of the VMs, final size of the VBK, and the time it takes to be executed?

Also, you say repository is a windows share, so which Veeam server is acting as a proxying server towards the smb share? IF you only have one Veeam server it's this one.

Exchange is known to create high change rate even with few mail activity on it, thus producing a high number of changed blocks that Veeam has to backup. Just a doubt, maybe over time these blocks are fragmented on the original volume and it takes more time to be read (even if in this case it would source bottleneck, the stats are telling you on average which one is the biggest bottleneck, not at a specific point in time...).

Finally, if you are not satisfied with support, you can always request directly from the support portal a manager escalation.

Luca.

PS: when you say version 6.x, which one you refer to? Not to trying to drop your requests, but V7 has several improvements related to reversed incremental, CPU and compression performances, and so on. Just checking why customer did not upgraded.