v5 Backup Speed

[itherrkr]

I have been running Veeam for about a little over a week on version 4.1 as an eval. It has worked out great so far. Yesterday, I upgraded to v5, and this morning my backups are still running. They appear to be running at about half the speed of the original full backup. I created a new job that contained the same vm's and same number of vm's. What gives?

Kent

[Gostev]

Hi Kent. With Veeam, by default all backups are forever-incremental after the first run (which is why they were fast before upgrade). From symptoms, most likely you have full backup running. See what speed will you get tomorrow. Actual backup engine did not change between v4 and v5, so if you do not change any job settings, the performance will be exactly the same as in version 4.1.2. Thanks!

[itherrkr]

My original full ran about 12 hours, this one has surpassed that by about 4 hours and is still only about 65% done. We will see how it goes over the next several days. Thanks.

[lobo519]

I think I have noticed the same speed issues -

Specifically backing up over WAN. I cant get the processing rate over 1MB/s. Which is a problem I originally had and started using HyperIP to resolve. I am also using HyperIP to throttle the bandwidth to 10 mbps. While a new full job was running I kicked up the bandwith to 20 and processing rate went up to 4 MB/s. Then I kicked it back down to 10 again and the processing rate held at 4 MB/s.

Thinking about installing version 4 again - maybe I'm going crazy...

[tsightler]

Well, not knowing anything about your environment, if your limiting the bandwidth to 10Mbps, then you'll typically see around 1MB/sec since 10Mbps = 1MB/sec although compression might get you 1.49MB/sec while the display would still show 1MB/sec as Veeam simply chops the fractional amounts.

It's important to note that the "MB/sec" rate reported in the Veeam GUI is an estimate based on the number of GB's of the VMDK that have been processed, it is not a representation of transfer speed. The number is effectively "# of GB's of VMDK files processed/time". Time includes things like initial job startup, snapshot creation, initial job freeze, etc., etc. If a VMDK has large empty sections those are skipped, but that's still considered a processed data by Veeam so the number is inflated.

[Gostev]

Exactly... you will not see any changes in processing rate (at least immediately) because it does not show actual throughtput, but overall processing rate. It will start changing after some time after you have changed throttle settings. 20Mbps = 2.5MB/s max theoretical throughtput, so 4MB/s is very decent results for *VM processing* rate (remember, actual data copy throughtput is even higher).

[axs@moffatt.com]

I have noticed the backups taking longer as well. I went back through my logs and found that the same backup with 2 servers (one being our Exchange server) took 2 to 2.5 hours every evening for the last couple of months. I upgraded to 5 on Wednesday and both nights it has taken nearly 4 hours to complete the same exact backup. Nothing else has changed at all.
So if nothing changed.... What changed with the upgrade?

[Gostev]

What version did you upgrade from exactly?

[itherrkr]

Looking at my various reports, I can see that on 4.1.3 I was getting around an average 25 to 35 MB sec. Since the upgrade, i am only seeing average of around 15 to 20 MB sec. Something has definitely changed.

[Gostev]

We did not have version 4.1.3... if you mean 4.1.2, then there is absolutely no changes in data processing engine between 4.1.2 and 5.0... also, as you can see from other thread about backup speed, other customers are reporting over 130MB/s full backup speed, so engine itself clearly does not have any issues or bottlencecks. While something has definitely changed, did you consider it could be something in your environment? I am not completely excluding possibility of bug in our product of course, there always a possibility.

If you still have 4.1.2 logs (and you should have since they are getting archived and stay around for few days), please copy them to a separate location before they get deleted, and open a support case. We will be able to compare and see what is different between processing the same VM with v4 and v5.

[cyreli]

I'm just speaking about Full Backup.

Veaam 4.1 - Test Platform Dell 1950 - Dual CPU - Dual Core 2.6Ghz - Windows 2003 R2 32 bits
Mode: Network Backup
Average Speed over 22 VMs = 12MB/s

Veaam 5 - Test Platform Dell R710 - Dual CPU - Quad Core 2.8Ghz - Windows 2008 R2 64 bits
Mode: Network Backup
Average Speed over 22 VM (same as above of course) = 4MB/s

And I have run those test 6 times
I have also try a copy VM, and the results are the same.

My conclusion so far:
On a much faster platform my backup are 2x slower
I'll stick with V4, until the speed issue is resolved by the developers

[joergr]

Cyreli, could you do a little test for me? I just noticed you were with w2003 before and now on w2008. Maybe you hit a network glitch. Could you test this please:

open an admin dos box and check this on your b+r 2008 machine:

netsh
interface
tcp
set heuristics disabled
set global autotuninglevel=disabled

then reboot your b+r machine. do the speed test again. please report you results here.

best regards,
Joerg

[Gostev]

Veaam 4.1 - Windows 2003 R2 32 bits
Veaam 5 - Windows 2008 R2 64 bits

This is actually huge change right there for network backup, given all the network "optimizations" Microsoft had added in 2008 and 2008 R2

[ctchang]

Cyreli, could you do a little test for me? I just noticed you were with w2003 before and now on w2008. Maybe you hit a network glitch. Could you test this please:

open an admin dos box and check this on your b+r 2008 machine:

netsh
interface
tcp
set heuristics disabled
set global autotuninglevel=disabled

then reboot your b+r machine. do the speed test again. please report you results here.

best regards,
Joerg

I've set this set global autotuninglevel=disabled only as I saw it in another about improving SAN speed for W2K8 R2.

But what does "set heuristics disabled" do?

Thanks,
Jack

[joergr]

Hi,

Microsoft introduced (i think since vista kernel) a new mechanism to automatically and dynamically adjust the tcp window size in real time depending on the network load, infrastructure and latency. Where this is mostly a very good one it can become an extremely bad one under certain special circumstances (sometimes when doing extremely fast and normal transfers to the machine at the same time). Another glitch i often saw is this: Every saw a black screen when rdp´ed to that special machine? Turning autotuning off will most likely solve it.

It is worth a try for ANY scenario where you see network performance problems dealing especially when comparing some w2008r2 machine to w2003 machine This is because this mechanism was not introduced in w2003.

best regards,
Joerg

[ctchang]

Hi,

Microsoft introduced (i think since vista kernel) a new mechanism to automatically and dynamically adjust the tcp window size in real time depending on the network load, infrastructure and latency. Where this is mostly a very good one it can become an extremely bad one under certain special circumstances (sometimes when doing extremely fast and normal transfers to the machine at the same time). Another glitch i often saw is this: Every saw a black screen when rdp´ed to that special machine? Turning autotuning off will most likely solve it.

It is worth a try for ANY scenario where you see network performance problems dealing especially when comparing some w2008r2 machine to w2003 machine This is because this mechanism was not introduced in w2003.

best regards,
Joerg

Thanks for the explaination.

I understand what autotuninglevel=disabled does, but still what does set heuristics=disabled do?

[joergr]

Aah I understand.
With vista sp2 ms introduced another layer to the whole autotuning schematics, the general heuristics, which won´t let you touch some tuning settings unless you disable the heuristics. Interested in more? Google for it, you will find TONS of users who banged their heads against the wall because of this....

[cyreli]

This is actually huge change right there for network backup, given all the network "optimizations" Microsoft had added in 2008 and 2008 R2

I agree, with you, but I didn't expect "optimization" decreasing the backup speed by 50%

[cyreli]

I have applied

netsh
interface
tcp
set heuristics disabled
set global autotuninglevel=disabled

as adviced, unfortunately the backup speed is exactly the same only 5MB instead of 12MB

[tsightler]

I just want to say that both of those speeds you are reporting are pretty horrible. There almost has to be something different between the two setups. It would be more interesting for you to run the 4.1 tests on the exact same hardware to prove that there is nothing wrong with the network setup on the Windows 2008 server.

Also, you say you are using "Network" mode for 4.1, are you referring to the legacy "agent based" network mode? I think that with Veeam 5 "Network" mode actually uses vStorage API "NBD" mode. It's well known that NBD mode is very slow on ESX 4 hosts.

[cyreli]

I'm find with 12MB.
I agree that I should try Veeam 5, on the 2003 plaftorm,and Veaam 4 on the 2008, but for now I'm not able to download the 64bit version of V4.1, if any of you have it, let me know where I can find it FTP? Http ?

Thank you

[Gostev]

Our support should be able to provide you with the download link.

[tsightler]

I understand that you are fine with 12MB/sec, but my point was that it's indicative of something being "unusual" in your environment. I run Veeam on very underwhelming hardware (IBM x336), with older Equallogic storage (servers from 2005 or so, storage from 2006). The hardware is nothing special at all. My performance is generally in the 30-40MB/sec range, not 12MB/sec (actually reported numbers are regularly in the 50-60MB/sec range, but that includes things likes the skipped sections of disks).

If there is something amiss in your environment it's possible that new hardware is simply exasperating this issue. For example, if there's a networking issue causing dropped packets, new hardware can request packets at an even quicker rate which can cause a higher rate of dropped packets, which then leads to worse overall throughput.

I'm not saying that this is the case in your environment, but I'm running Veeam 5 in three pretty different environments and I've seen nothing but improvement in Veeam performance (primarily due to the lower I/O overhead of forward incremental backups as opposed to reverse incrementals). I ran new full backups on 40+ VM's yesterday (6.5TB's of VM's) and overall performance was virtually identical to the performance I've seen with Veeam backup 4 (this was the same backup server upgraded to Veeam 5 so it's the identical jobs). If your seeing drastic differences between the two versions then it feels like something else has to be different and we would need to figure out what those differences are.

[cyreli]

Ok some news:

as I mentioned previously have a Windows 2003 with Veeam 4.1, which works well with a backup speed at 12 MB/s using NBD, I have just upgraded the same box to Veeam 5, and as I expected the backup speed dropped to 5MB.

So for now, I'm still very disappointed with Veeam 5, and the culprit seems definitely to be this new version.

[Gostev]

If you send the logs from v4 and v5 jobs processing the same VMs to our support, we will be able to see what is the difference, and what exactly takes longer time. You should still have v4 logs available since you did in-place upgrade to v5. When you send the logs, please let me know the support case number you receive.

Remember that what you are seeing is not backup speed (or data moving throughtput), but processing rate (total VM size divided by total time to process). So, especially on small VMs, any difference in job settings may affect processing rate significantly with actual data moving throughtput not being affected. Anyhow, we will be able to see from the logs exactly how much time each operation took (freeze, snapshot, actual disk copy, etc).

[tsightler]

I'll be very interested in hearing the results as it's definitely not what I'm seeing in our environments.

[joergr]

me to i am very interested in the results, please keep us updated on the internal research here.

best regards, Joerg

[chrisb]

I have what I believe to be the same issue and have some findings that may be of interest. The environment comprises of:

VEEAM b&r server = Windows Server 2008 Storage Standard Edition SP2 (64bit) - 2 x 2.27Ghz Xeon CPU - 12 GB RAM
VEEAM v5

4 x ESX hosts:

2 x ESX 3.5.0 153875
2 x ESX 4.0.0 208167

I have 2 separate jobs - 1 backing up VMs on the ESX 3.5 hosts and 1 backing up VMs on the ESX 4.0 hosts. Both jobs were configured to use NBD with reversed incrementals, and ran at a maximum of 8MB/s. That is until I enabled legacy processing modes and configured the jobs to run as 'Network Backup' - Now the speeds are much improved:

ESX 3.5 = 40MB/s
ESX 4.0.0 = 18MB/s

I'm working on the assumption that the ESX 4.0.0 host is experiencing slower performance due to the known service console performance issue in this version of ESX.

Also note that these jobs are being written to an MSA2312fc so combined speeds are around the maximum that we would expect for backups to this device.

[Gostev]

Chris, your issue is not connected to what is discussed here... your issue is specific to using NBD (network) backup mode with ESX 4.0, VMware had addressed it in ESX 4.1... well known issue for over a year now, and is even covered in sticky FAQ thread. Thanks!

[chrisb]

Hi Gostev,
I am aware of the known issue with ESX 4.0 - My point was regarding performance before and after enabling legacy processing mode in VEEAM. Ingoring the ESX 4.0 host for a moment:

On ESX 3.5:

Job configured to NBD = 8MB/s
Legacy Processing ebaled and job configured to network mode = 40MB/s