Veeam 6.5 full backup performance problems

[maverick964_uk]

We have veeam 6.5 update3.

This new environment uses the veeam server (virtual host) as the proxy as well.
2 months ago, full backups were approx 580GB and took 10 hours.

We now see the fulls at 780GB and taking 45-65hours.

Hard disk read speeds can vary from good in some VM's (200+MB/s) however most are between 1-7MB/s.

I'm really not sure why we are seeing this bad performance for alot of VM's. VM's are taking 8-10 hours each to backup but we have a few that backup fairly quickly.

What can we do to try and troubleshoot this...

thanks

[Vitaliy S.]

Mike,

Can you please post full bottleneck stats for your jobs? What VMs do have slow performance? Is it a SQL Server or any similar by any chance?

Thanks!

[maverick964_uk]

Hi, I'll grab that information tomorrow....

many thanks.

[maverick964_uk]

Vitaliy,

Our backup job is just one job that backs up the whole vcenter datacenter.
The job contains 4 x templates and 15vms so not that big.
Vm's are 1 x linux and 14 x windows 2008 r2

Full backup LOAD:
Source: 41%
Proxy: 18%
Network: 36%
Target: 71%

Processed: 1.8TB
Read: 1.7TB
Transferred: 760GB

Duration: 56:33:31
Processing rate: 9MB/s
Primary bottleneck: Target.

The backup target is a HP D2D via fileshares. I can copy files to the D2D at a rate of approx 90-100MB/s from the veeam server so I dont see this as the bottleneck.

So, fullbackups are the issue. Incrementals seem ok but they are small and use CBT.

Any help would be useful.
thanks!

[maverick964_uk]

also the slow VM's are not SQL server. One SQL server is a bit slow.....

The linux host is slow plus several windows VM's. WSUS server is always slow and some other application servers.
They are all in the same datastore as well as there is only one.

thanks

[vMO]

what backupo mode do you use? it looks like synthetic full if the slow backup is only with full backups.
if your backup target is a D2D, best practice is to use Active Full instead of Reverse incremental, this because D2D has some performacne problems if a Synthetic Full is created because of reading and writing blocks at the same time.. active full is only write to the target.
there is a whitepaper avilable how to configure veeam and hp D2D http://www.veeam.com/hp-protection-vmwa ... cture.html

[Gostev]

I can copy files to the D2D at a rate of approx 90-100MB/s from the veeam server so I dont see this as the bottleneck.

This is irrelevant, as Veeam backup is not just about copying a file from Veeam server to a share.

According to the bottleneck statistics, the problem is indeed the target storage performance. As a test, I would try backing up to some other storage with a test job (even locally to the backup server).

[Vitaliy S.]

Just a side note - if you upgrade to v7 and enable parallel disk and VM processing this will allow you to have better performance rates (saturate your target storage a bit more, since you have a headroom about 20%) during full job runs.

[maverick964_uk]

many thanks. We hope to move to v7 but not at the moment...

[maverick964_uk]

I think I've managed to solve this by setting the option "Local target" rather than "LAN target". From the whitepaper this means a larger block size is used.
Performance is alot better!!!

[Gostev]

Yes, the improvement is expected, because this means twice less blocks to process for the backup engine. The drawback here is slightly increased incremental backups size.

[maverick964_uk]

many thanks. I'm surprised no one mentioned about altering the storage optimization mode...

It would be nice to see more documentation about the performance benefits on the different optimization modes.

[foggy]

Mike, do you mean the current description is not sufficient?