-
- Expert
- Posts: 102
- Liked: 3 times
- Joined: May 09, 2013 8:57 am
- Full Name: Mike Lavery
- Contact:
Veeam 6.5 full backup performance problems
We have veeam 6.5 update3.
This new environment uses the veeam server (virtual host) as the proxy as well.
2 months ago, full backups were approx 580GB and took 10 hours.
We now see the fulls at 780GB and taking 45-65hours.
Hard disk read speeds can vary from good in some VM's (200+MB/s) however most are between 1-7MB/s.
I'm really not sure why we are seeing this bad performance for alot of VM's. VM's are taking 8-10 hours each to backup but we have a few that backup fairly quickly.
What can we do to try and troubleshoot this...
thanks
This new environment uses the veeam server (virtual host) as the proxy as well.
2 months ago, full backups were approx 580GB and took 10 hours.
We now see the fulls at 780GB and taking 45-65hours.
Hard disk read speeds can vary from good in some VM's (200+MB/s) however most are between 1-7MB/s.
I'm really not sure why we are seeing this bad performance for alot of VM's. VM's are taking 8-10 hours each to backup but we have a few that backup fairly quickly.
What can we do to try and troubleshoot this...
thanks
-
- VP, Product Management
- Posts: 27120
- Liked: 2721 times
- Joined: Mar 30, 2009 9:13 am
- Full Name: Vitaliy Safarov
- Contact:
Re: Veeam 6.5 full backup performance problems
Mike,
Can you please post full bottleneck stats for your jobs? What VMs do have slow performance? Is it a SQL Server or any similar by any chance?
Thanks!
Can you please post full bottleneck stats for your jobs? What VMs do have slow performance? Is it a SQL Server or any similar by any chance?
Thanks!
-
- Expert
- Posts: 102
- Liked: 3 times
- Joined: May 09, 2013 8:57 am
- Full Name: Mike Lavery
- Contact:
Re: Veeam 6.5 full backup performance problems
Hi, I'll grab that information tomorrow....
many thanks.
many thanks.
-
- Expert
- Posts: 102
- Liked: 3 times
- Joined: May 09, 2013 8:57 am
- Full Name: Mike Lavery
- Contact:
Re: Veeam 6.5 full backup performance problems
Vitaliy,
Our backup job is just one job that backs up the whole vcenter datacenter.
The job contains 4 x templates and 15vms so not that big.
Vm's are 1 x linux and 14 x windows 2008 r2
Full backup LOAD:
Source: 41%
Proxy: 18%
Network: 36%
Target: 71%
Processed: 1.8TB
Read: 1.7TB
Transferred: 760GB
Duration: 56:33:31
Processing rate: 9MB/s
Primary bottleneck: Target.
The backup target is a HP D2D via fileshares. I can copy files to the D2D at a rate of approx 90-100MB/s from the veeam server so I dont see this as the bottleneck.
So, fullbackups are the issue. Incrementals seem ok but they are small and use CBT.
Any help would be useful.
thanks!
Our backup job is just one job that backs up the whole vcenter datacenter.
The job contains 4 x templates and 15vms so not that big.
Vm's are 1 x linux and 14 x windows 2008 r2
Full backup LOAD:
Source: 41%
Proxy: 18%
Network: 36%
Target: 71%
Processed: 1.8TB
Read: 1.7TB
Transferred: 760GB
Duration: 56:33:31
Processing rate: 9MB/s
Primary bottleneck: Target.
The backup target is a HP D2D via fileshares. I can copy files to the D2D at a rate of approx 90-100MB/s from the veeam server so I dont see this as the bottleneck.
So, fullbackups are the issue. Incrementals seem ok but they are small and use CBT.
Any help would be useful.
thanks!
-
- Expert
- Posts: 102
- Liked: 3 times
- Joined: May 09, 2013 8:57 am
- Full Name: Mike Lavery
- Contact:
Re: Veeam 6.5 full backup performance problems
also the slow VM's are not SQL server. One SQL server is a bit slow.....
The linux host is slow plus several windows VM's. WSUS server is always slow and some other application servers.
They are all in the same datastore as well as there is only one.
thanks
The linux host is slow plus several windows VM's. WSUS server is always slow and some other application servers.
They are all in the same datastore as well as there is only one.
thanks
-
- Expert
- Posts: 176
- Liked: 21 times
- Joined: Jul 08, 2011 8:16 am
- Contact:
Re: Veeam 6.5 full backup performance problems
what backupo mode do you use? it looks like synthetic full if the slow backup is only with full backups.
if your backup target is a D2D, best practice is to use Active Full instead of Reverse incremental, this because D2D has some performacne problems if a Synthetic Full is created because of reading and writing blocks at the same time.. active full is only write to the target.
there is a whitepaper avilable how to configure veeam and hp D2D http://www.veeam.com/hp-protection-vmwa ... cture.html
if your backup target is a D2D, best practice is to use Active Full instead of Reverse incremental, this because D2D has some performacne problems if a Synthetic Full is created because of reading and writing blocks at the same time.. active full is only write to the target.
there is a whitepaper avilable how to configure veeam and hp D2D http://www.veeam.com/hp-protection-vmwa ... cture.html
-
- Chief Product Officer
- Posts: 31561
- Liked: 6724 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: Veeam 6.5 full backup performance problems
This is irrelevant, as Veeam backup is not just about copying a file from Veeam server to a share.maverick964_uk wrote:I can copy files to the D2D at a rate of approx 90-100MB/s from the veeam server so I dont see this as the bottleneck.
According to the bottleneck statistics, the problem is indeed the target storage performance. As a test, I would try backing up to some other storage with a test job (even locally to the backup server).
-
- VP, Product Management
- Posts: 27120
- Liked: 2721 times
- Joined: Mar 30, 2009 9:13 am
- Full Name: Vitaliy Safarov
- Contact:
Re: Veeam 6.5 full backup performance problems
Just a side note - if you upgrade to v7 and enable parallel disk and VM processing this will allow you to have better performance rates (saturate your target storage a bit more, since you have a headroom about 20%) during full job runs.
-
- Expert
- Posts: 102
- Liked: 3 times
- Joined: May 09, 2013 8:57 am
- Full Name: Mike Lavery
- Contact:
Re: Veeam 6.5 full backup performance problems
many thanks. We hope to move to v7 but not at the moment...
-
- Expert
- Posts: 102
- Liked: 3 times
- Joined: May 09, 2013 8:57 am
- Full Name: Mike Lavery
- Contact:
Re: Veeam 6.5 full backup performance problems
I think I've managed to solve this by setting the option "Local target" rather than "LAN target". From the whitepaper this means a larger block size is used.
Performance is alot better!!!
Performance is alot better!!!
-
- Chief Product Officer
- Posts: 31561
- Liked: 6724 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: Veeam 6.5 full backup performance problems
Yes, the improvement is expected, because this means twice less blocks to process for the backup engine. The drawback here is slightly increased incremental backups size.
-
- Expert
- Posts: 102
- Liked: 3 times
- Joined: May 09, 2013 8:57 am
- Full Name: Mike Lavery
- Contact:
Re: Veeam 6.5 full backup performance problems
many thanks. I'm surprised no one mentioned about altering the storage optimization mode...
It would be nice to see more documentation about the performance benefits on the different optimization modes.
It would be nice to see more documentation about the performance benefits on the different optimization modes.
-
- Veeam Software
- Posts: 21071
- Liked: 2115 times
- Joined: Jul 11, 2011 10:22 am
- Full Name: Alexander Fogelson
- Contact:
Re: Veeam 6.5 full backup performance problems
Mike, do you mean the current description is not sufficient?
Who is online
Users browsing this forum: No registered users and 156 guests