Availability for the Always-On Enterprise
Post Reply
sperkins
Influencer
Posts: 10
Liked: 1 time
Joined: Sep 22, 2011 9:38 am
Full Name: sebastian perkins
Contact:

Veeam 6.5 patch3 sudden slow down backup performance

Post by sperkins » Jan 20, 2014 5:51 pm

Dear All,

We are happily using veeam 6.5 patch3 with great satisfaction.

However since 1 / 2 weeks we have noticed that all of our backup jobs havec increased in time by 50%, even 100%. Veeam had been rebooted.

We have taken some steps investigating Storage, Backup performance, but all seems OK.

If I check the realtime stats of a job it seems that the job takes ages ( >10min) to perform a the hotadd element. Once that is done, the CBT of each disk is performed "as usual" ie fast :)

What can be going on ?


Thanks in advance,
Sebastian

Gostev
Veeam Software
Posts: 23215
Liked: 2970 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Veeam 6.5 patch3 sudden slow down backup performance

Post by Gostev » Jan 20, 2014 8:43 pm

Most likely, something broke in your VMware infrastructure, as hot add operation should not be taking this long under normal conditions. B&R does not really do much here - we just issue an API call asking VMware to hot add disks from the specified snapshot, and then are simply waiting until this is done. Thus, it might be worth investigating the logs on VMware side. Thanks!

dellock6
Veeam Software
Posts: 5548
Liked: 1538 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: Veeam 6.5 patch3 sudden slow down backup performance

Post by dellock6 » Jan 21, 2014 8:38 am

Just an assumption, maybe some new activity against the storage or in the ESXi servers that is creating higher I/O activity, thus slowing down all the hotadd operations?

Luca.
Luca Dell'Oca
EMEA Cloud Architect @ Veeam Software

@dellock6
http://www.virtualtothecore.com/en/
vExpert 2011-2012-2013-2014-2015-2016-2017-2018
Veeam VMCE #1

sperkins
Influencer
Posts: 10
Liked: 1 time
Joined: Sep 22, 2011 9:38 am
Full Name: sebastian perkins
Contact:

Re: Veeam 6.5 patch3 sudden slow down backup performance

Post by sperkins » Jan 21, 2014 8:45 am

Thats what we are investigating. However so far I/O seems ok... its a real head scratcher.

Could vcenter be creating an issue ?

dellock6
Veeam Software
Posts: 5548
Liked: 1538 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: Veeam 6.5 patch3 sudden slow down backup performance

Post by dellock6 » Jan 21, 2014 8:48 am

vCenter is indeed involved, but as Veeam only requests to vCenter to initiate an hotadd, also vCenter is not the one performing the hotadd, but it requires it to the ESXi server running that VM at a given time. Nonetheless, and overloaded vCenter could slow down any request that is going to pass to ESXi servers. Any indication of this into vCenter? If it's virtual, you can check on its own logs and performance monitor, or a quick move could be to increase its cpu and ram values, and see what happens.
Luca Dell'Oca
EMEA Cloud Architect @ Veeam Software

@dellock6
http://www.virtualtothecore.com/en/
vExpert 2011-2012-2013-2014-2015-2016-2017-2018
Veeam VMCE #1

sperkins
Influencer
Posts: 10
Liked: 1 time
Joined: Sep 22, 2011 9:38 am
Full Name: sebastian perkins
Contact:

Re: Veeam 6.5 patch3 sudden slow down backup performance

Post by sperkins » Jan 21, 2014 10:57 am

vcenter is snappy and has no issues (otherwise we would be having other issues with it).

This seems to be going on across mutliple storages and even on powered off servers.

foggy
Veeam Software
Posts: 17116
Liked: 1399 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Veeam 6.5 patch3 sudden slow down backup performance

Post by foggy » Jan 21, 2014 12:13 pm

As another test you could try to backup VMs via ESXi host directly and see whether hotadd operation still takes considerable time. This will allow to completely eliminate vCenter effect. Also, how about manually hotadding disks to a proxy VM?

sperkins
Influencer
Posts: 10
Liked: 1 time
Joined: Sep 22, 2011 9:38 am
Full Name: sebastian perkins
Contact:

Re: Veeam 6.5 patch3 sudden slow down backup performance

Post by sperkins » Jan 21, 2014 2:39 pm

So here are tests i did:
- Created a new job for one very small linux server (8gb)
- rebooted veeam (add 2GB of ram) and vcenter
- rescanned vcenter server in veeam (i cannot remove and readd it as it is used by jobs) and CIFS server (for storage)
- use only the local default proxy

1 tested this backup on a mix of 2 different storages (different racks) and 2 different servers (also in different racks)

Everytime the hotadd takes exactly 10m15sec
After that the bacup perf is "as usual" ie fast :)

I will try backupiong the server directory from the server...

sperkins
Influencer
Posts: 10
Liked: 1 time
Joined: Sep 22, 2011 9:38 am
Full Name: sebastian perkins
Contact:

Re: Veeam 6.5 patch3 sudden slow down backup performance

Post by sperkins » Jan 21, 2014 3:12 pm

So of course as vcenter is in the config I cannot add one of our servers as it is already in vcenter...

However 2 tests later :
- hotadd the test server's to a proxy server is instant and i can see it in the disk manager immedialety.
- our other datacenter uses vcenter too and does not suffer the hotadd latency.

foggy
Veeam Software
Posts: 17116
Liked: 1399 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Veeam 6.5 patch3 sudden slow down backup performance

Post by foggy » Jan 21, 2014 3:45 pm

sperkins wrote:So of course as vcenter is in the config I cannot add one of our servers as it is already in vcenter...
Actually, you can. You can have the same host added as standalone as well as through vCenter (using both DNS name and IP address).

sperkins
Influencer
Posts: 10
Liked: 1 time
Joined: Sep 22, 2011 9:38 am
Full Name: sebastian perkins
Contact:

Re: Veeam 6.5 patch3 sudden slow down backup performance

Post by sperkins » Jan 21, 2014 4:01 pm

Actually, you can. You can have the same host added as standalone as well as through vCenter (using both DNS name and IP address).
I add the standalone server... and i get

"VMware ESXi with name xxxxxx alreafy exists"

sperkins
Influencer
Posts: 10
Liked: 1 time
Joined: Sep 22, 2011 9:38 am
Full Name: sebastian perkins
Contact:

Re: Veeam 6.5 patch3 sudden slow down backup performance

Post by sperkins » Jan 21, 2014 4:17 pm

I cheated by adding the ip instead of the DNS

I added the VM, same issue.

This is really strange, that a distant veeam works fine !



Sebastian

foggy
Veeam Software
Posts: 17116
Liked: 1399 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Veeam 6.5 patch3 sudden slow down backup performance

Post by foggy » Jan 21, 2014 9:03 pm

Time to dig into VMware logs - they should show what is happening during these delays.

sperkins
Influencer
Posts: 10
Liked: 1 time
Joined: Sep 22, 2011 9:38 am
Full Name: sebastian perkins
Contact:

Re: Veeam 6.5 patch3 sudden slow down backup performance

Post by sperkins » Jan 24, 2014 3:13 pm 1 person likes this post

and digged i did !

After going through the logs, we were getting 10min timeouts in some strange snapshots.

I followed a similar issue in here
http://forums.veeam.com/vmware-vsphere- ... 15830.html

No backups, no snapshots check and I indeed found inaccurate entries in the table dbo.VmWareSnapshots about invalid snapshots (linked to perfectly valid servers)

Found a trick in the forums to get the VBM ids and correlate to names. The snapshots entries were old and obsolete.

deleted them and... boom ! all is stellar again.

Best,
Sebastian

foggy
Veeam Software
Posts: 17116
Liked: 1399 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Veeam 6.5 patch3 sudden slow down backup performance

Post by foggy » Jan 24, 2014 3:15 pm

Sebastian, glad you've nailed it. Thanks for sharing, much appreciated.

Post Reply

Who is online

Users browsing this forum: Exabot [Bot] and 25 guests