Veeam 6.5 patch3 sudden slow down backup performance

[sperkins]

Dear All,

We are happily using veeam 6.5 patch3 with great satisfaction.

However since 1 / 2 weeks we have noticed that all of our backup jobs havec increased in time by 50%, even 100%. Veeam had been rebooted.

We have taken some steps investigating Storage, Backup performance, but all seems OK.

If I check the realtime stats of a job it seems that the job takes ages ( >10min) to perform a the hotadd element. Once that is done, the CBT of each disk is performed "as usual" ie fast

What can be going on ?


Thanks in advance,
Sebastian

[Gostev]

Most likely, something broke in your VMware infrastructure, as hot add operation should not be taking this long under normal conditions. B&R does not really do much here - we just issue an API call asking VMware to hot add disks from the specified snapshot, and then are simply waiting until this is done. Thus, it might be worth investigating the logs on VMware side. Thanks!

[dellock6]

Just an assumption, maybe some new activity against the storage or in the ESXi servers that is creating higher I/O activity, thus slowing down all the hotadd operations?

Luca.

[sperkins]

Thats what we are investigating. However so far I/O seems ok... its a real head scratcher.

Could vcenter be creating an issue ?

[dellock6]

vCenter is indeed involved, but as Veeam only requests to vCenter to initiate an hotadd, also vCenter is not the one performing the hotadd, but it requires it to the ESXi server running that VM at a given time. Nonetheless, and overloaded vCenter could slow down any request that is going to pass to ESXi servers. Any indication of this into vCenter? If it's virtual, you can check on its own logs and performance monitor, or a quick move could be to increase its cpu and ram values, and see what happens.

[sperkins]

vcenter is snappy and has no issues (otherwise we would be having other issues with it).

This seems to be going on across mutliple storages and even on powered off servers.

[foggy]

As another test you could try to backup VMs via ESXi host directly and see whether hotadd operation still takes considerable time. This will allow to completely eliminate vCenter effect. Also, how about manually hotadding disks to a proxy VM?

[sperkins]

So here are tests i did:
- Created a new job for one very small linux server (8gb)
- rebooted veeam (add 2GB of ram) and vcenter
- rescanned vcenter server in veeam (i cannot remove and readd it as it is used by jobs) and CIFS server (for storage)
- use only the local default proxy

1 tested this backup on a mix of 2 different storages (different racks) and 2 different servers (also in different racks)

Everytime the hotadd takes exactly 10m15sec
After that the bacup perf is "as usual" ie fast

I will try backupiong the server directory from the server...

[sperkins]

So of course as vcenter is in the config I cannot add one of our servers as it is already in vcenter...

However 2 tests later :
- hotadd the test server's to a proxy server is instant and i can see it in the disk manager immedialety.
- our other datacenter uses vcenter too and does not suffer the hotadd latency.

[foggy]

So of course as vcenter is in the config I cannot add one of our servers as it is already in vcenter...

Actually, you can. You can have the same host added as standalone as well as through vCenter (using both DNS name and IP address).

[sperkins]

Actually, you can. You can have the same host added as standalone as well as through vCenter (using both DNS name and IP address).

I add the standalone server... and i get

"VMware ESXi with name xxxxxx alreafy exists"

[sperkins]

I cheated by adding the ip instead of the DNS

I added the VM, same issue.

This is really strange, that a distant veeam works fine !



Sebastian

[foggy]

Time to dig into VMware logs - they should show what is happening during these delays.

[sperkins]

and digged i did !

After going through the logs, we were getting 10min timeouts in some strange snapshots.

I followed a similar issue in here
http://forums.veeam.com/vmware-vsphere- ... 15830.html

No backups, no snapshots check and I indeed found inaccurate entries in the table dbo.VmWareSnapshots about invalid snapshots (linked to perfectly valid servers)

Found a trick in the forums to get the VBM ids and correlate to names. The snapshots entries were old and obsolete.

deleted them and... boom ! all is stellar again.

Best,
Sebastian

[foggy]

Sebastian, glad you've nailed it. Thanks for sharing, much appreciated.