multiple vcenters and proxies in single VB&R server

[fast_cat]

Hi,

I have added three virtualcenter environments to a single VB&R server and as a result also have multiple VM proxies across both VCs. So I then have backup jobs that backup by datastores and use hotadd whenever possible. I am noticing that lots of the jobs appear to stall and it seems to be (according to the log files doing the following - [soap] Loading 'group-d1:Folder' heirarchy. It is doing this against a VC that is quite heavily loaded (over 2000 VMs) but the datastore that the job should be backing up is onlyavailable on the other VC....

Hope that makes sense ?

I guess my initital question is - Is it common practice to have multiple VCs in a single VB&R server ? If it is then I think I will raise a support case to try and understand what these jobs are doing

thanks

[dellock6]

Hi, my environment is around half of yours, and we use two vCenter, and only one Veeam Server with several virtual proxies.
From my experience, everytime I saw some [soap]-something error, at the end it was related to some undersizing of the Veeam server, especially memory if it also holds its own SQL database. Our Veeam server does not run proxy neither repository role, nonetheless it has 4 vcpu and 16 Gb of memory right now.

Luca.

[fast_cat]

ok thanks,
I've done some more research of the veeam backup job files and it seems I may have a lot of stale 'snapshot' entries. For example at around the time of 'pause' or 'hang' of the job there are lots of attempts to delete a snapshot such as -

Code: Select all

[05.03.2013 21:47:33] <01> Error    RemoveSnapshot failed, snapshotRef 'snapshot-636170', timeout '3600000' (System.Exception)
[05.03.2013 21:47:33] <01> Error       at Veeam.Backup.ViSoap.CSoapConnection.RemoveSnapshot(String snapshotRef, Int32 timeout)
[05.03.2013 21:47:33] <01> Error       at Veeam.Backup.Core.CVmSnapshotTrackerUtils.CloseSnapshotImpl(CVmSnapshotDbInfo snapshotInfo)
[05.03.2013 21:47:33] <01> Error       at Veeam.Backup.Core.CVmSnapshotTrackerUtils.CloseSnapshot(CVmSnapshotDbInfo snapshotInfo)
[05.03.2013 21:47:33] <01> Error    Failed to execute SOAP command "CRemoveSnapshotOperation". Details: "<ManagedObjectNotFoundFault xmlns="urn:vim25" xsi:type="ManagedObjectNotFound" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"><obj type="VirtualMachineSnapshot">snapshot-636170</obj></ManagedObjectNotFoundFault>" (System.Exception)
[05.03.2013 21:47:33] <01> Error       at Veeam.Backup.ViSoap.CSoapService.Execute(IServiceOperation op)
[05.03.2013 21:47:33] <01> Error       at Veeam.Backup.ViSoap.CSoapService.ExecuteAndWaitForCompletion(IServiceOperationAsync operation)
[05.03.2013 21:47:33] <01> Error       at Veeam.Backup.ViSoap.CSoapConnection.RemoveSnapshot(String snapshotRef, Int32 timeout)
[05.03.2013 21:47:33] <01> Error    The object has already been deleted or has not been completely created (System.Web.Services.Protocols.SoapException)

The thing that is confusing is that it seems these snapshot Ids are old and associated with VM-ids that should not be part of this backup job.

Do I have a number of stale Snapshot info somewhere that I need to clear out ?

regards

[dellock6]

You can run some tools like RVTools to find out any leftover Snapshots. On such a huge environment keeping track of all snapshots otherwise can become a real problem, and potentially lead to problems; first of all about performances, since many VMs are running under those snapshots.

Luca.

[Vitaliy S.]

Also you can go and install Veeam ONE to detect all orphaned snapshots and VMs with old, unneeded snapshots.

It is doing this against a VC that is quite heavily loaded (over 2000 VMs) but the datastore that the job should be backing up is onlyavailable on the other VC....

I believe this is expected behavior as Veeam B&R needs to load the entire hierarchy to find and detect VMs you're going to backup, though I agree that loading information from another vCenter Server should not be performed.

[fast_cat]

ok, looks like I have made some progress with why my Veeam jobs are now going very slowly.
within the Veeam database there is a table -dbo-VMwareSnapshots which contained in one instance up to 25 entries. When I queried Virtualcenter for the VM Id against each of these snapshots it was clear that these snapshots were old , stale entries. Veeam Reporter and RV-tools did not locate any of these snapshots against the VMs. This is why in most cases the error being returned from Veeam was "The object has already been deleted or has not been completely created"

For every VM in every Backup job it seems that Veeam needs to query each VirtualCenter environment to determine if these snapshots really exist. It doesnt seem to matter whether they are associated with any VMs within my backup job - they still get queried, and for each one a call is made to load 'group-d1:Folder' heirarchy (which as I said earlier can take up to 3 minutes in my heavily used Vcenter environment). So in my backup jobs which may have up to 30 VMs to backup this call seems to be made 30 times for every snapshot in the Veeam database -hence giving the impression of the jobs hanging.

So since my test environment had a number of these old snapshot entries in Veeam database I deleted them using SQL and the difference was incredible.

My main concern now though is it safe to just delete these 'stale' snapshots entries from the Veeam database - dbo-VMwaresnapshot table ?
( checked to make sure each one I removed did not correspond to a real snapshot)

regards

[dellock6]

Never took a look at that table, now you gave me another check to add to my "veeam health check"
Thanks for these informations, for sure in large enviroments these optimizations can save so much time. Do not know if it's safe to simply delete those entries, but anyway those snapshots are no more existents, so even if I'm sure the official reply from Veeam would be "is not supported, open a ticket instead" (and they are right to do so), well, if it worked you are good to go.
Maybe in some next release Veeam can add this check to clean old snapshots from that db table.

Luca.

[Vitaliy S.]

My main concern now though is it safe to just delete these 'stale' snapshots entries from the Veeam database - dbo-VMwaresnapshot table ?

We definitely do not recommend making any changes to our database manually.

Veeam Reporter and RV-tools did not locate any of these snapshots against the VMs.

For all old snapshots you should have an alarm fired in Veeam ONE Monitor. Btw, what report did you use to locate these snapshots?

[dellock6]

Uhm, I'm doubtful Veeam One can get different informations than RVtools, both query vCenter and scan the datastores as exposed by the ESXi servers. RVTools can show "zombie" disks and snapshots, if they are not there at all it could not be right a remnants inside Veeam database?

Luca.

[Vitaliy S.]

Veeam ONE Monitor queries vCenter Server MOB information to detect snapshots, while Veeam ONE Reporter scans actual datastores for orphaned files, so definitely either of approaches should work

[dellock6]

That's what I wrote too, but fast_cat was suggesting those snapshots are no more inside the datastores, but they are still registered inside Veeam database...

Luca.

[Vitaliy S.]

Oh, ok! I probably misread your posts.

[fast_cat]

Yes, you are correct, Vmware, vcenter, Veeam One, RV-Tools think these snapshots don't exist. But the Veeam BAckup database does.

I am not totally sure how things got to this stage. but it may relate to when we had some datastores filled up and had to manually tidy things up by deleting snapshots from VMs in Vcenter. This may have left these stale entries in Veeam Backup database.

There may be a number of scenarios where a Veeam snapshot can be removed from a VM (outside of Veeam) and these stale entries remain ???

Its easy to check if you are seeing the same scenario -
1. Ensure no backups are running
2. Ensure no Veeam snapshots exist on any VMs
3. Check the dbo-VMwaresnapshots table for any entries. If there are some then I think these are stale and will potentially slow down your backup jobs.
4.Run a test backup (any VM , any job) and check the backup job log file for the point where it does things with snapshots. Search for [VmSnapshotTracker] and see if you have any other entries saying the snapshot does not exist. I think it will do this check against each old snapshot it finds for each VM in the backup job.

regards

[deduplicat3d]

I noticed my backups / replication were stalling. After looking through the logs I came up with the same scenario and then ran into this thread. The stale entries in that table significantly affect the backup window. Please look into this issue. I use Veeam ONE... it seems like we should be able to leverage that technology in B&R.

[deduplicat3d]

I believe this is probably from overtime deleting snapshots manually on failed jobs (although I don't remember deleting quite so many)... hard to tell though because I've been using the product for so long. All over the forum I've seen instances saying it's OK to delete manually, but I'm not sure that's the case.

Hi Kevin, yes, feel free to consolidate this snapshot via vSphere Client manually. Thanks!

In path 3, I saw

• Deleting temporary VM snapshot manually instead of letting the job delete it results in vCenter connection duplication. As the result, vCenter Server may stop responding due to too many connections already opened with the following error: 503 Service Unavailable

Not sure if this is related or not... I'm still on path 1.

[foggy]

Not sure if this is related or not... I'm still on path 1.

Good reason to upgrade. Have you already contacted support with that?

[deduplicat3d]

Patch 3 didn't help. I'm submitting a case now.

[deduplicat3d]

Support told me to delete the rows. He said that he just had finished writing up a KB.