Host-based backup of VMware vSphere VMs.
Post Reply
jlester
Enthusiast
Posts: 56
Liked: 5 times
Joined: Mar 23, 2010 1:59 pm
Full Name: Jason Lester
Contact:

Slowness at Beginning of VM Processing

Post by jlester »

I'm trying to track down a speed issue with our backup jobs. What we are seeing is a long pause at the point where it says "VM processing started at date/time" then "VM size". It is pausing several minutes right there before continuing. After the long pause, it goes past "Inventorying", "Preparing guest for hot backup", etc. pretty quickly. If a VM takes 15 minutes to backup, 7-8 minutes of that is taken up during that pause.

Any ideas what to check?
jlester
Enthusiast
Posts: 56
Liked: 5 times
Joined: Mar 23, 2010 1:59 pm
Full Name: Jason Lester
Contact:

Re: Slowness at Beginning of VM Processing

Post by jlester »

Case #00517593

In doing more research, when I look at the logs for the job, I see lots of messages during that time where it seems to be attempting to remove snapshots. It will fail saying the snapshot was already deleted or has not been completely created.
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Slowness at Beginning of VM Processing

Post by Vitaliy S. »

Hi Jason,

What snapshots are you referring to? Do you mean your job tries to remove the VM snapshot?

Thanks!
jlester
Enthusiast
Posts: 56
Liked: 5 times
Joined: Mar 23, 2010 1:59 pm
Full Name: Jason Lester
Contact:

Re: Slowness at Beginning of VM Processing

Post by jlester »

I'm not sure what snapshots the logs are referring to. The VM's do not have any snapshots. I assume it is the process that was added a few versions ago to clean up old snapshots, but there are none to clean up. Here is what the logs shows:

Code: Select all

[14.02.2014 21:16:43] <25> Info     [Soap] Removing snapshot '23-snapshot-2568'
[14.02.2014 21:16:43] <25> Info     [VimApi] RemoveSnapshot, type "VirtualMachineSnapshot", ref "23-snapshot-2568", removeChildren "False"
[14.02.2014 21:16:43] <25> Info     [Soap] Execute failed('The object has already been deleted or has not been completely created'). Trying again in 3 seconds.
Those lines will repeat several times and then you see this:

Code: Select all

[14.02.2014 21:17:10] <25> Error    [VmSnapshotTracker] Failed to close snapshot id: '23-snapshot-2568' on Host: '###.###.###.###'
[14.02.2014 21:17:10] <25> Error    Failed to execute SOAP command "CRetrievePropertiesOperation". Details: "<ManagedObjectNotFoundFault xmlns="urn:vim25" xsi:type="ManagedObjectNotFound" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"><obj type="VirtualMachine">23</obj></ManagedObjectNotFoundFault>" (System.Exception)
[14.02.2014 21:17:10] <25> Error       at Veeam.Backup.ViSoap.CSoapService.Execute(IServiceOperation op)
[14.02.2014 21:17:10] <25> Error       at Veeam.Backup.ViSoap.CSoapService.RetrieveProperties(PropertyFilterSpec[] pfSpecs)
[14.02.2014 21:17:10] <25> Error       at Veeam.Backup.ViSoap.CSoapConnection.GetRootSnapshot(String vmRef)
[14.02.2014 21:17:10] <25> Error       at Veeam.Backup.Core.CVmSnapshotTrackerUtils.CloseSnapshotImpl(CVmSnapshotDbInfo snapshotInfo)
[14.02.2014 21:17:10] <25> Error       at Veeam.Backup.Core.CVmSnapshotTrackerUtils.CloseSnapshot(CVmSnapshotDbInfo snapshotInfo)
[14.02.2014 21:17:10] <25> Error    The object has already been deleted or has not been completely created (System.Web.Services.Protocols.SoapException)
[14.02.2014 21:17:10] <25> Error       at Veeam.Backup.ViSoap.CServiceSession.Execute(CServiceConnState connState, Action executeOp)
[14.02.2014 21:17:10] <25> Error       at Veeam.Backup.ViSoap.CSoapService.Execute(IServiceOperation op)
It will then repeat the first section, except the "23-snapshot-2568" will now be "23-snapshot-2569".

It does this multiple times before the job finally starts processing. It seems to be trying to clean up old snapshots that are not actually there, hence the errors.

Maybe a database issue?
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Slowness at Beginning of VM Processing

Post by Vitaliy S. »

Could this be a similar issue? What does your support engineer think on this?
jlester
Enthusiast
Posts: 56
Liked: 5 times
Joined: Mar 23, 2010 1:59 pm
Full Name: Jason Lester
Contact:

Re: Slowness at Beginning of VM Processing

Post by jlester »

That sounds exactly like it. I have not heard back from support yet. We just have Standard and I didn't open the case until late on Friday.

I'm going to run some Select queries and see if it returns those old snapshots from the database.

Thanks,
Jason
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Slowness at Beginning of VM Processing

Post by Vitaliy S. »

I would be careful with removing any records from the SQL configuration database manually. Let's wait till our engineer confirms that you experience the same issue.
jlester
Enthusiast
Posts: 56
Liked: 5 times
Joined: Mar 23, 2010 1:59 pm
Full Name: Jason Lester
Contact:

Re: Slowness at Beginning of VM Processing

Post by jlester »

That was definitely the problem. After clearing out that table, each VM is processing 6-7 minutes faster. Since some VMs were only taking about 10-12 minutes total, that is a significant savings.
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Slowness at Beginning of VM Processing

Post by Vitaliy S. »

Glad that solution above helped you and thanks for coming back and updating the topic with your findings.
oli4
Enthusiast
Posts: 63
Liked: 3 times
Joined: Aug 14, 2012 11:34 am
Contact:

[MERGED] : Veeam Backup takes extremly long

Post by oli4 »

Hello all

We are using Veeam Backup V7 for backuping our vSphere infrastructure. We have physical proxies with Win 2008 (they see the SAN LUNs ready only) and virtual repository servers with linux.

Since 4 days the backup takes extremely long and because of the backup windows they time out in the morning. The transfer rate of the data is very low 3, 4, 5 MB/s instead of 70, 80 MB/s in the past.

When we start a backup job as a test we see, that he already has a long time for building the VM list. In the backup job logfile we see, that in the past he made hundreds of queries to the SAN in one second (for querying if the SAN is accessible), now he only make 3 or 4 per seconds.

We have restarted all the servers Center, proxy and repos, but no luck.

Also the SAN Admin stated, that there is no special activity on the SAN.

Where could the bottleneck be?

Regards
Oli4
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Slowness at Beginning of VM Processing

Post by veremin »

When we start a backup job as a test we see, that he already has a long time for building the VM list.
Could you problem be similar to that referenced by Vitaliy? Thanks.
oli4
Enthusiast
Posts: 63
Liked: 3 times
Joined: Aug 14, 2012 11:34 am
Contact:

Re: Slowness at Beginning of VM Processing

Post by oli4 »

Hello Eremin
No.
It seems to be Antivirus. It think that MS changed something in Endpoint Protection and this slowed down everything. We have now AV disabled and backup seems processing with normal speed (We already have scan exclusions for .mdf and .ldf files).

Thanks
Regards
Oli4
nefes
Veeam Software
Posts: 643
Liked: 162 times
Joined: Dec 10, 2012 8:44 am
Full Name: Nikita Efes
Contact:

Re: Slowness at Beginning of VM Processing

Post by nefes »

I've had some problems with SCEP update from February 18 and overall performance. Next night update solved that problems, so it is worth trying to update it again.
dellock6
Veeam Software
Posts: 6137
Liked: 1928 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: Slowness at Beginning of VM Processing

Post by dellock6 »

Or even better, disabling completely any antivirus control, at least on the files and folders involved in backup operations ;)

Luca.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
oli4
Enthusiast
Posts: 63
Liked: 3 times
Joined: Aug 14, 2012 11:34 am
Contact:

Re: Slowness at Beginning of VM Processing

Post by oli4 »

We have latest MEP Pattern (21.02.2014) and still slow.
oli4
Enthusiast
Posts: 63
Liked: 3 times
Joined: Aug 14, 2012 11:34 am
Contact:

Re: Slowness at Beginning of VM Processing

Post by oli4 »

Seems that there was a new antimalware engine released on 17.02.2014 and since then we have this problem:
http://blogs.technet.com/b/enginenotifi ... -2014.aspx
dellock6
Veeam Software
Posts: 6137
Liked: 1928 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: Slowness at Beginning of VM Processing

Post by dellock6 »

Honetly, I had a hard time understanding the business requirements for having an antivirus on the Veeam backup server, or the repository. I see two main reasons for NOT having the antivirus here:
- the content of the VBK/VIB/VRB files is compressed and deduped, and its real content can be read only using VBR software itself. For the antivirus, those are totally unreadable files
- those files have the same content of the source VMs that are saved with the backup jobs. Once the VMs have an antivirus on-board and are cleaned, there is no way a virus can inject into a Veeam backup file

I can only advice to have the antivirus in the backup server if this is a corporate requirement, but exclude the Veeam directories from scanning engine.

Luca.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
t_hadberg
Novice
Posts: 3
Liked: never
Joined: Jun 13, 2013 6:45 am
Full Name: Torben Dalton Hadberg
Contact:

Re: Slowness at Beginning of VM Processing

Post by t_hadberg »

Hi oli4

We are also seeing the same issues with ForeFront Endpoint protection slowing down all jobs since the definition update from February 17. If we disable the Forefront services everything goes back to normal.

The forefront Client Security definition file that started the whole mess is (Definition 1.165.4192.0), we are up at the newest def update which is (Definition 1.167.464.0) and it is still causing problems on the Veeam Backup Jobs.

Our Veeam installation is on the newest patch, which is 7.0.0.833 (patch 3)

Best regards
T_Hadberg

Has anyone created a case with Veeam and/or Microsoft on this issue.
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Slowness at Beginning of VM Processing

Post by Vitaliy S. »

Hi Torben,

What do Veeam backup job logs show you? Did you have a chance to review them?

Thanks!
oli4
Enthusiast
Posts: 63
Liked: 3 times
Joined: Aug 14, 2012 11:34 am
Contact:

Re: Slowness at Beginning of VM Processing

Post by oli4 »

Hello Luca
of course all Veeam Backup files should be clean if a AV scanner runs on every VM. But the Veeam center server itself is also a Windows server and from my point of view every Windows server should have a AV scanner. We have excluded scaning of Veeam based files and .mdf and .ldf. This has worked until the newest malware engine from 17.02.2014.
Regards
Oli4
oli4
Enthusiast
Posts: 63
Liked: 3 times
Joined: Aug 14, 2012 11:34 am
Contact:

Re: Slowness at Beginning of VM Processing

Post by oli4 »

Hello T_Hadberg

this sounds exactly like the issue we have.
Have you applied the SQL Server Antimalware templates from Endpoint Protection? We will try this in a next step.

Regards
Oli4
t_hadberg
Novice
Posts: 3
Liked: never
Joined: Jun 13, 2013 6:45 am
Full Name: Torben Dalton Hadberg
Contact:

Re: Slowness at Beginning of VM Processing

Post by t_hadberg »

Hi oli4

No I have not applied the SQL Server Antimalware templates. If you do, please let me know if this will resolve the issues.

We are also seeing problems with a share on one of our file servers. If we are excluding the effected share that is causing problems the cpu load of the affected fileserver goes back to normal.

So this could certainly have something to do with certain file extensions and the Antimalware engine. The affected share holds our engineering tools (various CAD files)

Regards

T_Hadberg
oli4
Enthusiast
Posts: 63
Liked: 3 times
Joined: Aug 14, 2012 11:34 am
Contact:

Re: Slowness at Beginning of VM Processing

Post by oli4 »

Hi

I have now applied the SQL template for System Center Endpoint Protection and additionally add alle Veeam folders and *.mdf and *.ldf to the exclusion list for scanning, but the problem still exists.
I have now opened a case with Microsoft.

Regards
Oli4
oli4
Enthusiast
Posts: 63
Liked: 3 times
Joined: Aug 14, 2012 11:34 am
Contact:

Re: Slowness at Beginning of VM Processing

Post by oli4 »

Hello

For our environment the solution was to exclude also *.lof file scanning under:
C:\ProgramData\Veeam

Regards
Oliver
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Slowness at Beginning of VM Processing

Post by foggy »

Oliver, do you probably mean *.log files?
rhilmer
Lurker
Posts: 2
Liked: never
Joined: Mar 05, 2014 3:23 pm
Full Name: Richard Hilmer
Contact:

[MERGED] Slow backups Veeam and MS Forefront/SCEP

Post by rhilmer »

All,

Between 2/17 and 2/18 we started experiencing slow backup times on all of our Veeam deployments. Prior to the 17th our jobs were very consistent with their run times and all of a sudden job times increased considerably across the board. We opened a case with Veeam but the logs were clean and the Veeam server configurations were fine as well. It appeared to be an issue with the source storage being slow as it was nickel and diming us across all VMs. I decided to go through the Windows logs again line by line comparing the environments to look for anything that may have happened. After digging through the system log I found that Microsoft released an updated engine version #1.1.10302.0 for Forefront/SCEP that was automatically installed right before the problem started. After doing some testing I verified that the new engine is the culprit and when disabling it or excluding all Veeam directories things run fine. We are going to play it safe and turn off real-time scanning and get a new full for safety. Microsoft now has a few cases open on the matter so if jobs started acting funny recently check your logs and verify your backups/restores. Hope this helps

Best regards
rhilmer
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Slowness at Beginning of VM Processing

Post by Vitaliy S. »

Hello Richard,

Yes, it is seems like you're not the only one who has issues with Forefront update and backup job performance. Please review this thread for possible workarounds, might help.

Thank you!
rhilmer
Lurker
Posts: 2
Liked: never
Joined: Mar 05, 2014 3:23 pm
Full Name: Richard Hilmer
Contact:

Re: Slowness at Beginning of VM Processing

Post by rhilmer »

Thanks. We have worked around our issues by two methods.

1. Exclude all Veeam directories and repositories
or
2. Turn off real time scanning and set up a scan of the system drives outside the backup window

We had some problems with test restores so we will be doing a new full without the AV on for the sake of safety.
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Slowness at Beginning of VM Processing

Post by Vitaliy S. »

Thanks for sharing your workarounds.
Post Reply

Who is online

Users browsing this forum: No registered users and 76 guests