Slowness at Beginning of VM Processing

[jlester]

I'm trying to track down a speed issue with our backup jobs. What we are seeing is a long pause at the point where it says "VM processing started at date/time" then "VM size". It is pausing several minutes right there before continuing. After the long pause, it goes past "Inventorying", "Preparing guest for hot backup", etc. pretty quickly. If a VM takes 15 minutes to backup, 7-8 minutes of that is taken up during that pause.

Any ideas what to check?

[jlester]

Case #00517593

In doing more research, when I look at the logs for the job, I see lots of messages during that time where it seems to be attempting to remove snapshots. It will fail saying the snapshot was already deleted or has not been completely created.

[Vitaliy S.]

Hi Jason,

What snapshots are you referring to? Do you mean your job tries to remove the VM snapshot?

Thanks!

[jlester]

I'm not sure what snapshots the logs are referring to. The VM's do not have any snapshots. I assume it is the process that was added a few versions ago to clean up old snapshots, but there are none to clean up. Here is what the logs shows:

Code: Select all

[14.02.2014 21:16:43] <25> Info     [Soap] Removing snapshot '23-snapshot-2568'
[14.02.2014 21:16:43] <25> Info     [VimApi] RemoveSnapshot, type "VirtualMachineSnapshot", ref "23-snapshot-2568", removeChildren "False"
[14.02.2014 21:16:43] <25> Info     [Soap] Execute failed('The object has already been deleted or has not been completely created'). Trying again in 3 seconds.

Those lines will repeat several times and then you see this:

Code: Select all

[14.02.2014 21:17:10] <25> Error    [VmSnapshotTracker] Failed to close snapshot id: '23-snapshot-2568' on Host: '###.###.###.###'
[14.02.2014 21:17:10] <25> Error    Failed to execute SOAP command "CRetrievePropertiesOperation". Details: "<ManagedObjectNotFoundFault xmlns="urn:vim25" xsi:type="ManagedObjectNotFound" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"><obj type="VirtualMachine">23</obj></ManagedObjectNotFoundFault>" (System.Exception)
[14.02.2014 21:17:10] <25> Error       at Veeam.Backup.ViSoap.CSoapService.Execute(IServiceOperation op)
[14.02.2014 21:17:10] <25> Error       at Veeam.Backup.ViSoap.CSoapService.RetrieveProperties(PropertyFilterSpec[] pfSpecs)
[14.02.2014 21:17:10] <25> Error       at Veeam.Backup.ViSoap.CSoapConnection.GetRootSnapshot(String vmRef)
[14.02.2014 21:17:10] <25> Error       at Veeam.Backup.Core.CVmSnapshotTrackerUtils.CloseSnapshotImpl(CVmSnapshotDbInfo snapshotInfo)
[14.02.2014 21:17:10] <25> Error       at Veeam.Backup.Core.CVmSnapshotTrackerUtils.CloseSnapshot(CVmSnapshotDbInfo snapshotInfo)
[14.02.2014 21:17:10] <25> Error    The object has already been deleted or has not been completely created (System.Web.Services.Protocols.SoapException)
[14.02.2014 21:17:10] <25> Error       at Veeam.Backup.ViSoap.CServiceSession.Execute(CServiceConnState connState, Action executeOp)
[14.02.2014 21:17:10] <25> Error       at Veeam.Backup.ViSoap.CSoapService.Execute(IServiceOperation op)

It will then repeat the first section, except the "23-snapshot-2568" will now be "23-snapshot-2569".

It does this multiple times before the job finally starts processing. It seems to be trying to clean up old snapshots that are not actually there, hence the errors.

Maybe a database issue?

[Vitaliy S.]

Could this be a similar issue? What does your support engineer think on this?

[jlester]

That sounds exactly like it. I have not heard back from support yet. We just have Standard and I didn't open the case until late on Friday.

I'm going to run some Select queries and see if it returns those old snapshots from the database.

Thanks,
Jason

[Vitaliy S.]

I would be careful with removing any records from the SQL configuration database manually. Let's wait till our engineer confirms that you experience the same issue.

[jlester]

That was definitely the problem. After clearing out that table, each VM is processing 6-7 minutes faster. Since some VMs were only taking about 10-12 minutes total, that is a significant savings.

[Vitaliy S.]

Glad that solution above helped you and thanks for coming back and updating the topic with your findings.

[oli4]

Hello all

We are using Veeam Backup V7 for backuping our vSphere infrastructure. We have physical proxies with Win 2008 (they see the SAN LUNs ready only) and virtual repository servers with linux.

Since 4 days the backup takes extremely long and because of the backup windows they time out in the morning. The transfer rate of the data is very low 3, 4, 5 MB/s instead of 70, 80 MB/s in the past.

When we start a backup job as a test we see, that he already has a long time for building the VM list. In the backup job logfile we see, that in the past he made hundreds of queries to the SAN in one second (for querying if the SAN is accessible), now he only make 3 or 4 per seconds.

We have restarted all the servers Center, proxy and repos, but no luck.

Also the SAN Admin stated, that there is no special activity on the SAN.

Where could the bottleneck be?

Regards
Oli4

[veremin]

When we start a backup job as a test we see, that he already has a long time for building the VM list.

Could you problem be similar to that referenced by Vitaliy? Thanks.

[oli4]

Hello Eremin
No.
It seems to be Antivirus. It think that MS changed something in Endpoint Protection and this slowed down everything. We have now AV disabled and backup seems processing with normal speed (We already have scan exclusions for .mdf and .ldf files).

Thanks
Regards
Oli4

[nefes]

I've had some problems with SCEP update from February 18 and overall performance. Next night update solved that problems, so it is worth trying to update it again.

[dellock6]

Or even better, disabling completely any antivirus control, at least on the files and folders involved in backup operations

Luca.

[oli4]

We have latest MEP Pattern (21.02.2014) and still slow.

[oli4]

Seems that there was a new antimalware engine released on 17.02.2014 and since then we have this problem:
http://blogs.technet.com/b/enginenotifi ... -2014.aspx

[dellock6]

Honetly, I had a hard time understanding the business requirements for having an antivirus on the Veeam backup server, or the repository. I see two main reasons for NOT having the antivirus here:
- the content of the VBK/VIB/VRB files is compressed and deduped, and its real content can be read only using VBR software itself. For the antivirus, those are totally unreadable files
- those files have the same content of the source VMs that are saved with the backup jobs. Once the VMs have an antivirus on-board and are cleaned, there is no way a virus can inject into a Veeam backup file

I can only advice to have the antivirus in the backup server if this is a corporate requirement, but exclude the Veeam directories from scanning engine.

Luca.

[t_hadberg]

Hi oli4

We are also seeing the same issues with ForeFront Endpoint protection slowing down all jobs since the definition update from February 17. If we disable the Forefront services everything goes back to normal.

The forefront Client Security definition file that started the whole mess is (Definition 1.165.4192.0), we are up at the newest def update which is (Definition 1.167.464.0) and it is still causing problems on the Veeam Backup Jobs.

Our Veeam installation is on the newest patch, which is 7.0.0.833 (patch 3)

Best regards
T_Hadberg

Has anyone created a case with Veeam and/or Microsoft on this issue.

[Vitaliy S.]

Hi Torben,

What do Veeam backup job logs show you? Did you have a chance to review them?

Thanks!

[oli4]

Hello Luca
of course all Veeam Backup files should be clean if a AV scanner runs on every VM. But the Veeam center server itself is also a Windows server and from my point of view every Windows server should have a AV scanner. We have excluded scaning of Veeam based files and .mdf and .ldf. This has worked until the newest malware engine from 17.02.2014.
Regards
Oli4

[oli4]

Hello T_Hadberg

this sounds exactly like the issue we have.
Have you applied the SQL Server Antimalware templates from Endpoint Protection? We will try this in a next step.

Regards
Oli4

[t_hadberg]

Hi oli4

No I have not applied the SQL Server Antimalware templates. If you do, please let me know if this will resolve the issues.

We are also seeing problems with a share on one of our file servers. If we are excluding the effected share that is causing problems the cpu load of the affected fileserver goes back to normal.

So this could certainly have something to do with certain file extensions and the Antimalware engine. The affected share holds our engineering tools (various CAD files)

Regards

T_Hadberg

[oli4]

Hi

I have now applied the SQL template for System Center Endpoint Protection and additionally add alle Veeam folders and *.mdf and *.ldf to the exclusion list for scanning, but the problem still exists.
I have now opened a case with Microsoft.

Regards
Oli4

[oli4]

Hello

For our environment the solution was to exclude also *.lof file scanning under:
C:\ProgramData\Veeam

Regards
Oliver

[foggy]

Oliver, do you probably mean *.log files?

[rhilmer]

All,

Between 2/17 and 2/18 we started experiencing slow backup times on all of our Veeam deployments. Prior to the 17th our jobs were very consistent with their run times and all of a sudden job times increased considerably across the board. We opened a case with Veeam but the logs were clean and the Veeam server configurations were fine as well. It appeared to be an issue with the source storage being slow as it was nickel and diming us across all VMs. I decided to go through the Windows logs again line by line comparing the environments to look for anything that may have happened. After digging through the system log I found that Microsoft released an updated engine version #1.1.10302.0 for Forefront/SCEP that was automatically installed right before the problem started. After doing some testing I verified that the new engine is the culprit and when disabling it or excluding all Veeam directories things run fine. We are going to play it safe and turn off real-time scanning and get a new full for safety. Microsoft now has a few cases open on the matter so if jobs started acting funny recently check your logs and verify your backups/restores. Hope this helps

Best regards
rhilmer

[Vitaliy S.]

Hello Richard,

Yes, it is seems like you're not the only one who has issues with Forefront update and backup job performance. Please review this thread for possible workarounds, might help.

Thank you!

[rhilmer]

Thanks. We have worked around our issues by two methods.

1. Exclude all Veeam directories and repositories
or
2. Turn off real time scanning and set up a scan of the system drives outside the backup window

We had some problems with test restores so we will be doing a new full without the AV on for the sake of safety.

[Vitaliy S.]

Thanks for sharing your workarounds.