Surebackup A/V scan extremely slow

[ned]

I'm running Surebackup with VBR v10 (P1), which is running fine. I'm now trying to leverage the A/V scan option using Microsoft Windows Defender. This works but is extremely slow. The Windows Defender scan runs sequentially - one VM at a time and only one volume at a time. This is a major bottleneck and basically makes it unusable in my current environment.

Is anyone leveraging Surebackup with the A/V option? What A/V tool are you using? Any recommendations in general to optimize this process?

Thanks, Ned


[ID# 04165665] Surebackup A/V scan slow and sequential one VM at a time
Back to My Cases
Product: Veeam Backup & Replication
Severity: 3
Opened: May 8, 2020 22:54
VBR v10 (P1), VMware ESXi 6.7u3, Windows 2016, Windows Defender A/V scanner
SOBRs on EMC VNX5700 hybrid storage array

[HannesK]

Hello,
I could not find any information what you mean with "extremely slow" (5Mbyte/s, 100 MByte/s, whatever speed).

Did you test with parallel scan performance and did you see better values with parallel scanning?

There are several ways to test it
1) start multiple instant VM recoveries in parallel with secure restore
2) start multiple Windows file level recoveries in parallel and do manual parallel scans of c:\veeamflr\... (that's what I did)

I'm not sure whether parallel scanning really helps. in my system I have 100% CPU load with only one scan, but test system is not really large.

Best regards,
Hannes

[ned]

I will test with "parallel" scans using Instant secure restore.

The idea was to use Surebackup with A/V enabled, but regardless of your "process simultaneously up to X VMs" setting, the A/V option reduces it to one VM (and one volume).

I was hoping to hear from other customers using this option; their configs and experience.

Thanks, Ned

[PetrM]

Hi Ned,

While we're waiting for results of test proposed by Hannes, you may take a look at this thread because there are some interesting ideas and may be it's worth trying to run a quick scan?

As far as I understand from this article, "-1" should be passed as the value of "ScanType" argument to perform quick scan, just edit .xml file accordingly.

Thanks!

[ned]

I'm running 3 Instant secure restores now. One VM is scanning and the other two are in a "waiting for antivirus resource to become available: ..." state.

5/18/2020 5:11:46 PM [Windows Defender] Waiting for antivirus resource to become available: currently in use by "VM name" restore job...

[HannesK]

ah good point, so my idea with parallel instant restore was not good (at least not for Windows defender).

No further ideas from my side, but I'm still not convinced that parallel scanning would improve the overall speed.

[ned]

I am hoping to hear from other customers using this option; their configs and experience.

[michael.strine]

As far as I understand from this article, "-1" should be passed as the value of "ScanType" argument to perform quick scan, just edit .xml file accordingly.

I just want to point out, don't use ScanType -1 permanently in the Veeam AntivirusInfos.xml

From the link:
"ScanType are: 0 Default, according to your configuration, -1 Quick scan, -2 Full scan, -3 File and directory custom scan. "

Veeam uses the -3 option to scan only files in the backup (by feeding MpCmdRun.exe the path to the volume mounted in VeeamFLR)

-1 sets a quick scan for the entire system. The secure restore will be scanning the mount server; and likely not the backups, if ScanType -1 is used.

[r2d2]

I'm running Surebackup with VBR v10 (P1), which is running fine. I'm now trying to leverage the A/V scan option using Microsoft Windows Defender. This works but is extremely slow. The Windows Defender scan runs sequentially - one VM at a time and only one volume at a time. This is a major bottleneck and basically makes it unusable in my current environment.

Is anyone leveraging Surebackup with the A/V option? What A/V tool are you using? Any recommendations in general to optimize this process?

Thanks, Ned


[ID# 04165665] Surebackup A/V scan slow and sequential one VM at a time
Back to My Cases
Product: Veeam Backup & Replication
Severity: 3
Opened: May 8, 2020 22:54
VBR v10 (P1), VMware ESXi 6.7u3, Windows 2016, Windows Defender A/V scanner
SOBRs on EMC VNX5700 hybrid storage array

Sorry for letting you down. This functionality depends on command lines of antiviruses. Unfortunately it's an infrastructural limitation and couldn't be resolved by Veeam's programming code.

[ned]

How's it an infrastructure limitation? Windows Defender (command line) can run multiple instances.

[imnotarobot]

Yes, the first VM I am scanning now is up to 2 hours and counting. The limitations described make it very slow.

[Dima P.]

Hello Jason,

Can you please clarify the actual processing rate, size of the backup (i.e. full backup size vs raw vm size), the type of the backup repository where backup files are located and how virtual lab is connected with the repository? Thank you in advance!

[cgsm]

Hello all,

I also experience this same issue with SureBackup's virus scan feature. I am running SureBackup with virus scanning on about 2TB of VM data, roughly 20 VMs. The job rarely completes because it takes so long to scan the VMs that the SureBackup job overruns into a backup job that cancels the SureBackup. I have a relatively small amount of data so I can only imagine that larger Veeam customer's simply aren't using the virus scanning feature.

I am running SureBackup set to process 6 VMs at a time. According to Task Manager, it does look like 6VMs are scanned for viruses at the same time since I have 6C/12T pegged at near 100% during the virus scanning process. The VBR server itself reports anywhere from 50%-90% usage so at some point the other cores are getting involved somehow. My VBR server is 12C/24T for reference.

I think the "slowness" is in comparison to running a virus check within a VM itself. For example, if I run a Windows Defender full system scan within one of my VMs it takes a substantially shorter amount of time. However, that VM can then use all "its" cores for the scan, versus Veeam using only one core. I.e.: if my VM has 4 cores, it will use all 4 cores for Windows Defender; when Veeam scans a VM disk, only one core is used.

My advice: set SureBackup to process more VMs at once. You get some better parallelization of the scanning which reduces overall runtime.

I don't necessarily care how long the virus scanning process takes, except that the SureBackup job then gets cancelled because a backup job runs that needs access to the backup files. Still trying to figure out a way around this, no luck yet.

[cgsm]

An update...

I noticed that total CPU utilization for the VBR drops to near 0% when a VM is done scanning for viruses and changes to the "publishing" step. If multiple VMs were being scanned for viruses in parallel, I would expect CPU usage to drop only 1/6th (if SureBackup is set to process 6 VMs at a time). This does not seem to be the case. Once a VM is completely handled and a new VM starts to be processed, the total VBR server CPU usage goes back up (pegging the number of cores equal to the number of VMs being processed at one time).

[Dima P.]

Hi cgsm,

Did I get it right: Veeam B&R is set as a mount server for the repository with the backup files in the question? Thanks!

[cgsm]

Yes, the VBR server is set as the mount server for the repo the backup files are located on. vPower NFS is enabled.

The repo where the backup files are stored is a SOBR using direct-attached SSDs for performance tier and Azure for capacity tier.

[cgsm]

Here is a few oddities:

I have two VMs, both domain controllers that are more-or-less identical. The first VM was processed at the start of the SureBackup job and the "canning for viruses" step took "00:28:47". The second VM was processed sometime in the middle of the of the SureBackup job and the "scanning for viruses" step has taken "02:30:00" and has still not completed. VMs are the same size, 60GB. They should take the same amount of time to scan.

The CPU utilization of the VBR server is also not consistent. Currently it is scanning 6 VMs and the MsMpEng process is only using ~15% CPU. None of the cores are pegged like when the scanning first started (first starts processing a new VM). I would expect the cores to be at 100% during the entirety of the scan. Disk usage is also very low so I am not exactly sure what the MsMpEng scan is doing... VeeamAgent.exe is reading at ~2MB/s and is the largest disk user.

[Dima P.]

cgsm,

Thank you for sharing the updates. Veeam B&R just mounts the backup and triggers the scan command on the AV installed at the repository mount server, it does not interfere, control or somehow tune the way how AV scans the data. Let me check with QA team is there are any additional parameters we could pass to the AV via configuration xml.

[cgsm]

Some more information:

Per my referenced note above talking about the scanning of two nearly identical VMs: The first took 00:28:47 to complete the virus scan, the second took 10:28:20. Clearly something is amiss here.

The overall SureBackup job is still running; one VM has been scanning for viruses for 20+ hours. If I scan this VM with built-in virus scanning it completes far quicker (couple of hours at most).

[cgsm]

I reconfigured my SureBackup job to process only 1 VM at a time. Doing so does cause the virus scanning to handle only 1 VM at a time, as expected, and the time to scan all of my VMs is very long, again as expected. CPU usage doesn't show any one VM as pegged and overall usage bounces around the low teens. I would have expected to see one CPU pegged since prior testing with 6 VMs scanned at once showed 6 CPUs pegged (although the usage dropped into the teens after a while).

This being said, it does look like multiple VMs are scanned in parallel. However, this brings up two questions:
(1) Why is the scan time for two nearly identical, and same drive size, so different (wee comment above, one VM takes 28 min. while the other took 2.5 hours).
(2) Why is the scan so slow? Scanning via Veeam takes multiple times longer compared to scanning from within the VM. I think this has to do with CPUs being used/assigned. When scanning a VM from within a VM, all the CPUs are used. When scanning from Veeam, I initially though only one CPU was used per VM for the scan although this relationship degraded when I tried to process only one VM at a time.

[cgsm]

Something else I have noticed... When you process multiple VMs, even though the SureBackup job shows multiple "Scanning for viruses" lines in the session log and the "Running" job list shows multiple "FLR_[vm-name]" items, it seems as if only one VM is being processed at a time. If you look at the VeeamFLR directory where the VMs get mounted (mine is located at C:\VeeamFLR), only one VM is shown at a time. Furthermore, if you double-click on the "FLR_[vm-name]" items in the "Running" job list, only one VM shows the step "Mounting restore point" with a green "completed" check. All other VMs only show "File level restore started". Looking in Windows Resource Monitor, the disk usage does only show one VM's files being read.

At this point, I can nearly 100% state that only one VM is scanned for viruses at a time.

[cgsm]

This may be tied into [1] although I am not 100% sure. In short, SureBackup validates VM backup files sequentially, not in parallel regardless of the "process x many VMs at once" setting. This may affect malware scanning as well (and does based on my testing).

[1]veeam-backup-replication-f2/multi-threa ... 79282.html

[cgsm]

I have submitted case # 05364809 for this.

[Dima P.]

cgsm,

Thank you for the case and sorry for keeping silence! I've check with QA team an it looks like there is no possibility to run a parallel scan via Windows Defender for mounted volumes. We will review the case details once again with the team, to make sure we are not missing anything.

[cgsm]

How are large-scale Veeam users performing malware scanning since scanning sequentially takes sooooooo long? I am only scanning a couple dozen VMs with ~2TB of data at a time and it takes so long the SureBackup job ends up getting cancelled since other backup jobs need access to the files.

Are large scale users simply not running the malware check part of SureBackup?
Are large scale users creating individual SureBackup jobs per VM and scheduling the jobs so they don't overlap?

If I were to scan 10TB+ of VMs it would take many days to complete the malware check to complete.

[HannesK]

I would scan in production where I have much better IO performance. Backup storage is usually much cheaper and there are additional layers involved (backup software compression plus virtual file system to present backups to the antivirus software).

The costs for fast backup storage usually prevent scanning in the backups.

[cgsm]

@HannesK I do not understand what you are saying, please clarify. Are you suggesting scanning the VMs from within the VMs? Of course I do this already.

I am not resource constrained during the SureBackup malware check. My backup infrastructure is FAST (it is some older production hardware we replaced [12-core, 24-thread, 96GB of RAM, RAID 1 SSDs for OS and RAID10 SSDs for backups]). The slowness appears to solely be the sequential nature of how the malware check is performed.

Based on your reply, and my understand to "scan VMs within each VM", it seems like the SureBackup malware check is pretty useless. I don't really understand it's purpose.

[Gostev]

While Veeam can't control how your antivirus chooses to scan files within a VM (sequentially one by one or in parallel), our SureBackup jobs are certainly capable of processing multiple VMs at once.

As for scan performance, in my tests SureBackup malware scan was only about 2-3 times slower than the same scan in production environment. So I don't see how such a difference makes it "useless". It's always nice to get some load off of the production environment and move it to a storage device that does nothing most of the time outside of the backup window.

[cgsm]

@Gostev Yes, SureBackup can process multiple VMs at once, but only certain steps. For example, starting up and heartbeat testing is done in parallel, but backup file validation is sequential (determined by support and in testing). I don't think you can broadly say SureBackup can process multiple VMs at once.

I would love a 2-3 times slower malware scan! In my tests, I can scan my production (live, running) VMs (from within the VMs) 10x+ faster than with SureBackup. I can scan my production VMs easily during the work day while SureBackup can start on Friday evening and not complete by Sunday evening when the job is cancelled.

I believe the issue is, as Dimitry notes, that Windows Defender (on the VBR server) can only scan one mounted volume (VM) at t time.

[Gostev]

Then it probably make sense to investigate with support as 10x difference in scan time is unexpected assuming fast backup storage.