File Share - 100% CPU

[adb98]

I am doing a POC on this to see if this will be a good fit for us and on my test file share server when I run thet job, the CPU goes pegged out with Veeam Agent, I am also seeing Windows defender running at 40 to 50 percent at times.

Server 2019 - 5 small 10gb shares

Am I doing something wrong? Is this normal?

Bottleneck also shows proxy even though I build a VM on the same host as the test fileshare and gave it 4 CPU

[HannesK]

Hello,
yes, that's normal. Windows defender is a common issue and it reduces backup speed massively.

Best regards,
Hannes

PS: what is the reason that you use NAS backup instead of Veeam Agent for Windows to back up a Server 2019?

[adb98]

To start.... This is a POC (proof of concept) and this is a VM not Physical. Here is the breakdown of our current environment

We currently have 6 file shares (5 virtual and 1 physical). Each one has about 4TB worth of data. We currently back the servers up every 3 hours to catch file changes. We use the normal backup method for VMs via backup job and Veeam Windows Agent for the physical.

This works but there are issues with this. First and foremost for 28 days of retention we have to have 248 restore points. We are using Data Domain for our repository so I have to sprinkle in a full every few days as it it would take a long long time to stitch everything back together if I had to restore a full server.

Another issue is that there is no file versioning and no single pane of glass for FLRs.

I am sure everyone reading this has gotten the dreaded user call... Can you restore a file for me... Sure what is the file name and when do you need it restored from.... Well its foo.txt but I am not quite sure when it went missing. Ugh.. 3 hours hits I have to drill down all the way to where their tree is and check...nope... next restore point..nope. It takes up a lot of our time. Luckily we do indexing and using the Enterprise management or this would be pure hell having to mount each image.

With NAS Backup it checks off a lot of these boxes. I only have to backup the VM and Physical once a day and we keep our long term retention for the servers like normal. We get fie versioning and hopefully a cleaner backup solution for NAS.

If there is a better way I am all for it. Been doing it this way for years and it works but its not efficient and clean. Sorry I know that was long but hopefully it put it in context.

Do we have new fresh instructions for Windows Defender settings for Veeam with NAS Backup. I know we have the normal exclusions but I didn't see any exclusions for the new NAS solution. Any tips or tricks.

[HannesK]

Hello,
ah, makes sense. I just wanted to make sure that you know the differences, because many people here in the forums miss the basics

Do we have new fresh instructions for Windows Defender settings for Veeam with NAS Backup

I disabled real time protection of Windows defender. If the normal exclusions do not help, then that's the only idea that comes up to my mind.

Code: Select all

Set-MpPreference -DisableRealtimeMonitoring $true

Best regards,
Hannes

[adb98]

So I have resolved the Windows Defender issue but I am still seeing massive hits on the CPU and also the network when the job runs and it all comes from the backup agent. I tried to tame the beast by changing the IO control all the way to the left and yet when this runs it almost brings the network and CPU to its knees. Most of these file servers are only running off a 1gb network connection so it using almost all of it makes this solution a no go.

Anyone run into this with Windows shares?

[Gostev]

If network bandwidth usage is an issue, then you can always use our global network throttling rules to tone that down. Thanks!

[adb98]

Agreed.... That is what I have done. This is like a grey hound that just wants to bust loose and you have to grab it by the scruff of its neck.

So the gotchas I ran into and the fixes I came up with - This is for a basic windows file share server (server 2019)

1. Windows Defender - Ensure you add all the exclusions that usually come with Veeam. I added the following via PowerShell for my File Server, Vcache Server, and the repository. Follow the link for ones needed for the main VBR server. I used add instead of set with PS as set will clear everything out and only use these which we don't want to do. If you have multiple servers it would be easy to just build a script.

https://www.veeam.com/kb1999

File Server

add-MpPreference -ExclusionPath "C:\Windows\VeeamVssSupport", "C:\Windows\VeeamLogShipper"
add-MpPreference -ExclusionProcess "VeeamAgent.exe","VeeamAgent64.exe"

Don't forget about your Vcache server/location for that server I added the following:

Vcache Server

add-MpPreference -ExclusionPath "E:\ProgramData\Veeam","E:\Vcache"
add-MpPreference -ExclusionProcess "VeeamAgent.exe","VeeamAgent64.exe"
add-MpPreference -ExclusionExtension ".vmdk", ".flat" ,"*vmdk","*.flat","*.vbm","*.vbk","*.vib","*.vrb","*.vsb","*.vlb","*.erm","*.vcache","*.vstore","*.vsource","*.vindex","*.vslice","*.vblob"

Then the normal exclusions you should already have set on your proxies

add-MpPreference -ExclusionProcess "VeeamAgent.exe","VeeamAgent64.exe"
add-MpPreference -ExclusionPath "C:\Program Files (x86)\Veeam","C:\Windows\Veeam","C:\programdata\Veeam","D:\ProgramData\Veeam\Backup\NfsDatastore"
add-MpPreference -ExclusionExtension ".vmdk", ".flat" ,"*vmdk","*.flat","*.vbm","*.vbk","*.vib","*.vrb","*.vsb","*.vlb","*.erm","*.vcache","*.vstore","*.vsource","*.vindex","*.vslice","*.vblob"
Also don't forget to add path exclusions for repositories you have on your proxies


2. Ensure you have at least 4 Vcpus on each file server at least - This very CPU intensive - especially the first pass -

3. Keep a close eye on the network utilization - I was getting 950Mbps and I only have a 1gb connection and its a VM so other VMs utilizing that nic will suffer and start dropping packets
The fix for this is to create a new network throttle and set it to a certain speed.

I am sure there are others I might be missing but those are the big ones. Hope this helps someone.