Backup of NAS, file shares, file servers and object storage.
Post Reply
adb98
Enthusiast
Posts: 63
Liked: 13 times
Joined: Jul 21, 2016 5:03 pm
Full Name: Aaron B
Contact:

File Share - 100% CPU

Post by adb98 »

I am doing a POC on this to see if this will be a good fit for us and on my test file share server when I run thet job, the CPU goes pegged out with Veeam Agent, I am also seeing Windows defender running at 40 to 50 percent at times.

Server 2019 - 5 small 10gb shares

Am I doing something wrong? Is this normal?

Bottleneck also shows proxy even though I build a VM on the same host as the test fileshare and gave it 4 CPU
HannesK
Product Manager
Posts: 14287
Liked: 2877 times
Joined: Sep 01, 2014 11:46 am
Full Name: Hannes Kasparick
Location: Austria
Contact:

Re: File Share - 100% CPU

Post by HannesK » 1 person likes this post

Hello,
yes, that's normal. Windows defender is a common issue and it reduces backup speed massively.

Best regards,
Hannes

PS: what is the reason that you use NAS backup instead of Veeam Agent for Windows to back up a Server 2019?
adb98
Enthusiast
Posts: 63
Liked: 13 times
Joined: Jul 21, 2016 5:03 pm
Full Name: Aaron B
Contact:

Re: File Share - 100% CPU

Post by adb98 »

To start.... This is a POC (proof of concept) and this is a VM not Physical. Here is the breakdown of our current environment

We currently have 6 file shares (5 virtual and 1 physical). Each one has about 4TB worth of data. We currently back the servers up every 3 hours to catch file changes. We use the normal backup method for VMs via backup job and Veeam Windows Agent for the physical.

This works but there are issues with this. First and foremost for 28 days of retention we have to have 248 restore points. We are using Data Domain for our repository so I have to sprinkle in a full every few days as it it would take a long long time to stitch everything back together if I had to restore a full server.

Another issue is that there is no file versioning and no single pane of glass for FLRs.

I am sure everyone reading this has gotten the dreaded user call... Can you restore a file for me... Sure what is the file name and when do you need it restored from.... Well its foo.txt but I am not quite sure when it went missing. Ugh.. 3 hours hits I have to drill down all the way to where their tree is and check...nope... next restore point..nope. It takes up a lot of our time. Luckily we do indexing and using the Enterprise management or this would be pure hell having to mount each image.

With NAS Backup it checks off a lot of these boxes. I only have to backup the VM and Physical once a day and we keep our long term retention for the servers like normal. We get fie versioning and hopefully a cleaner backup solution for NAS.

If there is a better way I am all for it. Been doing it this way for years and it works but its not efficient and clean. Sorry I know that was long but hopefully it put it in context.

Do we have new fresh instructions for Windows Defender settings for Veeam with NAS Backup. I know we have the normal exclusions but I didn't see any exclusions for the new NAS solution. Any tips or tricks.
HannesK
Product Manager
Posts: 14287
Liked: 2877 times
Joined: Sep 01, 2014 11:46 am
Full Name: Hannes Kasparick
Location: Austria
Contact:

Re: File Share - 100% CPU

Post by HannesK »

Hello,
ah, makes sense. I just wanted to make sure that you know the differences, because many people here in the forums miss the basics :-)
Do we have new fresh instructions for Windows Defender settings for Veeam with NAS Backup
I disabled real time protection of Windows defender. If the normal exclusions do not help, then that's the only idea that comes up to my mind.

Code: Select all

Set-MpPreference -DisableRealtimeMonitoring $true
Best regards,
Hannes
adb98
Enthusiast
Posts: 63
Liked: 13 times
Joined: Jul 21, 2016 5:03 pm
Full Name: Aaron B
Contact:

Re: File Share - 100% CPU

Post by adb98 »

So I have resolved the Windows Defender issue but I am still seeing massive hits on the CPU and also the network when the job runs and it all comes from the backup agent. I tried to tame the beast by changing the IO control all the way to the left and yet when this runs it almost brings the network and CPU to its knees. Most of these file servers are only running off a 1gb network connection so it using almost all of it makes this solution a no go.

Anyone run into this with Windows shares?
Gostev
Chief Product Officer
Posts: 31455
Liked: 6646 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: File Share - 100% CPU

Post by Gostev »

If network bandwidth usage is an issue, then you can always use our global network throttling rules to tone that down. Thanks!
adb98
Enthusiast
Posts: 63
Liked: 13 times
Joined: Jul 21, 2016 5:03 pm
Full Name: Aaron B
Contact:

Re: File Share - 100% CPU

Post by adb98 » 1 person likes this post

Agreed.... That is what I have done. This is like a grey hound that just wants to bust loose and you have to grab it by the scruff of its neck.

So the gotchas I ran into and the fixes I came up with - This is for a basic windows file share server (server 2019)

1. Windows Defender - Ensure you add all the exclusions that usually come with Veeam. I added the following via PowerShell for my File Server, Vcache Server, and the repository. Follow the link for ones needed for the main VBR server. I used add instead of set with PS as set will clear everything out and only use these which we don't want to do. If you have multiple servers it would be easy to just build a script.

https://www.veeam.com/kb1999

File Server

add-MpPreference -ExclusionPath "C:\Windows\VeeamVssSupport", "C:\Windows\VeeamLogShipper"
add-MpPreference -ExclusionProcess "VeeamAgent.exe","VeeamAgent64.exe"

Don't forget about your Vcache server/location for that server I added the following:

Vcache Server

add-MpPreference -ExclusionPath "E:\ProgramData\Veeam","E:\Vcache"
add-MpPreference -ExclusionProcess "VeeamAgent.exe","VeeamAgent64.exe"
add-MpPreference -ExclusionExtension ".vmdk", ".flat" ,"*vmdk","*.flat","*.vbm","*.vbk","*.vib","*.vrb","*.vsb","*.vlb","*.erm","*.vcache","*.vstore","*.vsource","*.vindex","*.vslice","*.vblob"

Then the normal exclusions you should already have set on your proxies

add-MpPreference -ExclusionProcess "VeeamAgent.exe","VeeamAgent64.exe"
add-MpPreference -ExclusionPath "C:\Program Files (x86)\Veeam","C:\Windows\Veeam","C:\programdata\Veeam","D:\ProgramData\Veeam\Backup\NfsDatastore"
add-MpPreference -ExclusionExtension ".vmdk", ".flat" ,"*vmdk","*.flat","*.vbm","*.vbk","*.vib","*.vrb","*.vsb","*.vlb","*.erm","*.vcache","*.vstore","*.vsource","*.vindex","*.vslice","*.vblob"
Also don't forget to add path exclusions for repositories you have on your proxies


2. Ensure you have at least 4 Vcpus on each file server at least - This very CPU intensive - especially the first pass -

3. Keep a close eye on the network utilization - I was getting 950Mbps and I only have a 1gb connection and its a VM so other VMs utilizing that nic will suffer and start dropping packets
The fix for this is to create a new network throttle and set it to a certain speed.

I am sure there are others I might be missing but those are the big ones. Hope this helps someone.
Post Reply

Who is online

Users browsing this forum: No registered users and 6 guests