Backup Performance to VEEAM repository is painfully slow

[rpetrov1981]

Hi!
When I perform an Endpoint backup of my pc to the shared folder my computer works fine. But if I switch the destination to Veeam BAckup & Replication repository computer's performance decreases dramatically. The shared folder and Repository are the same disks on the same VEEAM Backup physical server. I would like to use repository as a destination just to have a possibility to manage all of my Endpoint backups through a single console. I have tried this task on different computers but the result is the same.
I suppose that problem is connected with network, because when I plug my computers via 1 Gbit network instead of 100Mbit the performance is fine. Has anybody faced with a similar problem? Any help would be greatly appreciated.
P.S. VEEAM Support has been looking for a solution but no luck so far.

[Mike Resseler]

Hey,

Since you are already are working with support could you post the support ID here and follow up when you have more information from our team? Dima P. will certainly look further into this

Thanks

Mike

[rpetrov1981]

Sure, Mike! Veeam Support - Case # 00925886. Thank you for your cooperation!

[Dima P.]

Hello rpetrov1981,
Thanks for the case ID. We will try to reproduce and investigate this behavior once we got any available QA resources.

[hoFFy]

Does the overall performance of the clients decrease, or just increases the time needed for backup?

[rpetrov1981]

Does the overall performance of the clients decrease, or just increases the time needed for backup?

Hi! Overall performance decreases dramatically. I am not sure about time amount needed for backup yet I do not concern seriously about it.

[hoFFy]

Okay, I've just been asking, because I set up a VBR with different NICs, some of them not reachable from the client's network. Therefore every client has several timeouts during backup, because there is no possibility to select the NIC / IP, which clients should use, if different are available. This increases backup times by several minutes.

Does the taskmanager, or ressource monitor help you?
Maybe you disabled the setting "throtttle nackup activity..." and Endpoint is able to use the network connection way more efficient than connecting via CIFS and therefore backup via VBR to the same server slows down your clients?!

[rpetrov1981]

Okay, I've just been asking, because I set up a VBR with different NICs, some of them not reachable from the client's network. Therefore every client has several timeouts during backup, because there is no possibility to select the NIC / IP, which clients should use, if different are available. This increases backup times by several minutes.

Does the taskmanager, or ressource monitor help you?
Maybe you disabled the setting "throtttle nackup activity..." and Endpoint is able to use the network connection way more efficient than connecting via CIFS and therefore backup via VBR to the same server slows down your clients?!

Well, I suppose my case is different. In general, backup time is similar whether I backup to CIFS or VEEAM Repo. I have been investigating the case via resource monitor and find some fun tips. When it performs well, network is utilized by process SYSTEM.Otherwise, network is utilized by process VEEAMAGENT.

Also I tried to switch on/off "throttle..." checkbox, result is the same

[hoFFy]

Sounds weird.
When using VBR as target, Endpoint is using the vveam.endpoint.agent, thats correct. It also uses another port. Maybe your AV-software is investigating this traffic more intensive than backing up to CIFS-share via port 445?!
Did you try to disable all other programs / services during backup? Especially the av-software

[rpetrov1981]

You may take a look on this:
http://tinypic.com/r/25zo5ev/8
We use MS System Center Endpoint Protection. I've just disabled real-time protection and try to start backup session - unfortunately the same

[Dima P.]

Hello guys,
Just want to check if the network throttling in the VBR for Endpoint machines is not enabled?

[hoFFy]

You are connected via a slow 100MBit link, does performance improve on clients with GBit NICs?
Are your local installed applications slow, or is it only slow when you work at the same time with software, which also uses the network? For example software started from a network share, or when opening larger files from network?
Maybe that the traffic via 445 has lower priority when backing up to VBR and therefore it seems to be slow. On the other hand, when backing up to a CIFS share the data streams are scheduled by Windows

[rpetrov1981]

Hello guys,
Just want to check if the network throttling in the VBR for Endpoint machines is not enabled?

Hello Dima!
I have tried either throttling enabled or not - no difference

[Dima P.]

Network Traffic Throttling in VBR, right? (Not backup throttling in the Endpoint)

[hoFFy]

Network Traffic Throttling in VBR, right? (Not backup throttling in the Endpoint)

What should be the difference? The client is connected via 100MBit and Endpoint sends data with ~90MB/sec.

[rpetrov1981]

You are connected via a slow 100MBit link, does performance improve on clients with GBit NICs?
Are your local installed applications slow, or is it only slow when you work at the same time with software, which also uses the network? For example software started from a network share, or when opening larger files from network?
Maybe that the traffic via 445 has lower priority when backing up to VBR and therefore it seems to be slow. On the other hand, when backing up to a CIFS share the data streams are scheduled by Windows

If i am connected via Gbit it works like a charm whether I backup to CIFS or VEEAM Repo. But it is not an option for all my clients
My local applications work well. For example I can work in Excel with large tables, listen music etc. yet all network connected activities fail. You may see what happens even with ping during backup session
http://tinypic.com/r/34g0vmo/8

[rpetrov1981]

Network Traffic Throttling in VBR, right? (Not backup throttling in the Endpoint)

Dima, I've got your idea. I have no rules for Network traffic Throttling in VBR

[hoFFy]

You should be able to enable network throttling in VBR under Backup Infrastructure -> Backup Proxies -> the proxy being used, or in the repository settings "Limit combined data rate to". You could try to limit data rate to about 60 - 70MB/s and retry backup to VBR.
How does the system behave?

[hoFFy]

Be aware that setting throttling via the proxy might slow down other server backups if they are using this proxy, too.
(BTW... I don't know if these settings apply if using VBR together with Endpoint)... so.. try throttling through the repository settings

[Dima P.]

The suggested solution might do the trick as Endpoint should leverage the said repository settings

[rpetrov1981]

Be aware that setting throttling via the proxy might slow down other server backups if they are using this proxy, too.
(BTW... I don't know if these settings apply if using VBR together with Endpoint)... so.. try throttling through the repository settings

Man, you've just made my day! I've just configured throttling with my IP only as a source and VBR IP as a target and it works perfect! Thank you very much! Probably this workaround should be included to the FAQ.

[hoFFy]

Thank you!

[Dima P.]

Well done guys!

[rpetrov1981]

Thank you for your cooperation! I will update the ticket and ask to close it.

[bainhammer]

Exact same decision that I've tested and planned to send to Roman.
Thank you, guys, for cooperation, ticket is closed.

[Sablerider]

Hello there,

We are testing VEB in our Company (MSP). We have tested it with 2 client computers (Both Win 8.1 64bit) in the same local LAN.

As target we first used a CIFS-Share on the local LAN. Both clients backed up with a speed of 40-70 MB/s.
When we switched the Backup-Target to B&R-Repository (same local LAN, same storage) the backup-speed was 4 MB/s.

Are there known performance-issues regarding B&R-Database?

Cheers,
Alex

[Vitaliy S.]

Hello,

What type of backup repository was that, also a CIFS share?

Thank you!

[Sablerider]

Hi Vitaly,

Nope, the B&R-Repository is our main-backup-repos with sql-database (at the moment of backing up the clients there were no other backup-jobs active)

Alex

[Vitaliy S.]

Alex, can you check if you have any throttling settings enabled for the backup repository? If this is not the case, then we would appreciate if you could open a support case and let us know your case ID. This would help us when talking with devs about this issue.