DXi Slow backup without manual intervention

[Aremac]

Support case : [ID#5189263]

I have a strange situation I wonder if others have experienced. Veeam backup 6.0.0.181

Writing backups to a Quantum Dxi using CIFS presentation backup performance is very poor. If looking at network stats on the proxy server in perfmon .5 - 1 % of 10GB connection. However If I open a window and browse via UNC or map a drive letter to the CIFS share a 'browse' my backup performance kicks up to 10-25% of the 10Gb connection. That performance persists until the drive being backed up completes and the next drive begins (on vm with multiple drives of course)

This behavior is consistent accross all backup proxies and the Veeam server itself. All servers are windows 2008.

Ideas?

[Gostev]

Do you see the same with a regular Windows share?

[Aremac]

No. copy with regular windows share works fine. I can also copy via command prompt from local system to the DXi without opening a window or browsing the CIFS share and performance is fine. The only time I see this behavior is with a Veeam Job. I have opened a support incident with Quantum to look at things from that side but have a hard time thinking its a DXi issue since Windows can write to it fine. Items i've checked include:

SMB signing, LDAP signing, rights & permissions (share and NTFS level), different user accounts, different proxy servers, Network settings (jumbo, etc)

[Gostev]

OK, let's wait and see what they find out then.

[Spex]

Support case : [ID#5189263]

I have a strange situation I wonder if others have experienced. Veeam backup 6.0.0.181

Writing backups to a Quantum Dxi using CIFS presentation backup performance is very poor. If looking at network stats on the proxy server in perfmon .5 - 1 % of 10GB connection. However If I open a window and browse via UNC or map a drive letter to the CIFS share a 'browse' my backup performance kicks up to 10-25% of the 10Gb connection. That performance persists until the drive being backed up completes and the next drive begins (on vm with multiple drives of course)

This behavior is consistent accross all backup proxies and the Veeam server itself. All servers are windows 2008.

Ideas?

I think we have the same situation/problem. What exactly did you do to improve backup speed?

[gibsonjv]

I'm seeing the same behavior with my DXi8500. I can copy directly to the CIFS share from my desktop at ~40MB/s, but my SAN connected proxy can only push ~14MB/s to the same share. The proxy does not appear to be experiencing network or processing bottlenecks.

[J1mbo]

Might Veeam proxy be opening files as uncached I wonder.

[b.vanhaastrecht]

Hi, this is a old post with no solution. I wonder if there is any?

I'm currently building a setup with a DXi 6702 and have problems utilizing enough SMB traffic to it. Can only push about 150MB/s on a complete 10GbE backed infrastructure. Veeam and normal windows copy show the same speed. DXi CPU/RAM/IO/Network does not seem to be constrained.

I've played with a lot of NIC/TCP/SMB related settings in Windows, won some performance but I want to see at least 200MB/s. Have disabled SMB signing, disabled/enabled RSS, MTU (Jumbo Frames), Flow Control, TCP offloading ext.

I wonder if the topic starters/other DXi users have any thoughts on this.

[dellock6]

How many cuncurrent jobs are you executing against the appliance? You can try to run multiple jobs at the same time so to leverage multiple I/O streams. the speed of a single job would probably not increase, but the overall backup speed will.

Luca.

[b.vanhaastrecht]

Hi Luca, I wil defenetely try to utilize more jobs to see if that inprove performance. Will get back on that.

I've just finished some other testing with an CentOS Linux with NFS mounted to the DXi. This improves trhuput, but only with a small percentage.

As I'm referring to copy job speed, I've found a mayor impact on thruput with the "Decompress data block before storing" option. It states it would slow down backup performance, but on my setup it's 300% impact. Both my NFS en CIFS tests, when enabled it gives me an average of 120MB/s, when disabled (so no compression) I get 360MB/s. Now thats the speed I want, but not Veeam compression/dedupe.

I take the decompress requires memory and mostly CPU performance. My (test) server has 8 core's, and they are all fully saturated when decompress is selected.
1) Is this normal? As for backup (compress) it needs only one core.
2) What can I do to improve decompress performance?

[foggy]

2) What can I do to improve decompress performance?

Just to be sure, how do you set up your target repositories? Apart from the fact that data decompression itself takes considerable time, it is performed by the target agent and, in case of remote job, the latter should be running in the remote location so that the data be transferred in a compressed format and then be decompressed just before being written to the disk.

[b.vanhaastrecht]

Hi Alexander. Let me sum the setup quickly:
- The NFS test is done with a CentOS VM with 4 vCPU and 8GB RAM with an NFS mountpoint to the DXI. This server is added as a Linux server with the mountpoint as repository.
- The proxy is a physical HP DL380 G6 with 16GB RAM and 8x 2.6 Ghz Cores and 900GB 15K SAS disks.
- The CIFS/SMB DXi repository tests are done on the same physical proxy server setup as a CIFS share.

All three nodes, proxy, NFS repository and DXi are connected via 10GbE infrastructure.

I was just about to post this: Done some more testing to see if the backup job compression and inline dedupe makes any difference regarding the copy job thruput with the repository set to decompress (NFS/CIFS tests show same result). The results are suprising!

Code: Select all

Backup Job settings: Inline dedupe enabled / No compression / Local target: 120MB/s (All CPU's at 100%, veeamagent.exe)
Backup Job settings: Inline dedupe disabled / No compression / Local target: 120MB/s (All CPU's at 100%, veeamagent.exe)

To my knowlegde I've set all posible compressions to disable, and still the copy job with decompress set uses 100% CPU and only tops 120MB/s. Whats there to decompress when the backupped data is no longer compressed?

[foggy]

To my knowlegde I've set all posible compressions to disable, and still the copy job with decompress set uses 100% CPU and only tops 120MB/s. Whats there to decompress when the backupped data is no longer compressed?

Do you also have compression disabled on the backup copy job itself?

[b.vanhaastrecht]

No I have not. How is this settings different from the Decompress settings on the target repository?

[foggy]

Decompression setting on the target repository applies to the backups that are compressed by the backup or backup copy job. If compression is enabled on the job and you have decompression setting enabled on the repository it is writing to, data will be decompressed by the agent running on this repository.

[b.vanhaastrecht]

Decompression setting on the target repository applies to the backups that are compressed by the backup or backup copy job. If compression is enabled on the job and you have decompression setting enabled on the repository it is writing to, data will be decompressed by the agent running on this repository.

Ok, but sorry, I'm having diffeculty to project this matter on our setup.

I have a backup job, no inline, no compression set. It's backupped to a local disk repository on the proxy server.
I have a copy job, linked to this backup job. It's set to copy to the DXi CIFS share repository set to decompress on save.

Were in this backup & backup copy chain setup does the compress setting on the copy job have any influence?

[foggy]

Please check compression settings on the backup copy job (Target step, Advanced > Storage tab).

[b.vanhaastrecht]

Please check compression settings on the backup copy job (Target step, Advanced > Storage tab).

Yes, thank you for pointing it out, but I know were the option is. But I dont understand why this option does anything because in our chain no compression is used. Could you elaborate, looking at my previous post what exactly this setting does? As to me it's unclear because the backup job has no compression, the target repository has set to decompress, so why does this copy job compression setting have any use? On what exactly?

Sorry for not understanding, but I would very like this particular part to be clear for me.

[foggy]

Backup copy job has its own compression and deduplication settings, which apply to the data it transfers between repositories. Even if the backup files created by the backup job are stored uncompressed, the data transferred by the backup copy job will be compressed according to its settings prior to be sent to the target repository (and decompressed there, if the repository has corresponding setting enabled).

[b.vanhaastrecht]

Decompression setting on the target repository applies to the backups that are compressed by the backup or backup copy job. If compression is enabled on the job and you have decompression setting enabled on the repository it is writing to, data will be decompressed by the agent running on this repository.

Supose the backups are stored on a local repository on the proxy server itself, and the copy job is writing to a CIFS share (were there's no active agent running), this copy job compress setting is irrelevant right? It just the proxy on the same box which copy's the local repository data to the CIFS share, no receiving repository agent. (Like in our setup).

[foggy]

If you have compression level for the backup copy job set to "Auto", the "no compression" setting will be inherited from the original backup job and data indeed will not be compressed. However, if there's some different compression level, I assume the data still will be compressed and decompressed, even though the agents are on the same machine (will try to confirm that with QA).

[pdimarco]

Hi there,

We are working on a POC at BNP Paribas , and are experiencing a similar issue...

On DXi 6200 as a CIFS repository, we are experiencing a "low" througput (around 25 MB/s) on a job of 4 VMs. Of course, paralelization has been setted OK.

A direct copy of a file to the DXI CIFS share via windows explorer is showing up at around 100 MB/s. This sounds good on a 1 Gb/s network link.

The workaround has been to create 4 backup jobs, each containing 1 VM.

We then experienced a nice throughput of 100 MB/s.

Has any one a new idea on why this happens ?

Cheers
Pascal

[b.vanhaastrecht]

What are your (de)compression settings on the job and DXi repository?

In my situation it has something to do with decompression, this pushes the CPU load to the max. Have you also seen this post? http://forums.veeam.com/veeam-backup-re ... ml#p119500

[foggy]

The workaround has been to create 4 backup jobs, each containing 1 VM.

We then experienced a nice throughput of 100 MB/s.

Has any one a new idea on why this happens ?

Multiple parallel jobs leverage multiple I/O streams.

[pdimarco]

What are your (de)compression settings on the job and DXi repository?

In my situation it has something to do with decompression, this pushes the CPU load to the max. Have you also seen this post? http://forums.veeam.com/veeam-backup-re ... ml#p119500

Hi,

We did check all this, this wasn't the issue.

Cheers
Pascal