VAL backup for RHEL insanely slow when using CIFS target

[Jablue]

Hi all,

I have been banging my head into a wall for months now trying to get this issue resolved and haven't made much progress. I'm hoping maybe someone on here can help. The core problem is that i'm trying perform a file level backup of specific directories on a RHEL server (7.1) Physical server running quad Xeon e5's w/512GB RAM. The system is running our ERP containing a Progress DB.(I'm excluding the DB files from the backup) Veeam is running on a physical server as well and is the proxy/gateway etc. The backup target is a CIFS share running on a DataDomain repository (because DDBoost is not available). The backups will only run at 4-5 Mbps when all of our other backups run well over 100Mbps. Even when all other backup jobs are paused this job only runs at 4-5Mbps. The whole time the resources on any of the servers never get above 20% everything is connected via multiple 10G links.

The only time I was able to get above that is when I set the repository to the local physical disk of the Veeam/Proxy server in which case it writes at about 120 Mbps. the CIFS is running on the DataDomain, the same DD as all of our other backups and backups are the only thing that the DD is being used for.

Veeam is patched to the latest version as is the VAL. Any ideas? I have seen several similar issues reported online and some suggest setting up a linux proxy and mounting the CIFS share to the linux proxy, has anyone had any success with this?

Thanks!

case 02770224

[PTide]

Hi and welcome to the community.

The backups will only run at 4-5 Mbps when all of our other backups run well over 100Mbps.

What do those jobs backup? VMs or Linux hosts? If the latter then what type of jobs are those - "managed by server" or "managed by agent"?

Would you also share your bottleneck stats, please?

Thank you

[Jablue]

All other backups are VMware backups so I don't expect to get the same efficiency from a physical agent but I know it should be faster than 5mbps.

Bottleneck just says NA.

And the VAL is Managed by Server.

[PTide]

All other backups are VMware backups so I don't expect to get the same efficiency from a physical agent but I know it should be faster than 5mbps.

You've mentioned that the backup speed is OK if you set the job to use local VBR drive, which makes me thinks that it's not the network between the agent and gateway (VBR). I assume that VM jobs use DD boost. Have you tried to send VM backup to the CIFS share?

Bottleneck just says NA.

That is weird. Does it say "NA" on both VBR and Agent (session stats in UI), or is it just on VBR?

Thanks

[Jablue]

I agree it shouldn't be the network, it's 40GB. I have not tried sending the VM jobs the the CIFS share. I might be able to try it with a single test job, I think it definitely has something to do with it being a CIFS share.

All other VBR jobs show bottleneck stats it's just the one VAL job that shows NA.

From what I understand the Veeam server (acting as the proxy, and gateway etc.) mounts the CIFS share and then acts as the data manager for it and receives data from the VAL. The only thing that I think it could be is job configuration, maybe the block size or something?

I'm getting desperate as we have a major ERP upgrade this weekend and I have to get a good backup of this server. What I did for tonight's backup is to deploy a new Windows 2016 VM and assigned it a 2TB local disk and then added that as a repository created a new backup job and pointed it to the new repository. It should verify that it's definitely and issue with Veeam interacting with CIFS. The biggest issue is that I have not been able to find the best practice settings for backing up to a CIFS running on a DataDomain. Lots of settings for DD's and lots for CIFS but they are all opposite.

If my work around works, it will only be good temporarily but at least it should buy me some time

Thanks!

[PTide]

What I did for tonight's backup is to deploy a new Windows 2016 VM and assigned it a 2TB local disk and then added that as a repository created a new backup job and pointed it to the new repository.

If your VBR server acts as a gateway server for the cifs, then in both cases (VAL job or VM job) VBR should host datamover service. I see two options here:

A: If VM backup job pointed to the cifs that is located on DD repo runs slow too, then we need to check if everything is ok with the target datamover location and performance.

B: If VM backup runs just fine, then then we'll search for the reason closer to VAL.

In other words, that would be great if you could check how VM backup to cifs works.

Thanks

[Jablue]

So my VAL backup ran last night, pointed to the new windows VM as a repository instead of the CIFS share. It ran at 9MBps which is better but still far short of what it should be. The bottleneck stats are Load: Source 0% > Proxy 1% > Network 0% > Target 0%. So I don't know if that really helps us a whole lot. In the job configuration under advanced settings. What would give me the best backup performance if I wasn't worried about dedupe or compression? I'm guessing compression set to None? What should the storage optimization be set to? All components are local in the same data center, all connected via 10g network, so if I'm just looking for raw speed?

Thanks again for all of your input.

[Jablue]

Just for comparison a physical windows server completed being backed up to the DDboost repository and the processing rate was 418 MB/s Load: Source 73% > Proxy 10% > Network 4% > Target 1% This is another physical server next to and cabled identically to the linux server with issues and this backup is running concurrently with others. I have the linux job scheduled so that it is running alone.

[PTide]

Thank you for getting back to us with the results - it will help us to figure out what's going on.

So, to summarize:

1. VAL job shows decent speed if VBR's local drive is used as a repo.

2. VAL job crawls if it uses:

- VBR repository on another Windows VM
- DD cifs share presented as VBR repository

3. VMware jobs that use DD-boost repo work fine

4. Windows Agent that uses DD cifs share presented as VBR repository shows magnificent performance over 400Mbps

5. VMware backup performance to DD cifs share presented as VBR repository remains unknown.

Is everything correct?

[Jablue]

Everything here is correct except Item #4

The Windows Agent is writing to the DDBoost Repository not the CIFS. (just to clarify the CIFS share and the DDBoost are the same exact storage, just presented differently because Veeam would not allow us to select a DDBoost repository)

Additionally all other backup jobs in Veeam are completed by 11:30pm and job in question doesn't start until 1:50AM So it is the only job running. The DD is not being used or accessed by any other systems. The Veeam host (proxy/gateway etc.) is running dual 20core procs and 64GB of RAM. Peak usage of either is never above 75% peak network utilization has never gotten higher than 5% (dual 10GB Nics)...

Thanks again for your assistance!

[PTide]

The Windows Agent is writing to the DDBoost Repository not the CIFS. (just to clarify the CIFS share and the DDBoost are the same exact storage, just presented differently because Veeam would not allow us to select a DDBoost repository)

This is what makes the biggest difference. That is, I would expect Windows Agent file-level backup to run very slow, too, if you pointed it to cifs instead of dd-boost. Would you mind trying that for us, please?

What is important to be mentioned is the fact that VAL takes snapshot of the whole volume during file-level backup. That is to say, the exclusions ensure that the files are excluded from a backup file, not from a snapshot.

Thanks

[Jablue]

I have been out of the office sick, but there is one new piece of information. If you recall last week I created a windows VM and added it to Veeam as a new repository and pointed my VAL backup there.the first time it ran it showed a throughput of 9MB/s but during every subsequent run it shows 5MB/s. So if I backup to the Veeam Physical server I get 80+ MB/s but if I backup to any VM I get 5, and if I backup to the data domain I only get 5....So that completely takes the data domain out of the picture. I'm going to attempt to map the same Volume that is mapped to the VM to the directly to the veeam server and theoretically I should get similar 5MB/s performance.