Data Domain Single Gateway Server Bottle Neck

[bdunington]

Greetings Veeam Team,

The Per VM file change removed the massive backup file bottle neck but what about the single Gateway Server bottle neck for the Data Domain? All we see and hear lately is Veeam is "Enterprise Ready" but this is not a enterprise approach!

Having a single process doing what looks to me as a read & restore to the secondary in a copy process is crippling.

I'm all ears..........

[foggy]

In case per-VM backup chain is enabled, gateway is selected per every VM in the job.

[bdunington]

Thank you for your quick reply. Sorry I wasn’t clear in my earlier post. (Long day)

My questions/observations is mostly around the remote copy process using a Data Domain as the Repository.

1. You set a single Gateway server per Data Domain Repository
2. Local proxies can be configured to be their own gateway server when writing direct to Data Domain during a backup job. This scales with no issues so far.
3. When running a remote copy Job the Gateway Server does all the work. The reads form the source repository and the write to the remote repository. This doesn’t scale and is killing us.

How do we scale the remotes? We need to get to 6000 VM’s and we can barely do 2500.

[foggy]

Could you please describe your remote backup copy setup (specifically repositories) in more detail? Are you running backup copy between two DD repositories? How do you configure gateway servers for them (manual/automatic)?

[bdunington]

Thank you for your quick reply

Yes we do have Data Domains at both sides and the Gateways servers are manually assigned since we put the largest physical boxes in those roles. The Gateway also has that role only and is not also a proxy. The remote copy jobs kick-off after the match local is completed. We follow your written best practices cities for Data Domain.

More than happy do do a webex/support session with your team but I don't want to start from scratch with tier 1

My co-worker did open a ticket but it didn't go anywhere #02393435

[foggy]

So you have one gateway on the source that reads the data and another one on target that writes it to the target repository, while initially you've mentioned it is the same server that does both?

[bdunington]

Hi Foggy

Sorry for the confusion. We have manually set the Gateway Server for both the local Data Domain as well the remote Data Domain on the Backup Server. In the past we has set using two separate Gateways Server but this was not optimal since it added an additional hop.

any suggestions?

[foggy]

In this case all the processing will be indeed done by this single server. In case you switch to automatic gateway server selection and per-VM chains are enabled, processing will be distributed among proxy servers.

[bdunington]

We found in the past with that setting enabled the backup server did all the processing and ground everything to a halt!

remember I'm talking about the backup copy jobs not local.

[bdunington]

Hi Foggy,

Any update or suggestions? This limitation on remote copies has halted our purchase of additional Veeam capticy and started the discussion of Veeam future in our data centre. I will let our local Veeam team comment.

We really like working with Veeam ....but show us how it scales with out a massive re-investment in storage for repository every couple of years.

[Andreas Neufert]

Hi,
thanks for all the feedback. It is a bit hard to understand what you want to achive.

If you set Gateway Server to Automatic Selection, it will try to use the Mount Server ( of the Repository) and will fallback to Backup Server (B&R Server) for all kind of processing that involve not directly VM backup (Merges/Synthetic processing/Compact/...).

If you go over site, you have 2 options to transport all this. Placing a Proxy only on one side (hard selected gateway server) and let DDBoost do all the traffic optimization.
As well you can place a Veeam Managed Server on both sides and select them as gateway server for each site. This would optimize any background processing. I recommend to do this.
If you want to spread the load accross multiple Servers per site, it is maybe a good idea to look at SOBR or create multiple repositories and spread the load manually.

Andy

[bpayne]

I second bdunington. We have a very similar setup and I have the same problem. Automatic selection of the gateway server seems to ALWAYS use the Mount Server (which is my backup server). The solution I see is that it needs to perform true automatic selection by load balancing some 30 available proxy servers I have sitting around and not defaulting to the on mount server or Backup Server.

bdunington - as a work around, I had to create multiple Data Domain MTREE's and Veeam Backup Repositories (eventhough its the same Data Domain). By creating multiple backup repositories, I could then choose a different gateway server for each repository. Then multiple Backup Copy jobs tied its its respective backup repository. But careful with the concurrent tasks stream count, you can easily overwhelm your Data Domain depending on which model you're using (DD4500 here). You can apply this same logic to regular Backup Jobs, but I currently dont have the issue there - only with Backup Copy jobs. I'm not suggesting this as a good work-around, but it helps my environment.

[foggy]

Correct, in case of backup copy there's no similar mechanism to distribute the load, mount server is selected as a gateway for all VMs in the job. SOBR could be a solution, as Andreas has mentioned.