Install master on backup proxy / de-dupe: slow recovery rate

[Petrie]

Hi all,

we recently bought a de-dupe device (EMC DD2500). But we are not happy yet as we are experiencing unexpectedly slow recovery rates.

While backup jobs run satisfactory, recovery jobs run with approx. 20MB/s only. Performing an »Instant Recovery«, it takes a single server 13 minutes to boot (Instant Recovery from local SAN device: 5 minutes).

At least one of the reasons seems to be, that - to our surprise - the traffic runs from the de-dupe device to the VBR master server first, from there to the backup proxy (-> vPower NFS), and finally to the ESXi host. (We would have expected that the backup proxy itself would access the backup repository on the de-dupe device.)

As the VBR master is only a »small« VM, while our two backup proxies are powerful physical servers, we decided to give it a try and install VBR master on one of the backup proxies. We hope to have a big increase in recover processing rates.

1. Regarding the little information given, - would you agree to abandon the VBR master as a VM and to switch to a physical machine?

2. What would the basic steps be in this case?
Like this?
- backup config on old master
- install VBR on new master (this is one of the backup proxies!)
- import config backup
Will it make any trouble, if the former backup proxy now is the master server and if I restore a config backup on it, - a config it was a part of, before?

Thanks for your help!

[Vitaliy S.]

Hi Petrie,

At least one of the reasons seems to be, that - to our surprise - the traffic runs from the de-dupe device to the VBR master server first, from there to the backup proxy (-> vPower NFS), and finally to the ESXi host. (We would have expected that the backup proxy itself would access the backup repository on the de-dupe device.)

This should not be the case. When you run an instant VM recovery traffic does not go to backup server, unless you have a vPower NFS service there and that service is used by the restore job. Do you see traffic going to the backup server in all types of restores you've tried?

1. Regarding the little information given, - would you agree to abandon the VBR master as a VM and to switch to a physical machine?

Moving backup server to a more powerful server will not resolve restore performance issues.

Thanks!

[Petrie]

Hi Vitaly,

This should not be the case. When you run an instant VM recovery traffic does not go to backup server, unless you have a vPower NFS service there and that service is used by the restore job. Do you see traffic going to the backup server in all types of restores you've tried?

This is, what we believe, too. In fact, the traffic goes from the de-dupe device (connected via CIFS) to the VBR master server (the "weak" VM) first, and afterwards to the backup proxy and from there to the ESXi host. This happens when performing an Instant Recovery, as well as when doing a normal "Restore VM" job. - The CIFS backup repository is configured to use the vPower NFS installation on the backup proxy.
I have already opened a case for this (#00503220), but I was informed this behaviour was correct.

However, this would mean we have a bottleneck: the VBR server. I always thought I could avoid performance issues by simply adding further backup proxies, - for backup and recovery jobs.
If - in a desaster case - _every_ VM had to be routed through the VBR master server first (and there is only one of it!), this would have a strong impact on the recovery jobs.

To me, the most interesting about this is that any kind of recovery from _local SAN disks_ works without the detour via the VBR master server!
I do not understand why recoveries from DataDomain-/CIFS-targets only work the other way.

Moving backup server to a more powerful server will not resolve restore performance issues.

Sorry for not being precise enough. My idea was to install the VBR master server on the backup proxy itself in order to avoid the detour, - hoping that the VBR master installed on a virtual machine is the limiting factor. (although I don´t know why, as all machines are connected via 10GbE).

I really appreciate you help.
Thank you very much!

[Vitaliy S.]

The CIFS backup repository is configured to use the vPower NFS installation on the backup proxy.

CIFS repositories cannot host vPower NFS service, as you cannot run any service on the network share.

I was informed this behaviour was correct.

Yes, in your case this behavior is absolutely expected.

]However, this would mean we have a bottleneck: the VBR server. I always thought I could avoid performance issues by simply adding further backup proxies, - for backup and recovery jobs.
If - in a desaster case - _every_ VM had to be routed through the VBR master server first (and there is only one of it!), this would have a strong impact on the recovery jobs.

The main reason of the performance issues is repository type. Since it is CIFS, you cannot run vPower NFS Server on that repository, so the traffic has to go back and forth. What you could do to improve the current performance is to configure vPower NFS server on the proxy server/more powerful VM and specify this vPower instance in the repository settings. Give it a try!

[Petrie]

What you could do to improve the current performance is to configure vPower NFS server on the proxy server/more powerful VM and specify this vPower instance in the repository settings. Give it a try!

This is exactly what we did:

The CIFS backup repository is configured to use the vPower NFS installation on the backup proxy.

Nevertheless, the traffic is routed to the VBR master/management server first!

[dellock6]

Quick guess, is it the Veeam central server is running the repository agetn and so CIFS is mapped via this server? That's why usually is considered a best practice to have a linux or windows machine in front of a dedup appliance, in order to act as the repository.

Luca.

[foggy]

Also, just FYI, performance of backup browsing and restore from DataDomain is a known issue, you can find some details and a couple of hints to address it in this thread.

[Petrie]

Luca,

I am not quite sure, if I understand you right. We have a default installation, one VBR master/management/central server (as VM) and two powerful backup proxies (hardware). The de-dupe device was configured as "add backup repository, / shared folder".
What is a repository agent? And would there be the need to have one per backup proxy?

We also tried to map the DD share on the backup proxy and to add the new backup repository as type "microsoft windows server". But the mapped share does not show up on the next page.

[foggy]

We also tried to map the DD share on the backup proxy and to add the new backup repository as type "microsoft windows server". But the mapped share does not show up on the next page.

Luca is talking right about this type of repository, where you mount share to the Windows server (e.g. via iSCSI) and add this server to Veeam B&R as a repository. In this case Veeam data mover agent processing VM disks during restore is running on this Windows server (as well as vPower NFS service can be configured there).

[Vitaliy S.]

We also tried to map the DD share on the backup proxy and to add the new backup repository as type "microsoft windows server". But the mapped share does not show up on the next page.

Mapped drives are created only during interactive logins, so I believe this is expected. See this topic for further info > CIFS target at remote branch on private backup network.

[Petrie]

I am confused, - why can´t the backup proxy be this windows server? What is the benefit of another windows server in between?
I mapped the share to the backup proxy and added this server as a repository, - but the mapped share does not show up.
Where is my fault?

[foggy]

See Vitaliy's reply for the answer.

What you could do is specify your proxy server as the proxying server in the CIFS repository settings (at the Share step). This will run the corresponding agent directly on the proxy server.

[Petrie]

Vitaly, sorry, we answered at the time.
I just saw your post and will now have a look at the linked article! Thank you!

[Vitaliy S.]

Foggy's tip should allow you to run the agent on the "correct" server. Let us know if that helps.