Design Considerations- Physical Veeam Proxy/Repository

[marty9876]

Hello All,

I've moving away from a all virtual setup for Veeam proxies/CIFS target on a NAS to a physical Veeam proxy/repository and would appreciate your input. My biggest question is where do I put the backup proxies, on a VM on the VMware hosts still (one per host) or let the physical box do all the heavy lifting?

Decision criteria:

- Reduce impact on production VM's (I get into trouble with a job still running during production hours on a proxy, I want the job to finish is the struggle)
- Speed, I'm been averaging ~ 10MB/s to the junk NAS and I'm sick of long backup windows
- Reliability, still not sure if Veeam really does a great job picking which proxy to use when if there are more than one. I've seen Veeam pick a proxy not on the host frequently which means it's network mode anyways.
- Will run backup job to offsite target

Infrastructure will be:

- Three (3) VMware hosts all with DAS storage
- One (1) physical server dedicated to Veeam operations, both repository and other roles as needed. Server 2012 R2, Storage Spaces based on twelve 2TB SATA drives. HP DL380 G9, six-core/16GB RAM box. Will likely use Server 2012 deduplication at a later time.
- Simple flat network, Veeam target on same switches as the hosts

Questions:
- Veeam proxies on the VM's or just one on the physical Veeam box?
- Transport mode? If proxy on physical host, network is the only option. If on VM's, leave to auto? (I'm gotten into trouble with hot add before)
- Split the backup jobs into three different backup jobs and relay on Server 2012 to do the final deduplication or keep the backup job in one big job? Target for offsite backup copy is somewhat space challenged, unsure if three backup copy jobs can hit one backup copy job.
- Storage spaces configuration recommendations - mirror/parity/double parity etc.

Thanks,
Marty

[Shestakov]

Hello Marty,

Veeam proxies on the VM's or just one on the physical Veeam box?
- Transport mode? If proxy on physical host, network is the only option. If on VM's, leave to auto? (I'm gotten into trouble with hot add before)

If you want to get the fastest connection(Direct SAN mode), a machine used as a backup proxy should have direct access to the storage on which VMs reside or the storage where VM data is written. This way, the backup proxy will retrieve data directly from the datastore, bypassing LAN.
Please review FAQ to get more understanding about proxies deployment.

You cannot select a transport mode for writing data — Veeam Backup & Replication selects it automatically, based on the configuration of the backup proxy and transport mode limitations.
Direct SAN Access limitations, Virtual Appliance(HotAdd) limitations.
What kind of trouble have you experienced with the Hot-add mode?

Split the backup jobs into three different backup jobs and relay on Server 2012 to do the final deduplication or keep the backup job in one big job? Target for offsite backup copy is somewhat space challenged, unsure if three backup copy jobs can hit one backup copy job.

If you have VMs running same OS, same apps, it is recommended to combine them into one job to get better dedupe rate between them.

- Storage spaces configuration recommendations - mirror/parity/double parity etc.

It`s recommended to assign different repositories to jobs and limit the number of parallel jobs for each one, to balance the load across your backup infrastructure. Partitions are up to you.
This discussion contains a lot of useful information about storage configuration as well.
Thank you.

[marty9876]

A machine used as a backup proxy should have direct access to the storage on which VMs reside or the storage where VM data is written. This way, the backup proxy will retrieve data directly from the datastore, bypassing LAN.

Not trying to split hairs but you used the work "or" in there. In testing just to a temporary setup I was getting ~ 50MB/s by only using the proxy on the physical machine and only ~ 30MB/s by using a VM as a phoxy on the save physical host at the VMware datastore. This is whats driving some of my confision, I assumed proxy to proxy should have been the fastest but it wasn't.

What kind of trouble have you experienced with the Hot-add mode?

Back in the Veeam 6.x days I'd have drives not being removed from the proxies after a hot add operation.

Thanks for the input!

[Shestakov]

Well, yes, in some cases you can experience faster Network mode operations than Hot-Add, especially on small VMs (it takes more time for hotadd operations than the backup itself sometimes...), since hotadd requires some pre- and post-job activities related to VMware operations, to mount and unmount the vmdk to the hotadd proxy. Network mode instead starts immediately to copy data after the snapshot is created.
Here is another good explanation:

Hotadd is generally faster and reading and writing data than network mode, even over 10GbE (although the gap is narrowed significantly). However, there are several factors that can make network mode be a real win.

1. When you have a lot of servers that generally have a small change rate, and you have 10GbE, it's unlikely that hotadd is going to be a win for the incremental transfers. The reason why is simple to understand if you think about it. As was noted the hotadd process itself has a lot of steps, after the snapshot is taken Veeam has to tell vCenter to attach the disk to the proxy VM, the proxy OS has to scan and find the new disk, and at the end of the job all of this has to happen in reverse before the snapshot is taken. In a lab or very fast setup this can take 30 seconds, but in the "real world" this can easily be 60-90 seconds or more. That's 60-90 seconds where no data is being transferred at all. If I have 100 servers that 100-150 minutes of time spend doing nothing but setting up and tearing down hotadd. With replication you have to do all of this on both the source and target proxies (or twice on the same proxy if it's local replication).

If you have 10GbE for the ESXi mangement network, network mode can easily transfer 300-400MB/s per host, hotadd might be able to do faster, but how much? Even if it's twice as fast, and you have 6GB of data to transfer, that's 20 seconds to transfer data for Network mode, 10 seconds for Hotadd, but hotadd still needs 60-90 seconds for all of the setup/teardown, so that's 20 seconds vs 70 seconds best case, so network mode wins!!

2. With replication there is another special issue with using hotadd that can cause a huge amount of I/O on the target, especially for block devices. This is typically an issue when replication happens over a high speed link and you see "Target" at the bottleneck. In this case, you're going to be way better off using Network mode on the target side at least (you can still use hotadd on the source if it makes sense for you). The good thing I can say is that, at least so far in my limited testing, this specific issue is resolved in 8.0, but testing is ongoing and I'm waiting for some field results to confirm 100%.

3. Hotadd and NFS has a tendency to not get along well due to NFS locking issues. This doesn't really impact replication to the target since the VMs are powered off and powered off VMs don't have locks, but when using Hotadd with NFS at the source, it can be huge issue if you don't have a proxy on every host and you don't set a special registry key for Veeam to use only proxies on the same host as the VM in question. Or you can just use Network mode. You can read more about this here.

To optimize backup, and especially regular replication, you really have to look at the details of the logs and understand what the system is spending its time doing before you decide if hotadd is "faster". Hotadd will almost certainly be faster for full backups, but with a good solid Veeam setup you hopefully won't be running full backups very often, so that may not be the ideal focus of the design as, in most cases incremental data transfer won't be the number one contributor to the length of the job runs.

To sum everything up, there are 3 transport modes: Direct SAN (recommended), Hot-Add (Virtual Appliance) and Network. Please check the link to see how proxy should be configured for these modes.
Also if the mode doesn`t work as fast as should, the bottleneck analysis should be done first.