Veeam setup for evaluation

[VladV]

We are currently using Backup Exec 2012 and our support contract is about to expire, so this is a good time to evaluate other backup solutions.

I need some advice on an optimal configuration just to get things started. First of all our vm environment is as follows:

3 esxi hosts managed by vcenter (VCSA) 2 of which (let's call them ESX1 and ESX2) are identical thus members of a cluster and one (ESX3) is member of another
All 3 esxi hosts have 2x10G nics
The storage is a NFS server based on Nexenta enterprise
The connection to the storage is done through one of the two 10G nics on each esxi and the vms have access to 4 x 1GB nics.
ESX1 and ESX2 host the vm's which I want to protect and ESX3 will host the veeam backup server.
The third esxi also has access to a separate local storage of about 7TB RAID10 and a remote NFS storage also of 7TB RAID5.

So this is my plan:
As mentioned above I would use ESX3 to host the backup server and repository as it has direct access to the local storage.
From what I understand the only two transport modes available for NFS are HotAdd and network mode. From my experience with Backup Exec HotAdd made consolidating snapshots have a bad impact on the vm's and also the consolidation process took longer.

That leaves us with network mode. As with backup exec I believe that the proxy will pull the vmdks through the management vmk.
One question is: in case I have two management vmks (as recommended for HA) which of the two will veeam use.
The other question will be: if I move the management vmk on a 10G network and if the proxy and repository are on a 10G nic would I see a benefit or is there a speed limit somewhere disregarding the actual available speed. In backup exec, using NBD transport mode the speed was 3500MB/min but using 1GB nics for management and the backup server.

I would like to have your opinions on my thinking and maybe let me know if I am going in the right direction or if there is a better setup available. I also want to be able to benefit from Instant VM recovery (one of the main motives of trying veeam). Your advice will make our evaluation a lot more productive and hopefully shorter too.

Thank you in advance,
Vlad

[Vitaliy S.]

Hi Vlad,

That leaves us with network mode. As with backup exec I believe that the proxy will pull the vmdks through the management vmk.

Yes, when you're using network mode, all VM traffic will go through the network interface of your ESXi host.

One question is: in case I have two management vmks (as recommended for HA) which of the two will veeam use.

It will use the same interface as the vSphere Client will use while downloading/uploading files to the datastore.

The other question will be: if I move the management vmk on a 10G network and if the proxy and repository are on a 10G nic would I see a benefit or is there a speed limit somewhere disregarding the actual available speed. In backup exec, using NBD transport mode the speed was 3500MB/min but using 1GB nics for management and the backup server.

There is no hard speed limit, so you will definitely benefit from using network processing mode via 10 GBs NICs.

I also want to be able to benefit from Instant VM recovery (one of the main motives of trying veeam).

Instant VM recovery is indeed a great feature, and there is no extra configuration required. What your target repository will look like? Are you going to store all your backup on the virtual disk of the VM hosted on the ESX3 server?

Thanks!

[foggy]

From what I understand the only two transport modes available for NFS are HotAdd and network mode. From my experience with Backup Exec HotAdd made consolidating snapshots have a bad impact on the vm's and also the consolidation process took longer.

Indeed, using hotadd with NFS causes longer VM stun than usual (known VMware issue). However, this does not prevent you from using hotadd for faster full VM restores, so I'd recommend to have virtual proxy capable of hotadd on the source hosts to benefit from this ability.

It will use the same interface as the vSphere Client will use while downloading/uploading files to the datastore.

Moreover, if both are registered with the same hostname in DNS, each connection to the host (i.e. each backup job) will use a different interface. This will allow you to spread the load in some sense by having several simultaneous jobs.

Are you going to store all your backup on the virtual disk of the VM hosted on the ESX3 server?

Which is definitely not considered to be the best practice.

[VladV]

Indeed, using hotadd with NFS causes longer VM stun than usual (known VMware issue). However, this does not prevent you from using hotadd for faster full VM restores, so I'd recommend to have virtual proxy capable of hotadd on the source hosts to benefit from this ability.

Thank you for the tip. I will experiment full restores using hotadd.

It will use the same interface as the vSphere Client will use while downloading/uploading files to the datastore.
Moreover, if both are registered with the same hostname in DNS, each connection to the host (i.e. each backup job) will use a different interface. This will allow you to spread the load in some sense by having several simultaneous jobs.

In DNS i have only specified one IP for one VMK, the other is not specified. Moreover, the esxi hosts were added to vcenter using the dns name.
After posting I ran a first test backup job using network mode, and to my surprise it used the second VMK not the "primary" one. I had to delete it so I could run my tests.

There is no hard speed limit, so you will definitely benefit from using network processing mode via 10 GBs NICs.

Using the 10G vmk I wasn't able to pass 62MB/s processing rate (which is lower than I expected) using Inline dedup, optimal compression and optimize for local target settings. Also I ticked Enable application-aware image processing and Enable guest file system indexing. I used as a target storage an 50GB vmdk residing on the local RAID10 storage.

The Veeam backup server has 8 vcpu and 8GB ram. I will try to find where is the bottleneck.

Are you going to store all your backup on the virtual disk of the VM hosted on the ESX3 server?
Which is definitely not considered to be the best practice.

Unfortunately I cannot see a better solution. With backup exec I used the local storage (8 x 2 TB drives in HW RAID 10) by creating 4 vmdks (4 x TB) and then spanning them in Windows Server. I also configured a remote storage (SMB share) on a Synology server comprised of 5 x 2TB in RAID 5 for duplication target.

I am open to any advice so if there is a better solution to use the internal storage of the third esxi host please let me know.

[Vitaliy S.]

The Veeam backup server has 8 vcpu and 8GB ram. I will try to find where is the bottleneck.

You can quickly detect what the bottleneck is by reviewing the corresponding job session in the Veeam backup console.

Unfortunately I cannot see a better solution. With backup exec I used the local storage (8 x 2 TB drives in HW RAID 10) by creating 4 vmdks (4 x TB) and then spanning them in Windows Server.

I am open to any advice so if there is a better solution to use the internal storage of the third esxi host please let me know.

Since this is a local storage, then I'm afraid you're limited to what you have configured right now.

I also configured a remote storage (SMB share) on a Synology server comprised of 5 x 2TB in RAID 5 for duplication target.

Also are you planning to copy backup files to this Synology server (via script) or this is going to be just another backup repository for duplicated jobs?

[chrisdearden]

you could present iscsi from the synology to the backup server ( in guest initiated )

[VladV]

you could present iscsi from the synology to the backup server ( in guest initiated )

Thanks, I will give that a try later on.

You can quickly detect what the bottleneck is by reviewing the corresponding job session in the Veeam backup console.

From the beginning and for every backup test since then it has been the source as detected bottleneck and in the summary it was 99% source>27% proxy>1%target.

I managed to increase the speed to 112MB/s by changing the veeam server vm nic type from E1000E to vmxnet3 and enabling jumbo frames on switch, vswitch and guest OS. But the fact that it remains near 1GB/s speed points to other limits. Either way above 70MB/s for network mode I believe is ok.

Also are you planning to copy backup files to this Synology server (via script) or this is going to be just another backup repository for duplicated jobs?

I haven't studied this scenario yet and haven't decided on a strategy, but I will after further reading through the documentation and testing the application more.

After the first few hours using Veeam I can't believe we spent so much time using (debugging mainly) Backup Exec . I hope the tests go well with Veeam further on.

[Vitaliy S.]

I haven't studied this scenario yet and haven't decided on a strategy, but I will after further reading through the documentation and testing the application more.

Here you go a couple of topics that should help: onsite and offsite backup and Best backup setup for mirroring backup respository

[VladV]

After further tweaking for maximum throughput on network mode I have come to the conclusion that having the proxy on the same vswitch as the management network will slightly increase the speed and also save some bandwidth by not using the actual physical switch for reading the vmdks.

In my case of 2 ESXi hosts that hold our VMs, without any typical arrangement of the VMs on the 2 hosts, is it possible to tie one Proxy for each host to be able to use the above configuration?

Regards,
Vlad

[veremin]

In my case of 2 ESXi hosts that hold our VMs, without any typical arrangement of the VMs on the 2 hosts, is it possible to tie one Proxy for each host to be able to use the above configuration?

In fact, in the job settings you can explicitly select backup proxies that should be used.

So, in the settings of the job#1 that is responsible for backing up VMs that reside on Host#1 select Proxy#1 to be used, in job#2 – Proxy#2.

Additionally, if there are quite a lot of jobs that need to be changed that way, you can modify slightly and utilize this PS script that is responsible for setting given backup proxy.

Hope this helps.
Thanks.

[VladV]

So, in the settings of the job#1 that is responsible for backing up VMs that reside on Host#1 select Proxy#1 to be used, in job#2 – Proxy#2.

The thing is that the VMs are moving quite a few times a week and do not keep their existing position. I was hoping that, by selecting the 2 proxys in a job, the job would automatically select the proxy residing on the same host as the targeted VMs.

If I select the proxy manually, then after a week, the VMs are "shuffled" between the two hosts and the proxy will pull data through the physical switch to reach the management VMK of the host on which the VMs are residing.

Either way, this is not an important issue, I was just trying to squeeze every resource possible. It is still faster than our previous backup solution.

[VladV]

you could present iscsi from the synology to the backup server ( in guest initiated )

Is it better to use the Synology NAS as an iSCSI target or use it as a SMB/CIFS windows share for storing our backups?
One thing I noticed is that generating synthetic fulls after a week of incrementals it takes roughly the same as doing an active full on CIFS/SMB.

Regards,
Vlad

[veremin]

The thing is that the VMs are moving quite a few times a week and do not keep their existing position.

If VMs are being constantly moved between two existing ESXi hosts, then, you might want to add vm containers (rather than individual VMs) such as VM folders, host or datastores to the backup job and explicitly specify proxy to be used.

I was hoping that, by selecting the 2 proxys in a job, the job would automatically select the proxy residing on the same host as the targeted VMs.

Moreover, there is a detailed description of how the backup proxies are automatically selected. Thus, it stands to reason to spend some time thinking thoughtfully about your backup schedule, so that, each time the corresponding job is started the proxy that resides on required host is available (or has least tasks).

Hope this helps.
Thanks.

[foggy]

Is it better to use the Synology NAS as an iSCSI target or use it as a SMB/CIFS windows share for storing our backups?

iSCSI is generally preferred for reliability considerations.