Infrastructure design for Veeam using 3 hosts and 2 iscsi sa

[teza8]

Hi, I've had Veeam for a year now but gave up trying to set it up in my environment as the backups seemed too slow. I feel like I've wasted money here. Everyone tells me how easy it is and indeed it looks very straight forward. Set up was easy indeed (onto a 2008 x64 vm) and I know enough about VMware to get by. It’s simply to the two iscsi sans and my network that is causing the headache. My current backups are using retrospect and backing up files across the network to a file server backup nas. I also use vcenter to clone the servers at weekends using scheduled tasks to turn the vms off/clone/then back on again. The clone copies them from a lun on iscsi storage 1 to another lun on iscsi storage 2. This works but I'd rather get veeam up and running. I am still in my maintenance period and qualify for v6.5. My current infrastructure is VMware essentials v5.0 (not 5.1 yet) with 3 hosts clustered and one spare host with VMware on too. All hosts are Dell 2950 with 2x Xeon quad cores and 32Gb ram each. The single Dell 2950 has three large Hdds. The three clustered hosts are attached to two Qnap 859 rack sans. Both sans have 8 hard drives one has 8x 3Tb hard drives (production nas/san) the other has 8x 2Tb hard drives (back up san/nas). All production vms sit on the iscsi san with 8x 3Tb disk (called black due to colour schemes on web interface of san). The 2Tb x8 is called green.

I'd originally stated looking at backing up from black iscsi san to green iscsi san but quickly realised that’s not how veeam works. So I looked at the two different aspects ‘backup’ and ‘replication’ I then set half of green as NTFS and half as VMFS. The veeam virtual server then saw the NTFS as backup space on the windows (veeam virtual) server using Microsoft iscsi as I’d read this was correct. I also set the VMFS area for replications of vms. The slowness I think was due to iscsi traffic? Each of the 3x live VMware hosts have two networks cards (for redundancy) going into a separated network switch for iscsi traffic. The two san/nas has two nics each that go into the same switch too. The Qnap san/nas has functionality to bond both nics as one. The slowness I think was the backup was going down the iscsi nic from iscsi Qnap san 1 (black) then back up/down again to iscsi san 2 (green). Does that make sense? As this is my only storage really I wanted to use the two sans for this purpose but couldn’t obviously get the speed I expected. I did try to contact veeam and VMware for support but was correctly told (I think) that this is more of a training issue. I work at a school and have pretty much taught myself VMware so don’t have a trainer to ask. Each VMware host has 8 network cards. Two for production lan, two for management/vmotion, two for iscsi, one for ADSL and one for test.
Now that you can see my set up and what I’ve got is there a better way to set this up?
Any help would be appreciated.

[Vitaliy S.]

Hi Terry,

In order to help you with improving your backup/replication job performance it would help if you could post the bottleneck statistics for the jobs you've run. Can you do that?

Thanks!

[teza8]

Hi, thanks for the response.
I'm trying to get information on infrastructure set up more than anything else. I'm convinced I've got it wrong and removed it all. I really need to understand how to use what hardware I've got to best effect. The two sans are on the same ip range and only visible to the pairs of iscsi nics. The 3 clustered esxi hosts can see all luns on both sans. I can see what caused the bottlenecks as I researched it at the time it was using the veeam server to be everything and be proxy and backup. Also as traffic was using the iscsi nics it was slowing there also. I just couldn't figure out the network layout and had toward as I got really busy.

[Vitaliy S.]

Terry,

If I were you, I would give HotAdd processing mode a try. In this case you do not need to configure direct SAN access for your proxy VMs. Just place Veeam backup proxy on the VMware Cluster, add a backup repository role to this server and point it to your iSCSI SAN (via in-guest iSCSI initiator). Let me know what performance results you will get with this kind of deployment.

Thanks!

[teza8]

Hi, thanks for you ongoing help. I'll give a go what you've suggested. Could you suggest a typical network layout with all of the kit I've listed? I want to work out what should go where,what backs up what, on what network, using which nics. Can you see what I'm asking for? This is where my mental block is.

[Vitaliy S.]

If you're going to use HotAdd backup mode, then all VMs will be retrieved through the ESX(i) I/O stack. The thing that matters is how your backup repository will be connected to the backup proxy, doing that via iSCSI (to the green SAN) does seem like a right solution to me, but it's hard to say why you had poor performance without seeing the bottleneck stats.

[teza8]

Vitaliy, I think I finally got this working. I set up veeam again yesterday on a vm which was located on esx host number 4 (not part of the 3 clustered hosts for better processing of backups). Veeam was installed on Ms Win 2008R2. I ensured that automount was disabled on the server. I also set up iscsi on veeam server and took the iqn details and set them to read only on all luns on green (back up san) and black (live san). I then reset one of the empty luns on green as read write. Ensuring I had the correct lun, I then formatted it and initialised it and added it as local storage to the veeam server. So I was ready to start using veeam with all luns visible via iscsi and large storage ready. I added the vcenter server which in turn showed the three live hosts. I set up the backup repository as the large attached ‘local’ storage. I then set the veeam server as the only proxy (default) and the method to direct access with network fall back. A test back up was then created to back up a live windows 2003 server of 50Gb from one of the live luns on black.

I opened the job up to see its status whilst copying and also opened up task manager networking to see network activity. The job took 90 minutes and most of the activity was on the lan nic.
I repeated the test setting up another back up job of the same server using ‘virtual machine access as the backup method and got exactly the same results.

I left it for a day to think about the results.

The next day I set about looking at the network set up which is where I feel there is something wrong. I’d initially set up with a pair of nics using iscsi, a pair for production lan, a pair for vmotion/management and two spare for other lans. This works well and there are no issues. But bearing in mind the Veeam problems I looked at changing it. The management nic was on the same ip range as the production network in order that I reach the console. I changed this so it was on a separate lan. That way if failover happened then it wouldn’t impact production activity. The only other thing I changed was I reset the backup proxy (the veeam server itself) to use direct access again and REMOVED the tick to failover to network. This combination appears to have fixed it!

The next backup (a fresh backup – not a retry) was using the same server and repository. This time the lan graph simply flicked up a little but the iscsi graph shot right up. Processing went up to 35Mb (from 9 last time) and the job finished in ten minutes.

[teza8]

In terms of stats: yesterdays 90 minute job was
Source: 90%, Proxy: 13%, Network + target: 0%
processing rate 9Mb

Todays much fater job was:
Source: 95%, Proxy: 80%, Network: 14% target: 12%
Processing rate: 50Mb

Can you see anything from there?

[Vitaliy S.]

Hi Terry,

Actually I was proposing to try HotAdd mode and not Direct SAN, but anyway since you've found a way to improve your backup performance with direct SAN mode, try applying the recommendations that are given in this thread, should boost the performance a little: Improving direct-from-SAN backup speed with iSCSI SAN

Thanks!

[teza8]

I looked into this further at the weekend and tried changing the proxy method you suggested. I found that all backups and replications failed with errors. The error on all was “Unable to allocate processing resources. Error: No backup proxy is able to backup this VM. Check processing mode settings on proxies”. I then set this back to direct access mode and the backups worked. However, a replication of a 222 Gb server took over 2 ½ hours, 124Gb transferred, 27Mb processing rate and bottleneck of target. The target is was a lun on green (back up san).

I watched part of the replication run and noticed in task manager network activity window that the iscsi nic network activity was equal to the lan (management lan) nic’s activity. Does this seem right?

I’m hoping you can help with this but more importantly could you (as I’ve asked before) kindly look at the initial part of my thread and look at the layout of my vm network and what I’m using on it and quickly sketch out where things should be in a typical set up (using what I’ve got). I’m concerned I’m not utilising things to their best capacity and will continue to get things wrong until I fix the fundamentals. I’m happy to send you a more detailed sketch of how things look, IP and anything else you need.

[Vitaliy S.]

The error on all was “Unable to allocate processing resources. Error: No backup proxy is able to backup this VM. Check processing mode settings on proxies”.

Seems like you didn't configure your backup proxy VM properly to utilize HotAdd backup mode. Please take a look a this topic for system requirements of Virtual Appliance (HotAdd) backup mode: VMware : [FAQ] FREQUENTLY ASKED QUESTIONS

I’m hoping you can help with this but more importantly could you (as I’ve asked before) kindly look at the initial part of my thread and look at the layout of my vm network and what I’m using on it and quickly sketch out where things should be in a typical set up (using what I’ve got).

The sketch would definitely help here. Also what are the bottleneck stats? Still the same, source 90%?

[teza8]

I'd rather not post my network layout on here. can you supply me with an email address to send to?

[Vitaliy S.]

Just click on my nickname to see the email address in the account profile.