Install Considerations VM Cluster

[Soap]

Guys,

I am hoping some of you can help me understand the preferred method of deployment for my particular environment.

2 - 6 node Virtual Clusters running ESX 3.5
1 - 3 node Virtual Cluster running ESX 4.1

I have multiple 500GB to 1TB SAN LUNS shared between the nodes in there respective clusters and about 40 to 50 VM's running on each node. All nodes of the ESX 3.5 clusters are IBM x3850 M2's with 16 cores and 128GB RAM. The new 4.1 cluster nodes are IBM x3950 M2's with 24 cores and 128GB RAM. As a side note the plan is to upgrade/migrate the ESX 3.5 nodes over to ESX 4.1. My questions are how should I go about implementing my Veeam B&R solution? I was hoping to wait for v5 to come out as I think it may be this week or next from what I have read? Do I need to create Veeam backup servers for each node of my cluster? If I do that and create seperate jobs for each node then will I run into issues with jobs trying to access shared LUNS at the same time? What am I not taking into account here as well? Thanks in advance guys for the help.

[Gostev]

Hello, what kind of storage each cluster is using? This is most important information required before we can make any recommendation.

[Soap]

Hello, what kind of storage each cluster is using? This is most important information required before we can make any recommendation.

A combination of IBM DS8300 and XIV.

[Gostev]

Is it single IBM DS8300 SAN, and single XIV (what is it, by the way)?

Can you provide more details on how each cluster is connected to each storage?
E.g., each cluster is connected to IBM DS8300 SAN, but one one cluster is connected to XIV.

[Soap]

Is it single IBM DS8300 SAN, and single XIV (what is it, by the way)?
Can you provide more details on how each cluster is connected to each storage?

Multiple for both. IBM XIV (http://www-03.ibm.com/systems/storage/news/center/xiv/) All storage is fiber connected.

[Gostev]

OK, this is best possible setup and it gives you great flexibility. You can choose between physical Veeam Backup server (connected directly into SAN fabric) and use "Direct SAN access" mode for LAN-free backups which does not affect your hosts. You can also deploy virtual Veeam Backup server, and use "Virtual Appliance" mode in this mode our server access VM data through ESX I/O stack. This is still extremely fact, however Veeam server will eat CPU cycles from ESX you will place it on. Because you have all-to-all connection, you can place virtual Veeam server or any host - so if you have one which has lower load, you can place Veeam server there.

Hope this helps!

[Soap]

OK, this is best possible setup and it gives you great flexibility. You can choose between physical Veeam Backup server (connected directly into SAN fabric) and use "Direct SAN access" mode for LAN-free backups which does not affect your hosts. You can also deploy virtual Veeam Backup server, and use "Virtual Appliance" mode in this mode our server access VM data through ESX I/O stack. This is still extremely fact, however Veeam server will eat CPU cycles from ESX you will place it on. Because you have all-to-all connection, you can place virtual Veeam server or any host - so if you have one which has lower load, you can place Veeam server there.

Hope this helps!

This gives me some things to consider then. While I do like the physical option for avoiding impact to the ESX hosts, getting physical hardware approved and ordered would be a hassle if not an option. If I go the virtual route how many backup servers would be recommended? One per ESX host? If so that would mean about 40 to 50 VM's being backed up per 'JOB'.

[Gostev]

You can deploy virtual now, and then switch to physical. Physical backup server is typically a cheaper option. There are no miracles, and if you add a few bad VMs on your ESX hosts, you will reduce the overall capacity of your cloud, meaning that as your environment grows, you will have to add new ESX host sooner, rather than later. And typical ESX host is tons more expensive than regular server, plus there is VMware license too.

General rule of thumb for ESX4 is that you start from 1 Veeam server per 5TB of source data assuming 8 hours backup window, but this depends on VM quantity/sizes and workload they run (aside of hardware/storage/network of course). Deploy your first server like that and run for a few days, and see how this number needs to be adjusted for your specific environment.

For ESX 3.5, I recommend starting from 1TB per server. Having no changed block tracking really affects incremental backup performace on ESX 3.5, because our engine has to read the whole source VM to determine changes since last backup.

[Soap]

Thank you for your helpfull replies, I will start putting some ideas down on paper. I was really hoping to wait until veeam ver5 comes out if it is within the next week or so before moving on this if possible. I will update this thread as I move forward. Thank you again.

[Soap]

General rule of thumb for ESX4 is that you start from 1 Veeam server per 5TB of source data assuming 8 hours backup window, but this depends on VM quantity/sizes and workload they run (aside of hardware/storage/network of course). Deploy your first server like that and run for a few days, and see how this number needs to be adjusted for your specific environment.

For ESX 3.5, I recommend starting from 1TB per server. Having no changed block tracking really affects incremental backup performace on ESX 3.5, because our engine has to read the whole source VM to determine changes since last backup.

Do these sizing recommendations hold true no matter if I choose to go the virtual appliance route or the SAN mode? Is there any documentation showing measured performance between the 2 modes available? Thank you

[Gostev]

There is little to no difference in performance between these modes. Both modes are doing direct SAN access essentially, with Virtual Appliance mode using ESX iSCSI initiator, and SAN mode using whatever HBA or initiator you have on your server.

Most of all, job performance depends on backup server and backup target performance. Fast CPU and fast RAM = win. I’ve seen 4 vCPU VM on good ESX hardware outperform 3 year old quad core physical computer, and people making conclusions that Virtual Appliance mode is better. I’ve seen people running Virtual Appliance mode in 2 vCPU VM, then going to good physical server and direct SAN access mode, and making opposite conclusions

[Soap]

I just wanted to post results of my first job that I ran using Virtual Appliance Mode. I want to note that this is just a smidge of what this environment will be once we migrate about 800 systems over from ESX 3.5 into Vshpere 4.1 Enjoy...

INCREMENTAL PASS
8 of 8 VMs processed (0 failed, 0 warnings)
Total size of VMs to backup: 355.02 GB
Processed size: 355.02 GB
Processing rate: 268 MB/s
Start time: 9/30/2010 12:59:29 PM
End time: 9/30/2010 1:22:07 PM
Duration: 0:22:37


FIRST FULL BACKUP PASS
8 of 8 VMs processed (0 failed, 0 warnings)
Total size of VMs to backup: 355.02 GB
Processed size: 355.02 GB
Processing rate: 132 MB/s
Start time: 9/30/2010 11:04:57 AM
End time: 9/30/2010 11:50:45 AM
Duration: 0:45:47

[Gostev]

Sounds like you have very decent hardware, as well as production and backup storage oh, by the way, what is your backup target?

[Soap]

Sounds like you have very decent hardware, as well as production and backup storage oh, by the way, what is your backup target?

If I am not mistaken I believe it is IBM DS4500 storage, nothing fancy.

[Soap]

So is 5TB the tipping point in performance per veeam backup server being run against VSphere 4.1? If so will that change with veeam v5?

[Gostev]

How did you make IBM DS4500 available to Veeam Backup VM? As a CIFS share, or did you write to NTFS LUN via software iSCSI initiator?

5TB is not the tipping point, but the first point to start your deployment planning for 8 hours backup window. Backup performance can vary up to 10x times between environments due to hardware and VM workloads, so there is no universal recipe. My recommendation is to simply use 5TB as initial planning point. Looking at your results, you should be able to go much higher than that.

With v5 and new incremental backup mode, I expect to see this point increased to at least 10TB per server on 8 hour window, because incremental backup is significantly faster now. This comes at the cost of periodic synthetic full backups, but since those are typically scheduled for weekend, they do not have this 8 hour window to deal with.

[Soap]

Yes, a CIFS share. Thank you again for the quick reply and help. I look forward to getting my hands on v5 in a few weeks.

[Soap]

Okay, After some time with veeam in our environment I have mixed feelings about the results. It appears that our current setup for ESX 4.1 runs very well with all jobs completing in about an hour. Time will tell how performance goes as we begin migrating our VM's from 3.5 and increase the load and storage volume in ESX 4.1. Now for the ESX 3.5 environment performance. It sucks! Really no other way to say it and I am trying to get migrated over to ESX 4.1 as quickly as possible but I need to get this running better in the meantime. I have posted some performance information below in hopes someone may be able to offer some suggestions on what I should do differently.

(This is the new VSphere environment running like a champ!, I currently have 1 B&R server running 3 seperate jobs for each ESX NODE.)
VEEAMBACKUP1 - ESX 4.1
(ESX NODE A)
12 of 12 VMs processed (0 failed, 0 warnings)
Total size of VMs to backup: 515.02 GB
Processed size: 515.02 GB
Processing rate: 132 MB/s
Start time: 10/10/2010 9:00:36 PM
End time: 10/10/2010 10:07:00 PM
Duration: 1:06:24

(ESX NODE B)
10 of 10 VMs processed (0 failed, 0 warnings)
Total size of VMs to backup: 445.01 GB
Processed size: 445.01 GB
Processing rate: 162 MB/s
Start time: 10/10/2010 11:00:14 PM
End time: 10/10/2010 11:47:13 PM
Duration: 0:46:59

(ESX NODE C)
11 of 11 VMs processed (0 failed, 0 warnings)
Total size of VMs to backup: 425.00 GB
Processed size: 425.00 GB
Processing rate: 93 MB/s
Start time: 10/11/2010 1:00:15 AM
End time: 10/11/2010 2:17:53 AM
Duration: 1:17:37

(This is 1 of our ESX 3.5 clusters that happens to be running Update 1, I have 6 B&R servers each running on an ESX NODE to balance performance.)

VEEAMBACKUP2 - ESX 3.5
(ESX NODE D)
48 of 48 VMs processed (3 failed, 5 warnings)
Total size of VMs to backup: 1.30 TB
Processed size: 1.18 TB
Processing rate: 13 MB/s
Start time: 10/8/2010 7:00:30 PM
End time: 10/9/2010 9:50:59 PM
Duration: 26:50:28

VEEAMBACKUP3 - ESX 3.5
(ESX NODE E)
36 of 36 VMs processed (2 failed, 5 warnings)
Total size of VMs to backup: 1.11 TB
Processed size: 1.09 TB
Processing rate: 9 MB/s
Start time: 10/8/2010 7:00:28 PM
End time: 10/10/2010 5:07:38 AM
Duration: 34:07:09

VEEAMBACKUP4 - ESX 3.5
(ESX NODE F)
52 of 52 VMs processed (0 failed, 10 warnings)
Total size of VMs to backup: 1.33 TB
Processed size: 1.33 TB
Processing rate: 14 MB/s
Start time: 10/6/2010 7:00:29 PM
End time: 10/7/2010 10:35:12 PM
Duration: 27:34:42

VEEAMBACKUP5 - ESX 3.5
(ESX NODE G)
29 of 29 VMs processed (3 failed, 15 warnings)
Total size of VMs to backup: 1.15 TB
Processed size: 824.00 GB
Processing rate: 15 MB/s
Start time: 10/9/2010 7:00:37 PM
End time: 10/10/2010 10:40:58 AM
Duration: 15:40:21

VEEAMBACKUP6 - ESX 3.5
(ESX NODE H)
17 of 17 VMs processed (3 failed, 4 warnings)
Total size of VMs to backup: 747.02 GB
Processed size: 481.02 GB
Processing rate: 19 MB/s
Start time: 10/10/2010 7:00:24 PM
End time: 10/11/2010 2:06:22 AM
Duration: 7:05:58

VEEAMBACKUP7 - ESX 3.5
(ESX NODE I)
43 of 43 VMs processed (0 failed, 8 warnings)
Total size of VMs to backup: 1.48 TB
Processed size: 1.48 TB
Processing rate: 13 MB/s
Start time: 10/8/2010 7:00:18 PM
End time: 10/10/2010 3:05:21 AM
Duration: 32:05:03

What causes the processing rate to change? This fluctuates on my ESX 3.5 NODES during each backup run and can add an additional 10 or 12 hours to a job in some cases. I am using B&R 4.1.2 on all servers and running in VA mode with default compression settings. Some VM's the processing rate is much higher and some are lower which also has me scratching my head. I even have cases where 1 particular VM will fail causing the entire job to stop in a failed state. I have a few other issues as you can see from the warnings and failures shown above but I really want to know more about what effects the processing rate and if I should be implementing this a different way or not. Thanks in advance all.

[Gostev]

Check CPU load on Veeam Backup server with slow performance while backup is running - might be a bottleneck there. Are Veeam servers configured with 4 vCPUs? If CPU does not look to be the issue, may be better to open support case for investigations. Thanks!

[Soap]

Check CPU load on Veeam Backup server with slow performance while backup is running - might be a bottleneck there. Are Veeam servers configured with 4 vCPUs? If CPU does not look to be the issue, may be better to open support case for investigations. Thanks!

So CPU load would dictate the process rate? All 4 vCPU's stay pretty high on all of my boxes, this includes the one setup for ESX 4.1 which is very fast.

[Soap]

I opened a support case for this....