Not enough time to backup everything. Design help please!

[jzemaitis]

I'm evaluating Veeam and have a quote to licenses for my 5 ESX 3.5 hosts (10 sockets total). Does it matter if I add a 2nd Veeam backup server? Will the license cover that for the same ESX servers? Here is my situation...

I have two HP Blade Centers with 5 BL460 's blades between the two. Each blade is a quad core 3.0, 32 GB ram, dual 4 Gbps HBA and 4 1Gbps Nics. The VM's are stored on an HP EVA with 15k FC drives connected to each ESX server.

My backup server is a Dell 2850 w/ 2 GB Ram, 2Gpbs HBA, 2x1 Gpbs Nic, 2x2.8 Ghz proc's. I am connceting to a pretty new HP MSA that has 7200 rpm drives, raid 5 and is 6.8 TB formated. (I tried it in a Raid 0 and the speed was the same).

I'm using VCB and averaging about 30mpbs throughout my entire backup. Some as slow as 10 and upto 70. But over all 30. I'm trying to backup up 45 servers of varying sizes and would like to do so Mon-Fri. From what I've seen, if I run two jobs at once the speed is just cut in half for each, so I'm running each backup in a single job, so only one VM is being backed up at once. I''ve been adding servers slowly to my backup job and now I'm exceeding 24 hours in my job! Probably close to 36 hours at this point. These are my options (I think...)

Backup half over VCB and half over LAN and run the jobs at the same time.
Don't backup all servers each night. Split things up some how. Half one night, half the other, etc...
Upgrade my HBA to a 4 Gbps and hope it speeds things up.
Add a 2nd Veeam backup server (HBA 2 Gbps) (I have extra hardware so this would be pretty much free).

Another question... My HBA in my backup server is a dual port... Can VCB/Veeam leverage two Fibre connections and speed things up for me? I only have one connected now.

Help. Thanks!!
Joe

[Gostev]

Hello Joe, yes you can have multiple Veeam Backup servers and spread the backup jobs among them, this is what many of our larger customers are doing right now to be able to fit their backup window. It would be best to request and use separate licenses for each server though, otherwise it would be hard for your to track if you are in compliance with licenses.

VCB has very limited support for multipathing, and best results can be usually achieved when multipathing is disabled completely.

Thanks!

[jzemaitis]

I read about the VCB speeds being helped when multipathing is disabled. I wasn't sure if that logic applied to me or not. My storage is on our FC network for our VM's, but they're being backed up through my Veeam/VCB server and being stored on direct attached SCSI storage. I'm running a single port hba, obviously connected to just one switch. But I know the VM Lun's on our FC storage have 4 paths to each of our ESX servers, which have dual port HBA's.

Does my Veaam backup server have more than one path to the storage? How can I tell via Windows?

Not sure how I would split the licenses. I have 5 ESX servers connected to VC in a cluster. So all the VM's could be on any ESX server at any time via DRS. It sounds like I could use the same license on more than one server and maybe split the jobs based on VM LUN assignment to eliminate conflict.

Does the split VCB / Network backup make sense? It seems distrubing that I'm able to get on average better performance via my IP network. Hm. Would upgrading from 2 gbps to 4 gbps increase the speed? It seems dog slow when I know the system is not bottle necking with anything else. Is it VCB? Is it a pathing issue? How do I configure that on the Windows side?

Also, I'd rather not have two veeam servers. I have a massive amount of storage but I'm only able to connect it to one backup server. The other server I could only attach about 1.8 TB and it's old unsupported hardware. Ugh.

[Gostev]

Yes, upgrading to 4G HBA should definitely help increasing VCB processing speed...

[jzemaitis]

Something with Veeam is my issue. I made a call directly to Vmware as I have unlimited support. A good thing to have!!! We did several tests.

Here is what happened... I create a new LUN on my EVA storage unit that has the VMFS LUNs. Presented that to my Backup server. I mounted that storage as the servers E drive. I created a 10 GB Bin file on the EVA storage and kicked off a simple copy via the command line and also imployed "timethis.exe" from the Win2k resource kit to see how long the copy would take. The 10 GB copy took 1:56 which equals 170 mb/s. Not so bad. Much much much faster than the 17 mb/s I'm getting on some VM's!

Test 1:

Did straight VCB backup via the command line of one of my servers (35 GB 1 disk). The backup took about 5:50 seconds. That equals about 80-100 mb/s when figuring the time to start and end the snapshot. There is no speed listed when running VCB. I also noticed via task manager that the Proc's never went above 5% during the backup.

The VMware engineer had me create a new Veeam Job... I chose the VCB/SAN option, no compression, selected the same 35 GB server. Kicked off the backup. The backup look more than 12 minutes to backup and ran at an average speed of 58 mb/sec. I also noticed that the system was running at 60-65% CPU usage across 2 physical procs and the hyper threads. It was much faster then the backup I got last night at 8pm (on Sunday!). Then it ran at 23 mb/sec. I don't get that. No other backups or processes were running at that hour in my environment.

I ran another backup using Optimal compression and all other settings the same. The backup ran for roughlyb the same time give or take 20 seconds at similiar speeds.I also noticed that the system was running at 70-75% CPU usage across 2 physical procs and the hyper threads.

All backups were full backup / first backups from new jobs. I have a massive amount of storage (6.8 TB) for my disk staging backup server and would like to pull the data in faster to utilize it. Help! I'm calling Veeam tomorrow and if they can't speed things up this product will not work in my environment.

[Gostev]

Sounds like when testing VCB performance, you did not take vcbmounter command from Veeam Backup logs, so this comparison is not correct. Because If you just run vcbmounter without any keys, it will not fetch bit-identical copy of source VM files. Veeam Backup configures vcbmounter to fetch bit-identical copy, this is required to do things like file-level restores from image, calculate incremental changes for subsequent passes, and leverage synthetic backup.

Could you please perform VCB speed test using vcbmounter command from Veeam Backup logs? You need to take the command line, substitute target folder with GUID with real folder (your backup destination) and supply user name and password instead of asterisks.

[jzemaitis]

It looks like my problem was with the -M1 switch. Using the -M1 switch with VCB does a backup that doesn't sparse out the files. Putting in the M1 does a complete bit level backup of the VMDK just like Veeam does. Veeam seems to be faster doing it this way! I might end up going with Veeam after all. Has anyone here used vRanger from Vizioncore? Going through my comparisons but Veeam seems to be the most easy to use product so far.

[Gostev]

I don't think you will find many vRanger users on this forum

Without going into much details, vRanger does not have any of our main features and benefits listed on Veeam Backup web-page, except of VCB integration (poorly architected). I have also created detailed technical comparison recently, with thorough explanations for every bullet, and if you are interested, I can send one to you. There are about 3 pages worth of features that do make difference.

[jzemaitis]

Gostev, please send it over. Thanks. I was also told today by my manager that he spoke to Symantec and he wants me to eval netbackup for vmware. They claim to have a agent that runs on the esx servers. Maybe we could better better speed from it if it isn't VCB'ish?

I'm still leaning towards Veeam personally. The price is good and the interface is super intuitive to nagaviate. vRanger seems overly complicated and seems to have the same speed. That was the only thing I was concerned with when testing vranger. But the limit looks like it's VCB after all.

[Gostev]

Joe, we also have the mode available with agent running on the ESX server (our network backup mode leverages agent for "fat" ESX servers). Unlike Symantec, in this mode we additionally do traffic compression and empty block removal, so we are faster in this mode.

Just FYI we have 3 modes: VCB, service console agent based, and agentless (this one allows us to support ESXi and ESXi free without VCB requirement - no other vendor does that at the moment). But I would recommend using VCB SAN mode whenever possible, since it provides for LAN free backup that does not put any load on your ESX server or network. And it is typically faster on FC4 than what you can get with other modes, including ESX agent.

Main issue with Symantec is that it comes from physical world, so does not have many required virtualization-specific features. And it is extremely expensive too. Just compare us head to head and you will see how much we are better.

I will send you the document on vRanger to your forum registration email tomorrow when I get into the office.

[jzemaitis]

I'm curious. I decided to split my backup jobs out based on Lun / VM structure. So, all the VM's on VMFS1 are in one backup Job, VMFS2 are in another etc... Usually between 4-8 machines. I'm running three jobs MWF and three jobs on TuThSa. I'm kicking off all my non-critical and little change servers on Sunday at 2am. Also with backup jobs based on Lun / VM's. I'll be kicking off all 6 jobs at that time.

I read some place that the best practice is to not try to backup any two VM's that are sharing a Lun. I think if I stack my backup jobs I'll get better performance. Before I just created one big backup job with all my VM's. Maybe that wasn't the right idea. We'll see! I'll experiment and report back.

[Gostev]

Yes, running parallel jobs is good idea as long as they are not hitting the same LUNs (that would cause snapshot timeouts/reservations problems), and not sharing the same backup storage (backup storage speed often becomes bottleneck even in case of single VCB SAN job).