Very slow application testing in Virtual Lab

[Daveyd]

Some background:

Veeam 5.02 running on a VM (4 vcpu, 6GB RAM) residing on a EMC CX-4 fibre channel LUN. Backup target is an Exagrid appliance connected to the network via 1GB link.

I want to set up a simple, manual verification process for an application that uses a web and database server using Veeam's Virtual Lab. I created an Application Group consisting of the Web and DB server. I created the Virtual Lab correctly with a static mapping for the Web server. I created a job, chose the Application Group, chose the Virtual Lab and chose to leave the VMs powered on.

After starting the job, I was getting Failed jobs due to VMware tools heartbeat not being present. I upped the Application group time to 1800 seconds. Each VM then did finally start and it took 30 minutes for each VM to become accessible.

I am able to connect to the Web server from my local workstation using the masqueraded address. However, doing anything in the application is ungodly slow. Going from screen to screen from within the application takes around 30 seconds. Typing text in the application takes 7-8 seconds for each letter to appear. Logging into the Virtual Lab VMs and performaing tasks is a little quicker but not much.

I do understand that I am running off of a deduped backup file but it is not compressed and the backup resides on the Exagrid's "Landing Space", which is uncompressed and undepuded. Any idea where my bottleneck may be or how to troubleshoot? I have senior management staff that would like to be able to verify theses backups by doing what I am doing...loggin into the application from the backup file and pulling up information...hopefully a little faster than what I am experiencing now

[eric.siedentopf]

I am new to this but i have had similar problems when i didn't had a Domain Controller\DNS in the lab

[larry]

I do a AD server/DNS(W8R2) and a exchange server (W8R2) using OWA is it ran normal speed. I have this setup running every day. First I would look at what you are backing up to next look in VC and check the normal stats.

[Gostev]

Each VM then did finally start and it took 30 minutes for each VM to become accessible.

Well this is an issue, because start time should really be comparable to one in production, of course usually a bit slower depending on backup storage, but still comparable. In fact, one of our customers who did extensive testing reported that with backup residing on modern SAN, sandboxed VM boot time was faster than the one from production VM residing on older SAN.

[Daveyd]

Each VM then did finally start and it took 30 minutes for each VM to become accessible.

Well this is an issue, because start time should really be comparable to one in production, of course usually a bit slower depending on backup storage, but still comparable. In fact, one of our customers who did extensive testing reported that with backup residing on modern SAN, sandboxed VM boot time was faster than the one from production VM residing on older SAN.

I thought the startup time would be a heck of a lot quicker as well. I am backing up to an Exagrid appliance, which to my understanding, keeps the latest backups in its "landing zone" which are only deduped files from the Veeam job...Exagrid only dedupes and compresses the files when they go into Exagrid's retention space. I am not sure why they are so slow. I would think it would be compareable to normal startup time of the VM as well.

I was watching the VMs network traffic in Veeam monitor when I started the job and the Veeam backup VM receice rate averaged arounf 45MB/sec for the entire 30 mins while the 1st VM was trying to start

[Daveyd]

Any other incite?

[romwarrior]

I'll add that I see a similar thing here, with an Exagrid as well. VM's take (almost exactly) 30 minutes to power on for SureBackup jobs, but just a couple minutes for an instant recovery job. If it wasn't for the speed of the Instant Recovery job I would blame it on the Exagrid. I know there's a bit more going on for SureBackup jobs, but not that much. It seems like the SB job is waiting for something to indicate to it that the VM has completed the "Powering on" step, as that is the step that takes 30 minutes. Not until it decides that the VM is powered on does it proceed to the heartbeat test, which succeeds quickly. Not sure what tells it to proceed to that next step.

[Daveyd]

I'll add that I see a similar thing here, with an Exagrid as well. VM's take (almost exactly) 30 minutes to power on for SureBackup jobs, but just a couple minutes for an instant recovery job. If it wasn't for the speed of the Instant Recovery job I would blame it on the Exagrid. I know there's a bit more going on for SureBackup jobs, but not that much. It seems like the SB job is waiting for something to indicate to it that the VM has completed the "Powering on" step, as that is the step that takes 30 minutes. Not until it decides that the VM is powered on does it proceed to the heartbeat test, which succeeds quickly. Not sure what tells it to proceed to that next step.

Yep. Instant Recovery job is only a couple of minutes for me as well from the Exagrid as opposed to the 30 minute Surebackup jobs

[Gostev]

That's good news then, as you can see, vPower engine performance is not an issue. I would recommend opening a support case and have our staff investigate what is causing the delay with the logs files.

[mschlott]

[merged]

It seems to me that Surebackup jobs have a limited use. I thought I would use a surebackup job to verify backups of all my servers each night. Based on the time that it takes to run a surebackup job, that does not seem practical. In my testing, I see that it takes between 10 and 15 minutes for each server to start. If I have 300 VMs being backed up and want to test each of them, that is going to take 62 hours. If I divide that evenly across my 3 VEEAM servers, then that is 20 hours each for surebackup alone.

My Veeam server is running on a DL380 G7 with 144GB of RAM that is mostly sitting idle. It is configured with 6GB of RAM and 8CPUs. The backend storage is on a RAID5 of 1TB SATA drives. My virtual lab is running on the same physical server as my veeam server.

It seems like the logical fix for this situation is to run more Veeam servers and have less VMs backed up per veeam server. If I do this, my backup and surebackup jobs will complete faster, but it becomes harder to run a real vlab for ASR. Any vlab that I would run for the purpose of recovering data is going to consist of at a minimum a Domain Controller and the VM being recovered. So any Veeam server that backs up a SQL server, or Exchange server also needs to be backing up a DC. This means I may need 6DCs just so I can ensure restores work.

I have got to be missing something. Is it normal for a VM to take 15 minutes to boot in surebackup job? I am not running any application tests, only ping and vmware tools. Sometimes the vmware tools test fails, even though I can see the server is booted and vmware tools show OK in the vcenter console.

mike

[ian0x0r]

[merged]

Hi,

More of a question to gauge how long this should be taking and what other peoples experiences are like. I will give you my scenario here:

Veeam V5 5.0.2.230 running on physical hardware. Veeam backups are stored on an iSCSI mapped drive which is located on a Dell Equallogic PS4000E.

Application group consists of 1 2008 DC, 1 Exchange 2010 HT/CAS server and 2 Exchange 2010 Mailbox server. The start-up time for the lab is 50 minutes. I’ve tried this at numerous sites with different hardware specs and the result is usually the same. Am I doing something wrong or is this typical behaviour for this scenario?

Thanks,

Ian