SureBackup jobs shut down

[k00laid]

I've got a some SureBackup jobs setup to continue running after the testing so that we can do manual testing and reporting. After a while the jobs just shut themselves down. Is this normal behavior? Is there some place I can tell it not to do this?

[Alexey D.]

Hello Jim,

Did you tick "keep running" checkbox in the settings?

[Gostev]

Yep, behavior is normal as there is no point to keep VMs running after they have been verified. For advanced use cases like ones you mentioned (we call it "On-Demand Sandbox"), you can leave your VMs running with the setting Alexey mentioned. This, however, is not "ideal" way to achieve this, as this requires modifying your SureBackup configuration settings and thus may affect recovery verification jobs. Easier/proper way would be to use Universal AIR Wizard to request virtual lab for specific VM. This way, the VM will keep running for the requested lab duration no matter of the above checkbox setting. In addition, the wizard will also take care of routing automatically for you, so that you could access fenced-off environment over the network from whatever workstation you run Universal AIR Wizard on.

More detailed information on this is available in the sticky FAQ topic. Kindly please review it before creating new topics.

Thanks.

[k00laid]

Alexey, yes I have ticked the "keep running" checkbox.

Gostev, We are a financial company and have steep requirements for backup testing. What the end goal is to have staff run a backup verification where they access a lab with a series of servers all related to a specific application, run a report and get it out where it can be printed or imported into the Document Management system.

What I've lined out already is to create application groups for each of my different applications, tick the box to keep the job running after the automated testing is done and set a schedule and the e-mail of the staff member who is to run the manual test to be notified. User then RDP's to the masquerade address and runs management approved test, spits out a PDF file for output, connects to the masquerade address with Explorer and grabs the PDF file.

Where I'm having an issue is that like 2 hours after the job has sent the e-mail that the job was a success it tears itself down. I'm guessing there is an inactivity timer on the lab or something, but I can't seem to find the setting to either turn it off or, preferrably, set it to a higher number.

I guess in the terminology of your FAQ, which I did check before hand, what I need is an On Demand Sandbox. I just need one that doesn't kill itself so quickly.

[Gostev]

Jim, sounds like last question in this FAQ section is basically about your scenario. However, based on your description above, you are doing everything just right today! I am not sure what is causing shutdown in 2 hours though, there are no inactivity timers that I am aware off. The only thing that can cause shutdown is new backup cycle starting. If this is not the case, please send full logs files to our support, and they will should be able to tell what is causing the shutdown.

On a side note, I am curious what is preventing you to 100% automate the procedure? Sounds like the only manual procedure involves running some test utilities by hand, and grabbing output files they produce from a share. This sounds like something you should be able to script with custom SureBackup verification script?

Also, would you be able to have a call with us offline and provide some more details around your requirements, and how you accomplish them today with SureBackup? Looking to get just a little more details. Let me know and if this is possible, and I will email on your forum registration address to discuss this further. Of course, if you cannot provide us much details because of being financial institution, I understand that, no problems!

Thanks.

[k00laid]

I can do the phone call, looking forward to hearing from you.

[k00laid]

BTW, I have this as the last line of my job, just short of 2 hours after the lab successfully comes up.


3/21/2011 10:39:33 AM Fail
Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding.

[Gostev]

Yep, this error looks very much like an issue, not a feature

[Alexey D.]

Jim, this error may be related to unhealthy ESX host where your virtual lab resides.

Some days in my lab, I was also getting the similar error. Having checked host's event log in vSphere client I found "host communication error" which kept randomly happening during night hours. This made B&R console think it lost connection to lab's virtual appliance VM. After that the lab was shut down.

Replacing bad hardware on host solved that "host communication error" and now my labs never go down.
Maybe it's also your case.

[bunger]

[merged]

I have successfully create a virtual lab environment in which I am testing an Exchange migration from 2003 to 2010. Everything works great, except....

If I leave the environment running over night, the lab shuts down and I lose all of my work. I suspect it is related the nightly backup running, but am not 100% certain.

So.... is there a method to create a "persistent" virtual lab in that it doesn't shutdown or if it does, the machine states don't get reset?

Thanks in advance,
Bill

[bunger]

Is there a fix for this?

My virtual lab consists of a production Win2k3 server that gets backed up nightly and a testing Win2k8 server that has only gotten backed up once.

I changed the job associated with the virtual lab to use a day-older backup for the WIn2k3 server ( one that will be sure to be around for at least 7 more days ) and restarted the lab yesterday. Did a bunch of work through-out the day yesterday, came in this morning, and have encountered similar issue: instead of the lab shutting down, the Win2k3 server ( the one that had a backup job run against its production sibling last night ) was completely locked up. The only option was to power off that virtual server. Unfortunately, trying to start it up throws errors:

Reason: 5 (Input/output error).
Cannot open the disk '/vmfs/volumes/cdcd36fe-2d746a42/myServer_e4f51dec42634535858633fe4287866e/myServer.vmdk' or one of the snapshot disks it depends on.

This leads me to believe that once again the backup trumped my environment. Quite frustrating....

[Gostev]

Cannot check on fix as OP never posted his support case ID.

[bunger]

Independent of the original poster's case, I am wondering if there is a "proper" method for running a persistent virtual lab? If I use a backup job that gets overwritten, the virtual lab shuts down. If I use a backup job that doesn't get overwritten ( the backup is setup as Reversed Incremental, the VM in the virtual lab locks up and then can't find it's disk files.

Any help would be very much appreciated....

[Gostev]

Well, you could always deploy a separate Veeam server, copy backup files to it, import them there, and run required SureBackup jobs permanently.

I will also check if v6 fixes this behavior with backup jobs in incremental mode (were backup files do not get overwritten). I assume you meant regular incremental backup mode for when backup files are NOT overwritten.

[kjc3303]

I have a Surebackup that is failing on one VM with the below error


18/03/2013 09:13:41 Fail Error: Cannot open the disk '/vmfs/volumes/970937f8-6c9ec05c/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx_2.vmdk' or one of the snapshot disks it depends on.

I have checked and the VM is not running on a snapshot.

So far I have tried
Made sure there are no snapshots checked the
Active full to recreated the VBK (I am running in Rev incremental)
A brand new back up job

Instant recovery works perfectly against the backup and I can start the VM up in an isolated network with no issues

Anyone come across this?

Kev

[kjc3303]

Update

Veeam support have done a sterling job of identifying my issue after a bit of head scratching - thanks Yuri

My issue was that my vLab was created on a Datastore with a block size that doesn't support the largest disk on the Surebackup job
I have recreated the vLab on a new DS and everything is fine.

I'm sure Veeam can come back with a more detailed description of the issue but here is my explanation

It is to do with the disk redirects which is why the Instant recovery was working fine (with no disk redirects). With disk redirects to the same DS that the vLab is on - Fail, With disk redirects to the same DS that the VM is on - Success.

I guess this is going to be less of an issue with VMFS-5

Kev

[kjc3303]

From Veeam support

I am very happy to hear that SureBackup is finally working for you. SureBackup is actually redirecting the redo logs to the same datastore where the vLab is located, while Instant Recovery by default doesn't.

This is an expected behaviour, and the obvious way to make it work is to deploy vLab appliance to a datastore with the largest blocksize possible. I'll try to push some buttons to include this information in the next release of Best Practices.

[Gostev]

I guess this is going to be less of an issue with VMFS-5

Correct. Non-issue, even