Feature Request - Advanced Surebackup Boot Order

[Fizzer]

Hi There, sorry if this has been posted before.
I have an idea for surebackup that would make my farm restores a lot quicker.
I would like to be able to customise the behaviour of the boot order in surebackup. It would be good to be able to boot some of the VM's in the application group at the same time - especially as I have 2 x 6 server sharepoint farms.
An extra column in the Application Group properties could contain a number which refers to the boot order:

VM's with a lower number boot first
VMs with the same number boot at the same time

Local DC: 1
SQL server: 2
sharepointA: 3
sharepointB: 3
sharepointC: 3
sharepointD: 3

This way, all of my sharepoint servers boot when the domain and SQL services are available, reducing the job time significantly (currently 1.5 hours). It's also flexible enough for anyone with complex environments.

Cheers
Andy

[Gostev]

You can do this today by changing the boot delay to zero for the SharePoint servers?

[Fizzer]

Thanks for the prompt reply Gostev
Do you mean 'Maximum Allowed Boot time'? Or is there another parameter that I'm missing?
Thanks
Andy

[foggy]

Actually, you cannot set this parameter to zero - it is bounded from below by 60 seconds to allow the system to stabilize its IP and perform other checks (otherwise, the task will fail).

[Fizzer]

the behaviour I observe is that if I set 'maximum allowed boot time' to a high number the VM will remain 'powering on' until the VM has fully booted. Typically this is anywhere between 8 and 14 minutes in my environment.
If I set the 'maximum allowed boot time' to anything less than the actual time it takes to boot the VM, and the 'Powering on' phase reaches the 'maximum allowed booot time' (in seconds) - I receive a 'cannot detect VM starting because of timeout' error.
This is even with 'VMware Tools Heartbeat is present' and 'VM responds to ping on any network adaptor' unchecked in the Boot verification options.

I cannot see a way to make surebackup start a VM without first confirming that the previous VM has successfully booted (which always take more than 8 minutes per VM)

[Fizzer]

I think that this explains the above better than I can:

24/05/2012 16:13:10 Waiting for OS to boot for up to 60 seconds (stable IP algorithm)...
24/05/2012 16:13:10 Note: Will proceed to the next step at 24/05/2012 16:14:10 or earlier
24/05/2012 16:14:12 Fail Results: Cannot detect VM starting because of timeout
24/05/2012 16:14:12 Fail Error: Results: OS did not boot in the allotted time
24/05/2012 16:14:20 Powering off

[foggy]

What you are seeing is just exactly as it should work. Anton considered that setting this parameter to zero would force VMs to boot simultaneously but that is not the case, as I explained above. Aside from the fact that you cannot set it to zero (UI simply does not allow that), setting it too low results in the task failure as it requires the VM to boot up successfully to continue and the low allowed boot time value prevents this from happening. This parameter cannot affect the app group start time (unless you have no VMware Tools installed in the guest and this parameter acts as the only criteria of successful VM startup).

[Fizzer]

"it requires the VM to boot up successfully "
this is my point exactly. I would like to be able to tell Veeam 'skip checking this VM please, just carry on starting the next 3 virtual machines'

Or, if it must check all VMs (because that's what surebackup is designed for) could it not check 3 or 4 (or more) simultaniously?

I know surebackup is designed to check that your VM backup is good, however the nice 'side effect' is that it also allows me to boot all my Sharepoint virtual machines in an isolated environment so I can perform an item level restore.

The new feature I would like, is to be able to turn off the Virtual machine checking for some VM's during the surebackup startup, thus speeding things up.

[Gostev]

The new feature I would like, is to be able to turn off the Virtual machine checking for some VM's during the surebackup startup, thus speeding things up.

That should be possible today, just clear the corresponding check boxes in the application group settings on those VMs (heartbeat, ping and script tests).

[Fizzer]

the logs I posted above were with heartbeat and ping unchecked and with no scripts.
Should I log a support call?

[Gostev]

Yes.

[foggy]

The way it is currently implemented is that even if all checks are disabled for the VM with VMware Tools installed, the IP stabilization algorithm is still executed. This means that if the job cannot get the VM IP within the period specified in the Maximum Allowed Boot Time setting, it fails with the "Cannot detect VM starting..." error. However, if VMware Tools are not installed, this error is not logged.

[rbrambley]

I know surebackup is designed to check that your VM backup is good, however the nice 'side effect' is that it also allows me to boot all my Sharepoint virtual machines in an isolated environment so I can perform an item level restore.

As an alternative try this:

• 1. use the vSphere client to start your Veeam vLab proxy VM manually (power it on)
  2. do Instant VM Restores (IVMR) of all of your VMs in your Sharepoint application group but DO NOT connect to network
  3. edit each VM's vNIC to use the appropriate vLAB port group
  4. do your restore

Of course you will have to do your DC first and move it into the vLAB, then your SQL, but then you should be able to do the Sharepoint IVMRs at the same time.

Since you are using IVMRs you don't have to wait for the Surebackup "stabilization", and I would imagine the whole process will be significantly less the 1.5 hrs

For that matter you could reduce your App Group to just the DC and SQL, and then do IVMRs of the Sharepoint VMs and manually move them into the vLAB. I think that will add a little more time to the process, but maybe eliminate some service errors during the startups?

The new feature I would like, is to be able to turn off the Virtual machine checking for some VM's during the surebackup startup, thus speeding things up.

I agree. I've talked to many in the field that have said that Surebackup takes too long and that an admin should be able to bypass the "stabilization" and timeouts should not kill the whole job. Essentially, doing IVMRs in a vLAB where you could prioritize VM start orders as you described earlier in this thread would be a great new alternative.

[rbrambley]

Here's another alternative to try with they way Surebackup works now:

• 1. create (or reduce your current) application group to just the DC and SQL VMs
  2. create a new Surebackup job
  3. choose the reduced App Group on the Application Group page of the wizard
  4. On the Backup Jobs screen (the next one) check the box to link backups and pick a backup job with your Sharepoint VMs
  5. Be sure to change the "process simultaneously up to" field at the bottom to 4 VMs (in your case)
  6. Finish and try the job

The Backup Jobs screen does not use the "stabilization" by default (click the advanced button if you want to) like the Application Group settings.

something to consider with this method - The Backup jobs screen cannot pick only a few VMs out of a large job, so you might want to create a new backup job for your Sharepoint VMs by themselves to test this. Or, if you already have a job for just your 4 Sharepoint VMs then you are all set.

Maybe this would be another great new option for the future! - be able to pick individual VMs out of a larger backup job on the Backup Jobs screen and apply the priorities as described earlier.

Let me know if you try these and how they go, and your thoughts.

[Cokovic]

Here's another alternative to try with they way Surebackup works now:

• 1. create (or reduce your current) application group to just the DC and SQL VMs
  2. create a new Surebackup job
  3. choose the reduced App Group on the Application Group page of the wizard
  4. On the Backup Jobs screen (the next one) check the box to link backups and pick a backup job with your Sharepoint VMs
  5. Be sure to change the "process simultaneously up to" field at the bottom to 4 VMs (in your case)
  6. Finish and try the job

The Backup Jobs screen does not use the "stabilization" by default (click the advanced button if you want to) like the Application Group settings.

...

But the VMs in the linked jobs would be immediatly shut down after they are verified regardless the checkbox to keep the application group up and running

[rbrambley]

Haris,

You are right. Scratch that second idea then. Looks like manual IVMRs is the only alternative today.

It would be nice to have an option to keep them running in some future vNext!

[Cokovic]

It would be nice to have an option to keep them running in some future vNext!

Indeed. A checkbox for linked jobs equal to the application group to keep them up and running would perfectly match here.

[Fizzer]

Hi all, I'm creating a new surebackup job and was reminded of how inconvenient it is not to be able to turn off all boot checking. Any thoughts on this?
I can't imagine it would be that hard to add this and it would make many people's lives a LOT easier.

[foggy]

Actually, the reason for this is that the same algorithms are used in AIR where we need to detect the moment in time when the VM is booted and gets valid IP to connect to...

[baber]

how the problem solved ??????