Backups now REBOOT every target VM

[HMwkr]

Veeam v8 has been trouble-free for months.

Started upgrading from v8 to v9. First it says "Upgrade Enterprise Manager". This failed with a generic error message - opened case 01647842.

Then I found my existing v8 backups were failing, of multiple 2012R2 and 2008R2 VMs. Rebooted everything - no joy. Bunch of recent MS updates to deploy - did so. No change.

It now being mid evening ie out of hours, and not expecting a further response from Veeam till morning, based on past interactions with Veeam software and Veeam support, I decided to simplify things - uninstall VeeamONE v9 which I had installed some days previously (all backups had run OK thereafter, but still ...); and remove Enterprise Manager v8 which I had installed moons ago and never used. Ripped all that out, no problems. Rebooted. Ran v9 upgrade which this time upgraded B&R without asking to upgrade Enterprise Manager first. Then installed Ent Mgr v9. Ran Ent Mgr, seemed fine. Ran B&R, told me it needed to update VMs etc - which it did without error.

So that looked like progress. Confusion resolved, I guessed.

Wrong. I ran a backup - boom! All VMs being backed *rebooted* - one DC on 2012R2 and one Exchange on 2008R2 on one host, plus the Veeam backup server VM itself on 2012R2 itself on a different host. Argh!

Set up new backup job of source VM which hadn't been touched by Veeam before - ran Active Full - boom! - source VM rebooted.

I suspect Veeam Tech's advice will be "rip it up and start again" but I decided to post here meantime, in case (a) anyone has already come across a similar scenario or (b) anyone comes across upgrade failure in future, and maybe this message/thread will suggest to them that they do *not* plough on

Regards
W

[HMwkr]

The solution seems to be ...

wait for it ...

Yup.
- what?
Wait. That's all. Wait.

When Syed@Veeam dialled in an hour later (now that's what I call service) each job ran without incident.

My hunches are:

a) initial failure to upgrade EntMgr v8 to v9 was a hangover from initial v8 installation problems a year ago; probably resolved simply by uninstalling v8 then installing v9

b) backup reboots were related to the after-effects of Microsoft updates - maybe TrustedInstaller was still doing its thing.

Lessons learned:

1: pat self on back for starting upgrade on Monday evening; missing an evening's backup or two is unpleasant but not critical, so I had until Friday to resolve the problems, taking the pressure off

2: do more defensive tidying in advance next time - uninstall applications rather than upgrade in place, at least for those which I haven't customised (EntMgr, VeeamONE) and are non-critical

3: even if applying MS updates seems like a plausible tactic mid-problem-saga, leave updates for an hour to settle in before doing anything. Anything.

4: Veeam tech rocks. That's the second time I've had a callback after midnight, within an hour of raising a case.

Hope this helps someone in future.

Regards
W

[Vitaliy S.]

Hello,

Thank you for sharing your story and glad that our tech engineers were able to resolve that quickly. On a side note, Veeam cannot reboot source VMs during backup jobs, so in order to dig deeper I would investigate Windows event log at the Hyper-V host. This could shed some light on the observed behavior.

P.S. one hint for you - in Veeam ONE v9 we have predefined assessment reports for Hyper-V infrastructure (configuration and performance), please check them out.

Thanks!

[adrianIDC]

How is your infrastructure?
Veeam B & R is VM ?

I have thought of something similar and found a solution but it depends on your infrastructure.

[HMwkr]

>> Veeam cannot reboot source VMs during backup jobs

Semantics schemantics, Vitaliy

Me saying there's a causal link between running a Veeam backup job and the source VM rebooting, isn't the same as me blaming Veeam.

Equally, I am saying causation, not correlation. I am saying there is/was a scenario, a very unexpected (to me) scenario, wherein running a Veeam backup job caused a reboot of the source VM; repeatably, across multiple VMs on multiple hosts. If my hunch is right, that this was due to running Veeam backup jobs while MS updates were still thrashing post-install, then Veeam users could benefit from knowing this. Unless of course there's no other Veeam user dumb enough to run a Veeam backup job less than an hour after applying MS updates, which I fully accept could be the case

If you or anyone else is interested in following up on this I'd be happy to share event logs, whether that confirms my hunch or highlights a completely different cause.

The pre-assessment reports sound useful. Although, if my hunch is right then they wouldn't have flagged up any problems in my scenario, unless I'd run them while the MS updates were still processing - and even then, would they pick up on that? I wouldn't expect them to, or hold it against them if they didn't.

Regards
W

[HMwkr]

AdrianIDC: yes, Veeam B&R is in a VM on 2012R2 on host2. The source VMs which rebooted included some on host2 and some on host1; some 2012R2, one 2008R2; one a DC, one Exchange, one SQL, one a fileserver; and one the Veeam B&R VM itself. These VMs are in a variety of different backup jobs, not all the one job. VMs rebooted at the same point in the backup (to the naked eye). Utilities such as security software are installed on only some of these VMs. Reboots repeated even after rebooting all VMs and hosts. So, lots of potential common factors ruled out, it seems to me.

Anything else you want to know about the infrastructure? What's your idea?

W

[adrianIDC]

To discarding .... you can vary the scheduling of backup job ?

If you cast a backup job also manually restart the VM ?

[Vitaliy S.]

Equally, I am saying causation, not correlation. I am saying there is/was a scenario, a very unexpected (to me) scenario, wherein running a Veeam backup job caused a reboot of the source VM; repeatably, across multiple VMs on multiple hosts. If my hunch is right, that this was due to running Veeam backup jobs while MS updates were still thrashing post-install, then Veeam users could benefit from knowing this

Yes, sharing your experience with other users is highly appreciated in these forums

If you or anyone else is interested in following up on this I'd be happy to share event logs, whether that confirms my hunch or highlights a completely different cause.

Yes, if you catch a particular event that indicates that a reboot is going to happen/actually happened that would be interesting to see as well.

The pre-assessment reports sound useful. Although, if my hunch is right then they wouldn't have flagged up any problems in my scenario, unless I'd run them while the MS updates were still processing - and even then, would they pick up on that? I wouldn't expect them to, or hold it against them if they didn't.

These assessment reports analyze your infrastructure and highlight possible configuration issues that would prevent you from making a backup. In the scenario of MS updates, they wouldn't help, you're right.