Odd One - VM's shutting down before backup

[samuk]

A customer is using Veeam backup 4.1 to backup there virtual machines.

Day before Yesterday, the backup failed for there Domain Controllers at 18:00. It turned out the DC's were switched off. No errors in the logs they were simply off. Looking at the logs of the server, last entry was timed just before the Veeam backup was due to start.

Yesterday - the same thing happened with a Linux Server - switched itself off somehow just before the backup.

They have run a manual job and it ran fine...


Anyone got any thoughts?

[Vitaliy S.]

Hello Sam,

I believe the issue might be related to either VMware snapshot creation or some other operations. I bet that VMware log files for those VMs would tell us more about the reason for this shutdown.

You may find the log files in the same folder where main VM configuration files are located, just match the time when the shutdown occurred with the corresponding timestamp in vmware.log file.

If you need any assistance with this, you may contact our technical team directly.

Hope it helps!

[samuk]

Hi Vitaliy,

Last night another server shut down..

The customer logged a call with Veeam with Rf: ID#541638.

Though i do not think support understand the issue Logs were also sent -is this something that can be escalated. v (the Veeam guy asked for the Veeam Logs) then an explanation was asked for..no contact number has been provided.

This call was logged yesterday afternoon near 1pm

The customer is very concerned at present, and we are unsure what the problm maybe.

I have asked for the support guy to call me.


--Sam

[samuk]

Servers that have rebooted so far:

09/11
Server2DC - Shutdown at 18:36 (Backup time was 18:36 in Veeam) but this failed
Server3DC -Shutdown at 19:02 (Backup time was 19:02 in Veeam) but this failed

10/11
Server1 - Shutdown at 17:44 (Backup time was 17:45in Veeam) but this failed

11
Server4 - Shutdown at 19:17 (This is the virtual Center and is not in the backup schedule) Memory Allocation erros on the server relating to this link which i am reviwing now.

http://communities.vmware.com/thread/242569

[samuk]

I have spoken to support, and have refered them to this link.

Sam

[Vitaliy S.]

Hi Sam, thanks for the ticket number... We will look into your problem with our support engineer.

[Vitaliy S.]

Sam,

After investigating your log files, seems like ESX host initiates the shutdown process prior Veeam B&R tries to pull VM data. I remember to have the same issue in my own labs, VMware log files stated something similar to your message, I didn't bother much about this VM, as it was my test VM...

I would suggest showing those VM log files to VMware support team, I'm sure that they have observed such a behaviour before and could shed some light on this issue.

On top of that, could you please tell what ESX(i) version are you using to run those VMs? I've searched VMware KB for your error message, please have a look....might help: http://kb.vmware.com/kb/1023749

Thanks!

[lobo519]

Having a simlar issue - Its happened twice so far on different VMs. One was a replication job the other was a backup. In my case it seems to happen when the Snapshot is being removed - then there is a "Cannot reset this virtual machine" error in vCenter. and it powers off. I'll submit a ticket to support but wanted to get this out there for anyone else.

[virtualwatts]

Samuk -
We are on ESXi 4.1 and Veeam 5 and Veeam Backup was shutting down three of our servers. Actually, to be specific, when the Veeam snapshot was initiated vSphere shut down the vm guest. We found event logs on the ESXi server that said a duplicate server was detected.

The root cause turned out to be errant snapshots on the vm guests. Meaning snapshots that were not listed in snapshot manager - but you could see in datastore browser.
We cleared those up and the auto shutdown behavior with Veeam stopped.

You may have a different root cause, but the tie-in with Veeam may be the same.... an auto-accept of a user prompt.

Check your VMware events and logs, what we found was that normally VMware would have prompted the user with "Warning - [situation detected] press [OK] to shutdown or [Cancel] to continue" but because the backup is automated the next message in the log (same time stamp) was "automatically accepting message". To which the VM guest would promptly shutdown.

Took us a bit to figure it out. There were no errors reported in Veeam, just the event log in VMware.

Rick

[Gostev]

Wow...

[virtualwatts]

Just to confirm, we have had this happen now in version 4 and version 5.0. One of our key production servers was shutdown by the Veeam backup process over the weekend. Same messages as in version 4 - only this time the snapshots were automatically cleaned up.

[avong]

[merged]

When my backups run i have a single windows vm host that shuts down and does not power back up, has anyone experienced this before and what should i be looking into?

[PFSIT]

[merged]

Hi,

We've had a strange issue happen to a couple of different VMs lately. Its only happened a small number of times and there is no pattern to the behaviour, however, sometimes following a backup, we'll check in the morning and the VM will no longer be on.

Nothing has logged in the event viewer so it hasn't been manually shutdown by anyone.

The backup completes, but with a "Warning" status, with the following details:
"Removing snapshot
RemoveSnapshot failed, snapshotRef "snapshot-6755", timeout "3600000"
Performing disk cleanup. Cannot power off."

Anyone have any thoughts?

[jimmymc]

We've seen lots of these in the past - pretty much cured when vSphere 4.0 was upgraded to 4.1. I don't know if that fits your environment.

I'd strongly suggest searching the VMware Knowledge Base and Community forums for these issues. They really aren't likely to be Veeam issues. Your friendly VMware Partner should also be able to help. If you don't have a partner, do you have VMware support?

PS - A tip if you have an issue with snapshots being present, but not listed in Snapshot Manager (as mentioned in one of the above posts), try vMotioning the VM to another host and see if they re-appear in the Snapshot Manager.

[asimiqbl]

Hello,
I also had same issue with one of the Veeam Virtual machines. The machine turned off periodically without throwing any message in event viewer. I went to local security policy and allow system events to be audited. I noticed I was using evaluation version of windows 2016 datacenter and it was main reason actually that caused me pain for several weeks. As soon as I converted the eval version (expired) to retail version the system came back normal. I hope this helps someone who's having same issue. Check to see if you got a licensed windows install and event viewer is clear. I'm using vcenter 7.0 u2 and esxi 6.7 u3.