Comprehensive data protection for all workloads
Post Reply
samuk
Expert
Posts: 127
Liked: never
Joined: Mar 18, 2009 2:15 pm
Full Name: Sam
Contact:

Odd One - VM's shutting down before backup

Post by samuk »

A customer is using Veeam backup 4.1 to backup there virtual machines.

Day before Yesterday, the backup failed for there Domain Controllers at 18:00. It turned out the DC's were switched off. No errors in the logs they were simply off. Looking at the logs of the server, last entry was timed just before the Veeam backup was due to start.

Yesterday - the same thing happened with a Linux Server - switched itself off somehow just before the backup.

They have run a manual job and it ran fine...


Anyone got any thoughts?
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Odd One - VM's shutting down before backup

Post by Vitaliy S. »

Hello Sam,

I believe the issue might be related to either VMware snapshot creation or some other operations. I bet that VMware log files for those VMs would tell us more about the reason for this shutdown.

You may find the log files in the same folder where main VM configuration files are located, just match the time when the shutdown occurred with the corresponding timestamp in vmware.log file.

If you need any assistance with this, you may contact our technical team directly.

Hope it helps!
samuk
Expert
Posts: 127
Liked: never
Joined: Mar 18, 2009 2:15 pm
Full Name: Sam
Contact:

Re: Odd One - VM's shutting down before backup

Post by samuk »

Hi Vitaliy,

Last night another server shut down..

The customer logged a call with Veeam with Rf: ID#541638.

Though i do not think support understand the issue Logs were also sent -is this something that can be escalated. v (the Veeam guy asked for the Veeam Logs) then an explanation was asked for..no contact number has been provided.

This call was logged yesterday afternoon near 1pm

The customer is very concerned at present, and we are unsure what the problm maybe.

I have asked for the support guy to call me.


--Sam
samuk
Expert
Posts: 127
Liked: never
Joined: Mar 18, 2009 2:15 pm
Full Name: Sam
Contact:

Re: Odd One - VM's shutting down before backup

Post by samuk »

Servers that have rebooted so far:

09/11
Server2DC - Shutdown at 18:36 (Backup time was 18:36 in Veeam) but this failed
Server3DC -Shutdown at 19:02 (Backup time was 19:02 in Veeam) but this failed

10/11
Server1 - Shutdown at 17:44 (Backup time was 17:45in Veeam) but this failed

11
Server4 - Shutdown at 19:17 (This is the virtual Center and is not in the backup schedule) Memory Allocation erros on the server relating to this link which i am reviwing now.

http://communities.vmware.com/thread/242569
samuk
Expert
Posts: 127
Liked: never
Joined: Mar 18, 2009 2:15 pm
Full Name: Sam
Contact:

Re: Odd One - VM's shutting down before backup

Post by samuk »

I have spoken to support, and have refered them to this link.

Sam
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Odd One - VM's shutting down before backup

Post by Vitaliy S. »

Hi Sam, thanks for the ticket number... We will look into your problem with our support engineer.
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Odd One - VM's shutting down before backup

Post by Vitaliy S. »

Sam,

After investigating your log files, seems like ESX host initiates the shutdown process prior Veeam B&R tries to pull VM data. I remember to have the same issue in my own labs, VMware log files stated something similar to your message, I didn't bother much about this VM, as it was my test VM...

I would suggest showing those VM log files to VMware support team, I'm sure that they have observed such a behaviour before and could shed some light on this issue.

On top of that, could you please tell what ESX(i) version are you using to run those VMs? I've searched VMware KB for your error message, please have a look....might help: http://kb.vmware.com/kb/1023749

Thanks!
lobo519
Veteran
Posts: 315
Liked: 38 times
Joined: Sep 29, 2010 3:37 pm
Contact:

Re: Odd One - VM's shutting down before backup

Post by lobo519 »

Having a simlar issue - Its happened twice so far on different VMs. One was a replication job the other was a backup. In my case it seems to happen when the Snapshot is being removed - then there is a "Cannot reset this virtual machine" error in vCenter. and it powers off. I'll submit a ticket to support but wanted to get this out there for anyone else.
virtualwatts
Enthusiast
Posts: 50
Liked: never
Joined: Nov 18, 2010 2:41 pm
Full Name: Rick Watts

Re: Odd One - VM's shutting down before backup

Post by virtualwatts »

Samuk -
We are on ESXi 4.1 and Veeam 5 and Veeam Backup was shutting down three of our servers. Actually, to be specific, when the Veeam snapshot was initiated vSphere shut down the vm guest. We found event logs on the ESXi server that said a duplicate server was detected.

The root cause turned out to be errant snapshots on the vm guests. Meaning snapshots that were not listed in snapshot manager - but you could see in datastore browser.
We cleared those up and the auto shutdown behavior with Veeam stopped.

You may have a different root cause, but the tie-in with Veeam may be the same.... an auto-accept of a user prompt.

Check your VMware events and logs, what we found was that normally VMware would have prompted the user with "Warning - [situation detected] press [OK] to shutdown or [Cancel] to continue" but because the backup is automated the next message in the log (same time stamp) was "automatically accepting message". To which the VM guest would promptly shutdown.

Took us a bit to figure it out. There were no errors reported in Veeam, just the event log in VMware.

Rick
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Odd One - VM's shutting down before backup

Post by Gostev »

Wow...
virtualwatts
Enthusiast
Posts: 50
Liked: never
Joined: Nov 18, 2010 2:41 pm
Full Name: Rick Watts

Re: Odd One - VM's shutting down before backup

Post by virtualwatts »

Just to confirm, we have had this happen now in version 4 and version 5.0. One of our key production servers was shutdown by the Veeam backup process over the weekend. Same messages as in version 4 - only this time the snapshots were automatically cleaned up.
avong
Lurker
Posts: 1
Liked: never
Joined: Oct 27, 2011 2:10 am
Contact:

VEEAM back up causing a windows 7 vm host to shut down

Post by avong »

[merged]

When my backups run i have a single windows vm host that shuts down and does not power back up, has anyone experienced this before and what should i be looking into?
PFSIT
Novice
Posts: 6
Liked: never
Joined: Nov 19, 2010 9:37 am
Full Name: PFS IT

VM randomly powered off after backup

Post by PFSIT »

[merged]

Hi,

We've had a strange issue happen to a couple of different VMs lately. Its only happened a small number of times and there is no pattern to the behaviour, however, sometimes following a backup, we'll check in the morning and the VM will no longer be on.

Nothing has logged in the event viewer so it hasn't been manually shutdown by anyone.

The backup completes, but with a "Warning" status, with the following details:
"Removing snapshot
RemoveSnapshot failed, snapshotRef "snapshot-6755", timeout "3600000"
Performing disk cleanup. Cannot power off."

Anyone have any thoughts?
jimmymc
Service Provider
Posts: 34
Liked: 4 times
Joined: Dec 09, 2010 3:06 pm
Contact:

Re: Odd One - VM's shutting down before backup

Post by jimmymc »

We've seen lots of these in the past - pretty much cured when vSphere 4.0 was upgraded to 4.1. I don't know if that fits your environment.

I'd strongly suggest searching the VMware Knowledge Base and Community forums for these issues. They really aren't likely to be Veeam issues. Your friendly VMware Partner should also be able to help. If you don't have a partner, do you have VMware support?

PS - A tip if you have an issue with snapshots being present, but not listed in Snapshot Manager (as mentioned in one of the above posts), try vMotioning the VM to another host and see if they re-appear in the Snapshot Manager.
asimiqbl
Lurker
Posts: 1
Liked: 1 time
Joined: May 13, 2022 12:04 am
Full Name: Asim Iqbal
Contact:

Re: Odd One - VM's shutting down before backup

Post by asimiqbl » 1 person likes this post

Hello,
I also had same issue with one of the Veeam Virtual machines. The machine turned off periodically without throwing any message in event viewer. I went to local security policy and allow system events to be audited. I noticed I was using evaluation version of windows 2016 datacenter and it was main reason actually that caused me pain for several weeks. As soon as I converted the eval version (expired) to retail version the system came back normal. I hope this helps someone who's having same issue. Check to see if you got a licensed windows install and event viewer is clear. I'm using vcenter 7.0 u2 and esxi 6.7 u3.
Post Reply

Who is online

Users browsing this forum: Google [Bot], Semrush [Bot] and 242 guests