Host-based backup of VMware vSphere VMs.
Post Reply
mjgreen81
Enthusiast
Posts: 84
Liked: 2 times
Joined: Feb 18, 2013 3:06 pm
Full Name: Mike Green
Contact:

Automatic Snap Shot taken overnight - Causes problems

Post by mjgreen81 »

Hi all,

We have noticed recently that an automatic snapshot of our SQL server is taken and then deleted overnight (we recieve the emails to confirm that this action has happened). At the exact time the snapshot is deleted it seems to cause a temporary connectivity issue which seems to break the connection between the SQL server and one of the servers that relies on it (we then have to restart services to get everything working again).

This happened on Tuesday and again 5 weeks ago at the exact same time (also on a Tuesday).

We have had a look at the Veeam Backup jobs and cant see anything that might be causing this. I am not sure whether this is linked to the backups as the time's just don't seem to match up.

Can anyone advise why the system is taking an automated snapshot and why this might be causing the connectivity problem?
mjgreen81
Enthusiast
Posts: 84
Liked: 2 times
Joined: Feb 18, 2013 3:06 pm
Full Name: Mike Green
Contact:

Re: Automatic Snap Shot taken overnight - Causes problems

Post by mjgreen81 »

I should add, that I realise a snapshot is taken and deleted as part of the usual backup process, but we don't normally receive an email informing us of the status change from green to red and then back from red to green.

If we do receive the email, we have the problem.

It seems to behave the same way as when we manually trigger a snapshot.
mjgreen81
Enthusiast
Posts: 84
Liked: 2 times
Joined: Feb 18, 2013 3:06 pm
Full Name: Mike Green
Contact:

Re: Automatic Snap Shot taken overnight - Causes problems

Post by mjgreen81 »

Another thing I should add is the strange fact that one of the snapshots seems to be missing from the log:

This is the chain of events of a normal night:

23:31 Tasks & Events in vSphere shows Snapshot Created (Veeam Backup job due to start at 23:30)
01:32 Tasks & Events in vSphere shows Snapshot deleted

This is the chain of events when we have the problem:

23:31 Tasks & Events in vSphere shows Snapshot Created (Veeam Backup job due to start at 23:30)
01:15 Email received to warn that snapshot status has changed from Green to Red (there is no mention of this in Tasks & Events)
01:32 Tasks & Events in vSphere shows Snapshot deleted & Email received to confirm status has changed from red to green
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Automatic Snap Shot taken overnight - Causes problems

Post by veremin »

Hi, Mike, you're completely right in saying that this is the exact way the image based backup product work. So, I'm not sure whether we can do something about it. However, you might want to check this topic for some hints on how to avoid snapshot removal issues, such as lost of connectivity, etc.

Thank.
mjgreen81
Enthusiast
Posts: 84
Liked: 2 times
Joined: Feb 18, 2013 3:06 pm
Full Name: Mike Green
Contact:

Re: Automatic Snap Shot taken overnight - Causes problems

Post by mjgreen81 »

v.Eremin wrote:Hi, Mike, you're completely right in saying that this is the exact way the image based backup product work. So, I'm not sure whether we can do something about it. However, you might want to check this topic for some hints on how to avoid snapshot removal issues, such as lost of connectivity, etc.

Thank.
Thanks, I will take a look at the link. Can you think why the system sends me the status emails, when on a normal night (and when we don't have the problem) we don't receive the email ?

Also, that link doesnt seem to work :D
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Automatic Snap Shot taken overnight - Causes problems

Post by veremin »

Just a wild guess. It might happen that this very backup took longer time than expected. The snapshot was kept longer. As the result, you was notified about it.

The fact that you experience similar behaviour in case of manual snapshot makes me believe that the whole situation looks expected.

As to the link, I've just fixed it.

Thanks.
mjgreen81
Enthusiast
Posts: 84
Liked: 2 times
Joined: Feb 18, 2013 3:06 pm
Full Name: Mike Green
Contact:

Re: Automatic Snap Shot taken overnight - Causes problems

Post by mjgreen81 »

Thanks for your help.

After reading that link, it looks like it is a known issue and as you say, the emails are just a symptom of a longer than usual backup, which results in a longer than usual snapshot commitment, which results in a brief period when the server is not responding.

hmmmm, thats annoying.
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Automatic Snap Shot taken overnight - Causes problems

Post by veremin »

Yep, it's known situation with snapshot commit operation. Though, there are several tricks discussed in the referenced thread that might help you to solve the issue. Thanks.
dellock6
Veeam Software
Posts: 6137
Liked: 1928 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: Automatic Snap Shot taken overnight - Causes problems

Post by dellock6 »

Mike, out of curiosity, what storage are you using? Because right now Veeam support storage snapshots for HP StoreVirtual and HP 3Par, and this is by far the best solution to highly reduce the snapshot commit problems. And is obvious other storage vendors and models are going to be supported in the future, so chances are even if you do not have an HP storage, maybe your storage will be supported some day. (before you ask me, no I do not know anything about the roadmap for additional vendors).
Have a look at this thread: http://forums.veeam.com/veeam-backup-re ... 14700.html

Luca.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
Post Reply

Who is online

Users browsing this forum: No registered users and 112 guests