Automatic Snap Shot taken overnight - Causes problems

[mjgreen81]

Hi all,

We have noticed recently that an automatic snapshot of our SQL server is taken and then deleted overnight (we recieve the emails to confirm that this action has happened). At the exact time the snapshot is deleted it seems to cause a temporary connectivity issue which seems to break the connection between the SQL server and one of the servers that relies on it (we then have to restart services to get everything working again).

This happened on Tuesday and again 5 weeks ago at the exact same time (also on a Tuesday).

We have had a look at the Veeam Backup jobs and cant see anything that might be causing this. I am not sure whether this is linked to the backups as the time's just don't seem to match up.

Can anyone advise why the system is taking an automated snapshot and why this might be causing the connectivity problem?

[mjgreen81]

I should add, that I realise a snapshot is taken and deleted as part of the usual backup process, but we don't normally receive an email informing us of the status change from green to red and then back from red to green.

If we do receive the email, we have the problem.

It seems to behave the same way as when we manually trigger a snapshot.

[mjgreen81]

Another thing I should add is the strange fact that one of the snapshots seems to be missing from the log:

This is the chain of events of a normal night:

23:31 Tasks & Events in vSphere shows Snapshot Created (Veeam Backup job due to start at 23:30)
01:32 Tasks & Events in vSphere shows Snapshot deleted

This is the chain of events when we have the problem:

23:31 Tasks & Events in vSphere shows Snapshot Created (Veeam Backup job due to start at 23:30)
01:15 Email received to warn that snapshot status has changed from Green to Red (there is no mention of this in Tasks & Events)
01:32 Tasks & Events in vSphere shows Snapshot deleted & Email received to confirm status has changed from red to green

[veremin]

Hi, Mike, you're completely right in saying that this is the exact way the image based backup product work. So, I'm not sure whether we can do something about it. However, you might want to check this topic for some hints on how to avoid snapshot removal issues, such as lost of connectivity, etc.

Thank.

[mjgreen81]

Hi, Mike, you're completely right in saying that this is the exact way the image based backup product work. So, I'm not sure whether we can do something about it. However, you might want to check this topic for some hints on how to avoid snapshot removal issues, such as lost of connectivity, etc.

Thank.

Thanks, I will take a look at the link. Can you think why the system sends me the status emails, when on a normal night (and when we don't have the problem) we don't receive the email ?

Also, that link doesnt seem to work

[veremin]

Just a wild guess. It might happen that this very backup took longer time than expected. The snapshot was kept longer. As the result, you was notified about it.

The fact that you experience similar behaviour in case of manual snapshot makes me believe that the whole situation looks expected.

As to the link, I've just fixed it.

Thanks.

[mjgreen81]

Thanks for your help.

After reading that link, it looks like it is a known issue and as you say, the emails are just a symptom of a longer than usual backup, which results in a longer than usual snapshot commitment, which results in a brief period when the server is not responding.

hmmmm, thats annoying.

[veremin]

Yep, it's known situation with snapshot commit operation. Though, there are several tricks discussed in the referenced thread that might help you to solve the issue. Thanks.

[dellock6]

Mike, out of curiosity, what storage are you using? Because right now Veeam support storage snapshots for HP StoreVirtual and HP 3Par, and this is by far the best solution to highly reduce the snapshot commit problems. And is obvious other storage vendors and models are going to be supported in the future, so chances are even if you do not have an HP storage, maybe your storage will be supported some day. (before you ask me, no I do not know anything about the roadmap for additional vendors).
Have a look at this thread: http://forums.veeam.com/veeam-backup-re ... 14700.html

Luca.