Veeam temp snapshots not removed

[mbiesheuvel]

You can best say: if veeam is installed on the same server as vcenter .

[hansvh]

Thanks for your quick reply, but as I stated in my support case vCenter is NOT installed on the Veeam backup server so the article you refer to does not apply...

[foggy]

It's not about installing vCenter server on the Veeam backup server (just a subcase). It's about adding vCenter server to the Veeam B&R console as a Windows server.

[veremin]

Thanks for your quick reply, but as I stated in my support case vCenter is NOT installed on the Veeam backup server so the article you refer to does not apply...

Since you haven’t provided a support ticket ID, I wasn’t aware about that your issue was different and tried to provide you with the information regarding most likeable reason of this issue.

Then, let’s wait and see what support team says about this problem.

[hansvh]

Well, it looks like my problem was indeed caused by the vCenter Server which was added as a Windows server. The known issue therefore, apart from the fact that in my case this was not visible in the GUI anywhere at all!!!

Veeam support made some changes to the database, I'm confident this solved my issue.

[foggy]

Glad it resolved now, thanks for the update.

[monsterlab]

Does anyone know what needs to be deleted from the Veeam database to clear this up? I do not have the Vcenter server listed under Windows servers, but I know that I had it at one point and I've since deleted it. Perhaps there is something hanging around in the database that needs to be cleaned up. Nevertheless, I'm seeing the same issue where all of my temp snapshots aren't getting cleared out. I woke up this morning to nearly 500 snapshots on my ESX cluster and multiple datastores that were almost completely full.

What a nasty bug to not have a hotfix for yet...

[foggy]

Does anyone know what needs to be deleted from the Veeam database to clear this up?

Our support can definitely help you with cleaning up the database. Please contact them directly.

[veremin]

As to the hotfix, according to the plan, the patch#1 that addresses issue with left snapshots is supposed to be released later this week. Thanks.

[zak2011]

Do left over snapshots get failed to be removed if there are references of old snapshots in the Veeam snapshot Tracker table in Veeam SQL database?

Thanks

[foggy]

There could be several reasons, support should be able to detect the particular one in your case.

[zak2011]

This is the update from support:-

1. backup job requested vCenter to create a VM snapshot;
2. backup job created db records in [dbo].[VmWareSnapshots] for snapshot tracker, in order to remove them in the end of backup job;
3. backup job requested vCenter to delete the snapshot;
4. backup server lost connection to vCenter and did not receive reply that the snapshot is removed;
5. db record of the snapshot wasn't removed from db table [dbo].[VmWareSnapshots]
6. snapshot tracker tries to delete the snapshot listed in table [dbo].[VmWareSnapshots], but this snapshot doesn't exist on the vCenter, so goes error: "Object reference not set..."

Solution in this case is to clean up the db table [dbo].[VmWareSnapshots] (when no jobs are running).

I did not fully understand though how it applies in my case. The above resolution is right when snapshots are manually deleted from vcenter and not removed from the database.
In my case, I didnt remove the Veeam left over snapshots. The snapshots did not get removed either from vsphere or from the sql database.

Thanks

[foggy]

I did not fully understand though how it applies in my case. The above resolution is right when snapshots are manually deleted from vcenter and not removed from the database.
In my case, I didnt remove the Veeam left over snapshots. The snapshots did not get removed either from vsphere or from the sql database.

Have you discussed this with support engineer?

[zak2011]

Yes foggy. Have discussed this with the support engineer. They will have a look at this again.

Thanks

[cliffm]

Hmmm,

I have just upgraded to V7 today patch 3. Yep, Veeam temp snapshots not being removed but only on 1 Linux machine (so far), they are piling up. All other VMs the snapshots are gone OK at the end of a job. I have logged a support case 00554734. BTW I had noticed this forum topic which was the only reason I had not upgraded to V7 eralier. I was hoping it would be fixed by now. Alas!

[cliffm]

Hmmm,

Have not heard from support and got snapshots piling up all over the place.
I bit the bullet and moved the jobs to a B&R server on a network that does not have a vCentre server.
Now the jobs are working OK. The network is not on the Internet so no email messages, but I can live with that if I check in regularly.
Usually I have the vCentre server turned off, because when it is on most of my Microsoft server backups and replicas fail with VSS timeouts.
So kind of sorted, at least I have backups without having to manually delete the snaps every time. Very time consuming process!

I have to add a comment, I dread Veeam upgrades because every time I do something breaks and requires me to completely rearrange my backup strategies. This has been the case with every upgrade over the last 3 years now. But overall I think Veeam is gold and it has saved my bacon a couple of times...

[Vitaliy S.]

Hi Cliff,

I see that you're on Standard Support contract, so please be aware that Standard Support is provided during business hours only. As to the workaround that you've mentioned, then it seems like the vCenter Server might be one of the reasons why snapshot is not removed after the backup job.

Usually I have the vCentre server turned off, because when it is on most of my Microsoft server backups and replicas fail with VSS timeouts.

Hmm...not sure I follow you here. Can you please clarify? Do you have a vCenter Server added to the backup console and you periodically turn it off?

Thanks!

[veremin]

Just out of curiosity - were you able to reproduce this issue manually without Veeam presence? Or the problem occurred only when Veeam was involved in the process? Thanks.

[cliffm]

Hi Cliff,

I see that you're on Standard Support contract, so please be aware that Standard Support is provided during business hours only. As to the workaround that you've mentioned, then it seems like the vCenter Server might be one of the reasons why snapshot is not removed after the backup job.
Hmm...not sure I follow you here. Can you please clarify? Do you have a vCenter Server added to the backup console and you periodically turn it off?

Thanks!

Hi Vitaliy,

Yes I usually turn the vCentre server off. I have had a history of problems with Veeam and Vcentre server so my policy is to always set Veeam jobs direct to host. With the vCentre server off I usually have successful snapshots. If I turn it on snaps fail all over the place (only on MS Exchange VMs) but eventaully start to succeed after a few days or more (even though my jobs are direct to host). So it is easier to have it off, I don't need it unless I want to do VMware updates. I have been through this many times, including with support, I am not going to spend any more time on it. Backups are more important than vcentre server so whatever I have to do to get Veeam stable wins the race as far as I am concerned.

Cliff

[bsoosay]

I'm using Veeam Backup & Replication version 7.

I observed that after some of the backups had completed successfully, from vCenter, I could see that certain VM's had snapshots that were not removed by Veeam. I found this from Snapshot Manager on vCenter.

1. Any idea why this is happening?

2. Can I remove these snapshots manually by deleting them from Snapshot Manager on vCenter?

3. What am I doing wrong, that's causing this to happen?

I also observed that the hard disks of these VM's (where the snapshots have not been removed), are pointing to the snapshot files and not the parent .vmdk files.

[foggy]

That's not an intended behavior, however can occur occasionally due to various reasons, please review the thread above for details. Are you able to reproduce the same behavior manually, while taking snapshots using vSphere Client? Though you can safely remove those snapshots manually, it might worth contacting support.

[veremin]

Also, don't forget about using Veeam ONE to stay updated about orphaned snapshots' presence. Thanks.