Host-based backup of VMware vSphere VMs.
zak2011
Veteran
Posts: 367
Liked: 41 times
Joined: May 15, 2012 2:21 pm
Full Name: Arun
Contact:

Left behind snapshots are not removed by following job runs

Post by zak2011 »

If the snapshots created by a previous backups is not removed for some reason will Veeam tell vcenter to remove those temp Veeam snapshots on the second job run or will it create a totally seperate snapshot and remove the new snapshot.

Thanks
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Snapshot removal issues of a large VM

Post by foggy »

Arun, if you are talking about Veeam temporary snapshots, these are automatically removed by the job before the backup. Otherwise, the backup fails and you have to remove the snapshot manually via vSphere Client.
zak2011
Veteran
Posts: 367
Liked: 41 times
Joined: May 15, 2012 2:21 pm
Full Name: Arun
Contact:

Re: Snapshot removal issues of a large VM

Post by zak2011 »

I was talking about the Veeam temporary snapshots. In my case the snapshot was not automatically removed and the backup job has not failed.
So in my case, the previous temp Veeam snapshot remains and the backup job runs.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Snapshot removal issues of a large VM

Post by foggy »

Were you able to investigate this with technical support? I can see your posts above regarding the same issue and I guess you had a case open with support on this.
zak2011
Veteran
Posts: 367
Liked: 41 times
Joined: May 15, 2012 2:21 pm
Full Name: Arun
Contact:

Re: Snapshot removal issues of a large VM

Post by zak2011 »

I have contacted technical support on this issue again. They have found from the logs that veeam server had some disconnection with vcenter. They have stated the following " next time job runs irrespective of whether a previous Veeam temp snapshot exists.
Our R&D is aware of this behavior and is working on improving it"
skeeziix
Influencer
Posts: 19
Liked: 1 time
Joined: Jul 27, 2012 5:08 pm
Full Name: Erica

Re: Snapshot removal issues of a large VM

Post by skeeziix »

Is there any status on this. I am also experiencing this issue. The VM snapshot is not removed and eventually the HD will disassociate itself from the VM leaving it not working.
Gostev
Chief Product Officer
Posts: 31455
Liked: 6646 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Snapshot removal issues of a large VM

Post by Gostev »

Read up on future VMware technology called Virtual Volumes (vVols)... this is supposed to be the ultimate resolution to all current VMware snapshot pains, as this will (supposedly) bring more robust, and even hardware-accelerated snapshot implementation to VMware. I am really looking forward to this!
zak2011
Veteran
Posts: 367
Liked: 41 times
Joined: May 15, 2012 2:21 pm
Full Name: Arun
Contact:

Re: Snapshot removal issues of a large VM

Post by zak2011 » 2 people like this post

@Erica- Support contacted about this and said that they have discussed this with R&D and they have asked R& D to incorporate some prechecks for these temp snapshots in future releases.In my case both times the snapshots were not removed because there was some network disconnection with vCenter.
However for now, their recommendation was to use Veeam One to be aware of and monitor for any left over snapshots.
getzjd
Enthusiast
Posts: 25
Liked: 3 times
Joined: Mar 12, 2012 3:10 am
Full Name: Jay
Location: Cincinnati, OH
Contact:

Re: Snapshot removal issues of a large VM

Post by getzjd »

I believe we are experiencing the same issue with a VMof ours that has 3 datastores configured: 80gb, 1.9tb, and 1tb. Night shift is slow for us but a few users were complaining of their redirected desktops going offline around 12am. That is within 30 minutes of the end of that backup job every night. Tonight I adjusted the start time of the job by 3 hours and am going to have the users report if the issue now starts happening around 3am. If so, I will be opening a ticket for a fix. vsphere 5.0 , veeam 6.1
skeeziix
Influencer
Posts: 19
Liked: 1 time
Joined: Jul 27, 2012 5:08 pm
Full Name: Erica

Re: Snapshot removal issues of a large VM

Post by skeeziix »

We had this happen again. This is just unacceptable when it takes down a production server, we cannot have this. Is there any further progress on this besides using Veeam One to monitor for the issue?
Gostev
Chief Product Officer
Posts: 31455
Liked: 6646 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Snapshot removal issues of a large VM

Post by Gostev »

Hi Erica,
zak2011 wrote:the previous temp Veeam snapshot remains and the backup job runs.
This particular issue was addressed in B&R 6.5 Patch 1, as per the Resolved Issues list:
• Under certain circumstances, the job will not attempt to remove the VM snapshot left behind by the previous job run.
Thanks!
skeeziix
Influencer
Posts: 19
Liked: 1 time
Joined: Jul 27, 2012 5:08 pm
Full Name: Erica

Re: Snapshot removal issues of a large VM

Post by skeeziix »

Great thank you!
skeeziix
Influencer
Posts: 19
Liked: 1 time
Joined: Jul 27, 2012 5:08 pm
Full Name: Erica

Re: Snapshot removal issues of a large VM

Post by skeeziix »

This did not fix the issue I just had this happen on another VM. The snapshot was so huge it took up the whole HD and then took it off line. i looked at the B & R it was stuck at removing the snapshot. I tried to power off the VM it was in a suspended mode stating "Another Task was Already In Progress". I had to power off the backup machine and then I was able to consolidate and delete the snapshot. This is ridiculous in a production environment, this cannot happen. Luckily this wasn't an important machine. Last time it was our second DC with the Accounting ERP. If it was the email server god forbid.
skeeziix
Influencer
Posts: 19
Liked: 1 time
Joined: Jul 27, 2012 5:08 pm
Full Name: Erica

Re: Snapshot removal issues of a large VM

Post by skeeziix »

I did have safe removal for snapshots larger than 100mb, could this be the issue?
tsightler
VP, Product Management
Posts: 6009
Liked: 2842 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Snapshot removal issues of a large VM

Post by tsightler » 1 person likes this post

skeeziix wrote:I did have safe removal for snapshots larger than 100mb, could this be the issue?
This feature is only for very old ESX 3.5 and generally should not be used with newer versions.

Regarding your ongoing issues, it's important to note that you must have enough storage space on your datastores to handle the snapshot, and the removal. Also, if there are processes that are causing large amounts of change to occur while the backup is running, the snapshot can grow very large very quickly and it's possible it will grow faster than VMware can commit it. For example, and Exchange server running nightly database maintenance can easily cause a snapshot to grow large very quickly and then cause with snapshot removal. Database backups/dumps to local disk can cause large snapshots, etc.

In any scenario where snapshots are being used (either by a backup product like Veeam, or by admins), monitoring snapshots and/or datastore space is important. You can certainly set datastore alerts within vCenter to give space warnings. You should also make sure you have plenty of space for snapshots on those datastores during "normal" operation.
Gostev
Chief Product Officer
Posts: 31455
Liked: 6646 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Snapshot removal issues of a large VM

Post by Gostev »

skeeziix wrote:This did not fix the issue I just had this happen on another VM. The snapshot was so huge it took up the whole HD and then took it off line.
If it was so huge, then probably it existed way before the patch was installed. Alternatively, may be it just grew to this size during backup (just what Tom has just explained), and you simply need to revise your datastore disk space allocation to account for snapshots growth.

That said, if you believe you are still having the issue of Veeam not initiating left-behind snapshot automatically in the beginning of the following job, please open a support case and include case ID here. Most likely this would be something basic, for example patch not actually installed, or installed incorrectly (without going through Upgrade wizard after installation). There is just too little chance for a patch specifically built and tested to fix some issue not to be fixing one.

Also, I will be splitting this specific issue of left-behind snapshot into the new topic, because the current topic is talking about completely different issue.
skeeziix
Influencer
Posts: 19
Liked: 1 time
Joined: Jul 27, 2012 5:08 pm
Full Name: Erica

Re: Left behind snapshots are not removed by following job r

Post by skeeziix »

I patched everything installed Veeam ONE. I had the same issues again last night this time the HD became disassociated from the machine, it was showing up as offline and that there was no OS. I re-associated the drives it is working. This has happened on multiple machines. Thank goodness it wasn't an important one this time. This is an 80GB machine and I have terabytes available for storage on the host, it shouldn't be an issue with storage.
skeeziix
Influencer
Posts: 19
Liked: 1 time
Joined: Jul 27, 2012 5:08 pm
Full Name: Erica

Re: Left behind snapshots are not removed by following job r

Post by skeeziix »

Let me know if I can post some logs or anything to get to the root of this issue... (And how to post the logs)
tsightler
VP, Product Management
Posts: 6009
Liked: 2842 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Left behind snapshots are not removed by following job r

Post by tsightler »

This isn't the place for logs as this is not a support forum. You should open a support case and submit the logs.
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Left behind snapshots are not removed by following job r

Post by veremin »

As it has been previously mentioned, the fastest way to address your issue is to open a ticket with support team, since it’s way too complicated to investigate such a problem remotely through forum correspondence.

So, please visit Customer Support Portal to open a case, attach logs, etc. Our excellent support team will be in touch!

Hope this helps.
Thanks.
skeeziix
Influencer
Posts: 19
Liked: 1 time
Joined: Jul 27, 2012 5:08 pm
Full Name: Erica

Re: Left behind snapshots are not removed by following job r

Post by skeeziix »

This is obviously a known issue there are multiple references to this issue I would assume you have support cases open for this.
http://forums.veeam.com/viewtopic.php?f ... &start=135
http://community.spiceworks.com/topic/2 ... kup?page=2
tsightler
VP, Product Management
Posts: 6009
Liked: 2842 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Left behind snapshots are not removed by following job r

Post by tsightler » 1 person likes this post

You are correct, it is known that in some environments snapshot removal can fail, however, there's not one single root cause of this issue and there are no known issues in Veeam specifically that cause this as the issue has more to do with the vSphere infrastructure (vSphere itself is responsible for taking and removing snapshots, we simply make the calls to the API requesting it). That being said, our support team can analyze logs and help to determine the possible cause.

Also, the issue specifically addressed in this thread (that snapshots that are left behind are not removed by following job runs) was corrected in the most recent patch. This is a completely different issue from what you describe.

You're case does not appear to be exactly the same as the thread that you link to. The first thread specifically refers to performance issues and temporary disconnects due to a long snapshot removal process, not complete failure. The second thread also refers to disconnects of the VM during snapshot removal, a known issue caused by VM stun which is only temporary, but yet again is specific to VMware, not Veeam. The issue you are seeing appears far more serious than those as you link to as you are talking about crashes of VMs and disconnected datastores. Investigation by support would be required.
skeeziix
Influencer
Posts: 19
Liked: 1 time
Joined: Jul 27, 2012 5:08 pm
Full Name: Erica

Re: Left behind snapshots are not removed by following job r

Post by skeeziix »

I have all of those scenarios that is why I have made note on them. Sometimes the Snapshot is not removed and makes the VM go "offline" sometimes the snapshot is removed but the drives are disassociated. It is very bother some. I will submit a support ticket.
Gostev
Chief Product Officer
Posts: 31455
Liked: 6646 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Left behind snapshots are not removed by following job r

Post by Gostev »

As Tom already mentioned, this would indicate issues with your VMware infrastructure, as Veeam itself does not handle snapshot removal - this is done by VMware. VMware snapshots are very reliable actually (I've seen many customers running near-CDP jobs with zero issues - and that's a VM snapshot every few minutes). However, some infrastructure issues may affect snapshot processing, and then you will have all sorts of problems even with daily jobs or manual snapshot operations with vSphere Client.
zak2011
Veteran
Posts: 367
Liked: 41 times
Joined: May 15, 2012 2:21 pm
Full Name: Arun
Contact:

Re: Left behind snapshots are not removed by following job r

Post by zak2011 »

@erica- Its best to take this with the support as already mentioned. As Tom mentioned earlier this topic was for snapshots not getting removed after the next backup job run. Once i have applied the latest patch for 6.5 ..i havent got this problem again for any VM.
So best thing is to take it up with support directly as your situation is quite different.
Hope you can get these issues resolved soon.
zak2011
Veteran
Posts: 367
Liked: 41 times
Joined: May 15, 2012 2:21 pm
Full Name: Arun
Contact:

Re: Left behind snapshots are not removed by following job r

Post by zak2011 » 2 people like this post

Just to reiterate on this Erica..i had some issues two days ago again with vcenter disconnection and i got warnings on snapshot removal from Veeam for several VMs. However unlike previous times, the snapshots have all been removed after the second job run. Updating the patch has definitely worked.
zak2011
Veteran
Posts: 367
Liked: 41 times
Joined: May 15, 2012 2:21 pm
Full Name: Arun
Contact:

Case no:00174073

Post by zak2011 »

Unfortunately experiencing some left over snapshots for two VMs since yesterday and not removed after the second backup job run. This time vcenter has not experienced any disconnections and there are no warnings or errors during the job run.
Opened a case with support and awaiting further investigation.Will update the post once I get feedback from support.
skeeziix
Influencer
Posts: 19
Liked: 1 time
Joined: Jul 27, 2012 5:08 pm
Full Name: Erica

Re: Left behind snapshots are not removed by following job r

Post by skeeziix » 1 person likes this post

Do any of you have VEEAM running on multiple machines? Is it possible they are running into each other? Or that the Proxies are? You should only have one VM on each host as a designated proxy that will do all of the connections for all of your VMs ie. the machine which is running VEEAM if it is virtual. Schedule your jobs to run after each other in the time, instead of giving it a specific time. Also, set a window of a few hours allowed for it to run otherwise have it quit. Install VeeamOne free has helped me identify which ones are left with residual Snapshots. Add an alarm to your vCenter Server to monitor for snapshots... If you need help let me know and I will try to assist you.
zak2011
Veteran
Posts: 367
Liked: 41 times
Joined: May 15, 2012 2:21 pm
Full Name: Arun
Contact:

Re: Left behind snapshots are not removed by following job r

Post by zak2011 »

The case has been escalated to 'Tier 2 support' and awaiting further investigation
zak2011
Veteran
Posts: 367
Liked: 41 times
Joined: May 15, 2012 2:21 pm
Full Name: Arun
Contact:

Re: Left behind snapshots are not removed by following job r

Post by zak2011 »

The case has been escalated to 'Tier3' support and hoping that the reason behind the left over snapshots can be found soon.
Will update the post as soon as I have some update from support!
Post Reply

Who is online

Users browsing this forum: No registered users and 75 guests