Comprehensive data protection for all workloads
Post Reply
coolgod
Enthusiast
Posts: 58
Liked: never
Joined: Jun 27, 2013 6:00 am
Full Name: Vicky | techvicky.com
Location: MAURITIUS
Contact:

snapshot consolidation fails after backup completes

Post by coolgod » Oct 07, 2014 12:17 pm

Hi All,

I have also been searching for the solution on the vmware forum.

My infrastructure consists of Veeam BR 7 , 1 ESXi-PROD and 1 ESXi-DR.


ESXi-PROD--------------->Veeam BR7-------------->ESXi-DR


After backup was complete, I noticed the VM was running on helper snapshot...this snapshot was unusually growing bigger.

So at the right time, i noticed this and stopped the Veeam Job. Then I manually did a DELETE ALL from the snapshot manager.

The consolidation was successfully completed.

Apart from monitoring the datastore freespace, I want to know the exact cause of this issue.
This is critical because if not monitored, the helper snapshot will grow until it uses all the free disk space on the datastore.

Please advise.
Whatever the mind of man can conceive and believe, it can achieve. - Napoleon Hill

Dima P.
Product Manager
Posts: 10704
Liked: 880 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: snapshot consolidation fails after backup completes

Post by Dima P. » Oct 07, 2014 12:39 pm

Hello Vicky,
You may want to troubleshoot this issues with support team, however there was a really good KB article, describing possible reasons and troubleshooting steps of your issue: VM Loses Connection During Snapshot Removal

Additionally, using Veeam ONE free edition for monitoring the datastore performance is considered to be the best practice.

coolgod
Enthusiast
Posts: 58
Liked: never
Joined: Jun 27, 2013 6:00 am
Full Name: Vicky | techvicky.com
Location: MAURITIUS
Contact:

Re: snapshot consolidation fails after backup completes

Post by coolgod » Oct 07, 2014 1:12 pm

Hi,

I landed on the page (you mentioned) today during my research...thanks for the link.
My veeam backup server (physical) does proxy+repository+backup+replication.

I don't have additional source proxy.. the Veeam Backup server itself acts as proxy.

Please confirm if I manually change the proxy to use NBD transport mode on my Veeam Server, the issue will be solved.
Whatever the mind of man can conceive and believe, it can achieve. - Napoleon Hill

foggy
Veeam Software
Posts: 18439
Liked: 1588 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: snapshot consolidation fails after backup completes

Post by foggy » Oct 07, 2014 2:19 pm

No one can confirm this until the actual reason of this behavior is investigated (which would require deep logs analysis). Btw, here is also an existing topic discussing similar issues: Veeam temp snapshots not removed.

coolgod
Enthusiast
Posts: 58
Liked: never
Joined: Jun 27, 2013 6:00 am
Full Name: Vicky | techvicky.com
Location: MAURITIUS
Contact:

Re: snapshot consolidation fails after backup completes

Post by coolgod » Oct 08, 2014 1:26 pm

i noticed only one the VM which is quite bulky that experiences this issue..it is an SBS 2008 server.

It has SQL server/ exchange / AD/ File server / DHCP and other software. is it because of this ??
Whatever the mind of man can conceive and believe, it can achieve. - Napoleon Hill

foggy
Veeam Software
Posts: 18439
Liked: 1588 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: snapshot consolidation fails after backup completes

Post by foggy » Oct 08, 2014 3:36 pm

Only logs review can tell this for sure.

coolgod
Enthusiast
Posts: 58
Liked: never
Joined: Jun 27, 2013 6:00 am
Full Name: Vicky | techvicky.com
Location: MAURITIUS
Contact:

Re: snapshot consolidation fails after backup completes

Post by coolgod » Oct 09, 2014 5:30 am

please give me a hint which logs should i dig , the vmware ESXi logs or Veeam Logs.
Whatever the mind of man can conceive and believe, it can achieve. - Napoleon Hill

foggy
Veeam Software
Posts: 18439
Liked: 1588 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: snapshot consolidation fails after backup completes

Post by foggy » Oct 09, 2014 6:16 am

Vicky, technical support will be able to assist you in digging into the logs. Please contact them directly.

coolgod
Enthusiast
Posts: 58
Liked: never
Joined: Jun 27, 2013 6:00 am
Full Name: Vicky | techvicky.com
Location: MAURITIUS
Contact:

Re: snapshot consolidation fails after backup completes

Post by coolgod » Oct 09, 2014 8:49 am

please note that I am already liaising with veeam technical support.

but i also want to personally investigate deeper on this issue.

please analyze below what a vExpert on the vmware forum says:
This issue occurs because the consolidation attempt is made prior to the backup proxy lease being cleaned up.


The series of events should match:

Backup starts
Snapshot created for backup
Lease established
Failure of SAN mode, then fallback to NBD mode
Backup finishes
Consolidation attempt (fails)
Lease released

Note: The backup proxy must be a separate physical machine to trigger the issue; virtual machine based proxies cannot access SAN mode backups.

This is a known issue affecting the Virtual Disk Development Kit (VDDK) 5.0 / 5.1.


After the lease is released, normal consolidation is possible.


To work around this issue, configure the backup software to attempt NBD directly instead of attempting SAN-based backup first, or ensure the correct block devices are present on the backup proxy server prior to attempting a SAN-based backup.


Note: Auto mode attempts SAN-based backup first then falls back to NBD if that fails. This situation will also trigger the issue

VMware is planning to address the orphaned lease in a future release of the VDDK.
Please send me your comments and advices. thanks
Whatever the mind of man can conceive and believe, it can achieve. - Napoleon Hill

foggy
Veeam Software
Posts: 18439
Liked: 1588 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: snapshot consolidation fails after backup completes

Post by foggy » Oct 09, 2014 9:24 am

Well, you haven't provided your support case ID, so I was not aware of your ongoing investigation. I would appreciate if you post it here for reference.

VMware specialist response sounds reasonable. Have you tried switching to network mode already?

coolgod
Enthusiast
Posts: 58
Liked: never
Joined: Jun 27, 2013 6:00 am
Full Name: Vicky | techvicky.com
Location: MAURITIUS
Contact:

Re: snapshot consolidation fails after backup completes

Post by coolgod » Oct 09, 2014 10:01 am

Veeam Case # 00645776

Currently, i dont have any issue..but I have stopped the replication from PROD to DR site.

I have decided to investigate deeper why the VM continued to run on the Helper Snapshot causing the delta file to grow enormously. :evil:
This is a terrible nightmare if not monitored and discovered at the right time :wink:
Whatever the mind of man can conceive and believe, it can achieve. - Napoleon Hill

foggy
Veeam Software
Posts: 18439
Liked: 1588 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: snapshot consolidation fails after backup completes

Post by foggy » Oct 09, 2014 10:38 am

Ok, please keep us posted on your findings.

Once again, you can monitor snapshot size via free version of Veeam ONE and be notified when the snapshot grows large as well as be alerted on the latency issues that may affect snapshot consolidation process.

coolgod
Enthusiast
Posts: 58
Liked: never
Joined: Jun 27, 2013 6:00 am
Full Name: Vicky | techvicky.com
Location: MAURITIUS
Contact:

Re: snapshot consolidation fails after backup completes

Post by coolgod » Oct 09, 2014 11:05 am

Does Veeam One works in realtime or plot the graph every 5 minutes for the datastore disk space?
Whatever the mind of man can conceive and believe, it can achieve. - Napoleon Hill

foggy
Veeam Software
Posts: 18439
Liked: 1588 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: snapshot consolidation fails after backup completes

Post by foggy » Oct 09, 2014 11:29 am

Every 5 minutes.

coolgod
Enthusiast
Posts: 58
Liked: never
Joined: Jun 27, 2013 6:00 am
Full Name: Vicky | techvicky.com
Location: MAURITIUS
Contact:

Re: snapshot consolidation fails after backup completes

Post by coolgod » Oct 09, 2014 11:32 am

Thanks :)
Whatever the mind of man can conceive and believe, it can achieve. - Napoleon Hill

coolgod
Enthusiast
Posts: 58
Liked: never
Joined: Jun 27, 2013 6:00 am
Full Name: Vicky | techvicky.com
Location: MAURITIUS
Contact:

Re: snapshot consolidation fails after backup completes

Post by coolgod » Oct 10, 2014 6:26 am

if i want to disable Changed Block Tracking (CBT), should I disable it in Veeam or in VMWARE or in both ?
Whatever the mind of man can conceive and believe, it can achieve. - Napoleon Hill

foggy
Veeam Software
Posts: 18439
Liked: 1588 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: snapshot consolidation fails after backup completes

Post by foggy » Oct 10, 2014 6:32 am

Disable it in Veeam B&R, otherwise it will be re-enabled on the next job run.

coolgod
Enthusiast
Posts: 58
Liked: never
Joined: Jun 27, 2013 6:00 am
Full Name: Vicky | techvicky.com
Location: MAURITIUS
Contact:

Re: snapshot consolidation fails after backup completes

Post by coolgod » Oct 10, 2014 6:34 am

and this does not matter if on VMWARE it is enabled ?? please confirm
Whatever the mind of man can conceive and believe, it can achieve. - Napoleon Hill

coolgod
Enthusiast
Posts: 58
Liked: never
Joined: Jun 27, 2013 6:00 am
Full Name: Vicky | techvicky.com
Location: MAURITIUS
Contact:

Re: snapshot consolidation fails after backup completes

Post by coolgod » Oct 10, 2014 8:10 am

CBT Bug in VMWare - Explanation by Gostev.
For reference see this post.
Whatever the mind of man can conceive and believe, it can achieve. - Napoleon Hill

foggy
Veeam Software
Posts: 18439
Liked: 1588 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: snapshot consolidation fails after backup completes

Post by foggy » Oct 10, 2014 2:53 pm

coolgod wrote:and this does not matter if on VMWARE it is enabled ?? please confirm
VMware setting is ignored if CBT usage is disabled in the job settings.
coolgod wrote:CBT Bug in VMWare - Explanation by Gostev.
For reference see this post.
Not sure I follow you on this. Are you saying you're experiencing the referenced issue? I actually cannot connect it with the consolidation helper snapshot you've described originally.

coolgod
Enthusiast
Posts: 58
Liked: never
Joined: Jun 27, 2013 6:00 am
Full Name: Vicky | techvicky.com
Location: MAURITIUS
Contact:

Re: snapshot consolidation fails after backup completes

Post by coolgod » Oct 13, 2014 7:09 am

VMware setting is ignored if CBT usage is disabled in the job settings.
Do i need to worry about the ctk files that are created on the VM directory found on the datastore ? Do i need to remove
this ctk file?
Not sure I follow you on this. Are you saying you're experiencing the referenced issue? I actually cannot connect it with the consolidation helper snapshot you've described originally.
Don't worry about this one..I only posted for information purpose only..I though the bug in the VMWARE CBT feature might be the one causing this issue.
Whatever the mind of man can conceive and believe, it can achieve. - Napoleon Hill

foggy
Veeam Software
Posts: 18439
Liked: 1588 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: snapshot consolidation fails after backup completes

Post by foggy » Oct 13, 2014 7:47 am

coolgod wrote:Do i need to worry about the ctk files that are created on the VM directory found on the datastore ? Do i need to remove
this ctk file?
No, you do not worry about them. If the setting is not enabled, they are also not taken into account.

coolgod
Enthusiast
Posts: 58
Liked: never
Joined: Jun 27, 2013 6:00 am
Full Name: Vicky | techvicky.com
Location: MAURITIUS
Contact:

Re: snapshot consolidation fails after backup completes

Post by coolgod » Oct 14, 2014 11:37 am

I worked on a backup architecture that I want to implement. please advise.

[Veeam Backup Machine uses "VeeamProxy" VM as Proxy]<<<-------------[ESXi-PROD (VeeamProxy)(VM1)(VM2)]
|
|
|
[ESXi-DR]


On my Veeam Backup server, i will then use VeeamProxy as Proxy and I will choose transport mode as Virtual Appliance.

I forgot to ask..
I have only 1 datastore on my ESXi PROD.I dont have much free space on the datastore (datastore1) only around 300 GB. I am planning to add more hard disk on the machine and create a second datastore (datastore2). The VeeamProxy VM will then be located on datastore2.

I am asking this question because i read for the Virtual Appliance transport mode to work the proxy vm must be on the same ESXi host but am not sure whether it must also be on the same datastore. please confirm.
Whatever the mind of man can conceive and believe, it can achieve. - Napoleon Hill

foggy
Veeam Software
Posts: 18439
Liked: 1588 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: snapshot consolidation fails after backup completes

Post by foggy » Oct 14, 2014 12:42 pm

coolgod wrote:I worked on a backup architecture that I want to implement. please advise.
Looks good.
coolgod wrote:I am asking this question because i read for the Virtual Appliance transport mode to work the proxy vm must be on the same ESXi host but am not sure whether it must also be on the same datastore. please confirm.
Not necessarily on the same host/datastore. General requirement is: in order to use hotadd, proxy VM should have access to all datastores you're backing up VMs from.

In future, please do not derail this thread with unrelated messages, but rather create a separate topic for each question/issue. Thanks.

coolgod
Enthusiast
Posts: 58
Liked: never
Joined: Jun 27, 2013 6:00 am
Full Name: Vicky | techvicky.com
Location: MAURITIUS
Contact:

Re: snapshot consolidation fails after backup completes

Post by coolgod » Oct 15, 2014 9:23 am

Thanks man.
I will stick to what you said about posting rules.

Since i have more questions on Virtual Appliance Transport Mode I will create a new thread.
Whatever the mind of man can conceive and believe, it can achieve. - Napoleon Hill

foggy
Veeam Software
Posts: 18439
Liked: 1588 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: snapshot consolidation fails after backup completes

Post by foggy » Oct 15, 2014 9:40 am

Will be happy to help. Thanks for understanding.

Post Reply

Who is online

Users browsing this forum: Google [Bot] and 13 guests