snapshot consolidation fails after backup completes

[coolgod]

Hi All,

I have also been searching for the solution on the vmware forum.

My infrastructure consists of Veeam BR 7 , 1 ESXi-PROD and 1 ESXi-DR.


ESXi-PROD--------------->Veeam BR7-------------->ESXi-DR


After backup was complete, I noticed the VM was running on helper snapshot...this snapshot was unusually growing bigger.

So at the right time, i noticed this and stopped the Veeam Job. Then I manually did a DELETE ALL from the snapshot manager.

The consolidation was successfully completed.

Apart from monitoring the datastore freespace, I want to know the exact cause of this issue.
This is critical because if not monitored, the helper snapshot will grow until it uses all the free disk space on the datastore.

Please advise.

[Dima P.]

Hello Vicky,
You may want to troubleshoot this issues with support team, however there was a really good KB article, describing possible reasons and troubleshooting steps of your issue: VM Loses Connection During Snapshot Removal

Additionally, using Veeam ONE free edition for monitoring the datastore performance is considered to be the best practice.

[coolgod]

Hi,

I landed on the page (you mentioned) today during my research...thanks for the link.
My veeam backup server (physical) does proxy+repository+backup+replication.

I don't have additional source proxy.. the Veeam Backup server itself acts as proxy.

Please confirm if I manually change the proxy to use NBD transport mode on my Veeam Server, the issue will be solved.

[foggy]

No one can confirm this until the actual reason of this behavior is investigated (which would require deep logs analysis). Btw, here is also an existing topic discussing similar issues: Veeam temp snapshots not removed.

[coolgod]

i noticed only one the VM which is quite bulky that experiences this issue..it is an SBS 2008 server.

It has SQL server/ exchange / AD/ File server / DHCP and other software. is it because of this ??

[foggy]

Only logs review can tell this for sure.

[coolgod]

please give me a hint which logs should i dig , the vmware ESXi logs or Veeam Logs.

[foggy]

Vicky, technical support will be able to assist you in digging into the logs. Please contact them directly.

[coolgod]

please note that I am already liaising with veeam technical support.

but i also want to personally investigate deeper on this issue.

please analyze below what a vExpert on the vmware forum says:

This issue occurs because the consolidation attempt is made prior to the backup proxy lease being cleaned up.


The series of events should match:

Backup starts
Snapshot created for backup
Lease established
Failure of SAN mode, then fallback to NBD mode
Backup finishes
Consolidation attempt (fails)
Lease released

Note: The backup proxy must be a separate physical machine to trigger the issue; virtual machine based proxies cannot access SAN mode backups.

This is a known issue affecting the Virtual Disk Development Kit (VDDK) 5.0 / 5.1.


After the lease is released, normal consolidation is possible.


To work around this issue, configure the backup software to attempt NBD directly instead of attempting SAN-based backup first, or ensure the correct block devices are present on the backup proxy server prior to attempting a SAN-based backup.


Note: Auto mode attempts SAN-based backup first then falls back to NBD if that fails. This situation will also trigger the issue

VMware is planning to address the orphaned lease in a future release of the VDDK.

Please send me your comments and advices. thanks

[foggy]

Well, you haven't provided your support case ID, so I was not aware of your ongoing investigation. I would appreciate if you post it here for reference.

VMware specialist response sounds reasonable. Have you tried switching to network mode already?

[coolgod]

Veeam Case # 00645776

Currently, i dont have any issue..but I have stopped the replication from PROD to DR site.

I have decided to investigate deeper why the VM continued to run on the Helper Snapshot causing the delta file to grow enormously.
This is a terrible nightmare if not monitored and discovered at the right time

[foggy]

Ok, please keep us posted on your findings.

Once again, you can monitor snapshot size via free version of Veeam ONE and be notified when the snapshot grows large as well as be alerted on the latency issues that may affect snapshot consolidation process.

[coolgod]

Does Veeam One works in realtime or plot the graph every 5 minutes for the datastore disk space?

[foggy]

Every 5 minutes.

[coolgod]

Thanks

[coolgod]

if i want to disable Changed Block Tracking (CBT), should I disable it in Veeam or in VMWARE or in both ?

[foggy]

Disable it in Veeam B&R, otherwise it will be re-enabled on the next job run.

[coolgod]

and this does not matter if on VMWARE it is enabled ?? please confirm

[coolgod]

CBT Bug in VMWare - Explanation by Gostev.
For reference see this post.

[foggy]

and this does not matter if on VMWARE it is enabled ?? please confirm

VMware setting is ignored if CBT usage is disabled in the job settings.

CBT Bug in VMWare - Explanation by Gostev.
For reference see this post.

Not sure I follow you on this. Are you saying you're experiencing the referenced issue? I actually cannot connect it with the consolidation helper snapshot you've described originally.

[coolgod]

VMware setting is ignored if CBT usage is disabled in the job settings.

Do i need to worry about the ctk files that are created on the VM directory found on the datastore ? Do i need to remove
this ctk file?

Not sure I follow you on this. Are you saying you're experiencing the referenced issue? I actually cannot connect it with the consolidation helper snapshot you've described originally.

Don't worry about this one..I only posted for information purpose only..I though the bug in the VMWARE CBT feature might be the one causing this issue.

[foggy]

Do i need to worry about the ctk files that are created on the VM directory found on the datastore ? Do i need to remove
this ctk file?

No, you do not worry about them. If the setting is not enabled, they are also not taken into account.

[coolgod]

I worked on a backup architecture that I want to implement. please advise.

[Veeam Backup Machine uses "VeeamProxy" VM as Proxy]<<<-------------[ESXi-PROD (VeeamProxy)(VM1)(VM2)]
|
|
|
[ESXi-DR]


On my Veeam Backup server, i will then use VeeamProxy as Proxy and I will choose transport mode as Virtual Appliance.

I forgot to ask..
I have only 1 datastore on my ESXi PROD.I dont have much free space on the datastore (datastore1) only around 300 GB. I am planning to add more hard disk on the machine and create a second datastore (datastore2). The VeeamProxy VM will then be located on datastore2.

I am asking this question because i read for the Virtual Appliance transport mode to work the proxy vm must be on the same ESXi host but am not sure whether it must also be on the same datastore. please confirm.

[foggy]

I worked on a backup architecture that I want to implement. please advise.

Looks good.

I am asking this question because i read for the Virtual Appliance transport mode to work the proxy vm must be on the same ESXi host but am not sure whether it must also be on the same datastore. please confirm.

Not necessarily on the same host/datastore. General requirement is: in order to use hotadd, proxy VM should have access to all datastores you're backing up VMs from.

In future, please do not derail this thread with unrelated messages, but rather create a separate topic for each question/issue. Thanks.

[coolgod]

Thanks man.
I will stick to what you said about posting rules.

Since i have more questions on Virtual Appliance Transport Mode I will create a new thread.

[foggy]

Will be happy to help. Thanks for understanding.