Host-based backup of VMware vSphere VMs.
mkretzer
Veeam Legend
Posts: 1140
Liked: 387 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: SQL Server data inconsistencies with snapshot - ESX 7U2, latest patches

Post by mkretzer »

@Ioannis.T Bloom filter is currently the suspect for causing the data inconsistency. We are talking about a bug here which needs (and will be) fixed by VMware. Disabling the bloom filter is the current workaround. Since the issue seems to be highly IO pattern dependend and the inconsistency does not seem to be persistent after snapshot removal it does not make alot of sense to disable it without having any issue...
BackItUp2020
Enthusiast
Posts: 54
Liked: 3 times
Joined: Mar 24, 2020 6:36 pm
Full Name: M.S.
Contact:

Re: SQL Server data inconsistencies with snapshot - ESX 7U2, latest patches

Post by BackItUp2020 »

FWIW, I logged a case today w/ VMware and received the same workaround, but it lacked specifics on how to run the command.

1. Power off the SQL VM
2. Disable the bloom filter by setting the config option
/config/SE/intOpts/BFEnabled to 0
3. Turn on the VM
mkretzer
Veeam Legend
Posts: 1140
Liked: 387 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: SQL Server data inconsistencies with snapshot - ESX 7U2, latest patches

Post by mkretzer » 1 person likes this post

No, just set the option, all will be fine without any restarts whatsoever!
juanmachado
Enthusiast
Posts: 29
Liked: 8 times
Joined: Nov 18, 2020 3:59 pm
Full Name: Juan Machado
Contact:

Re: SQL Server data inconsistencies with snapshot - ESX 7U2, latest patches

Post by juanmachado »

BackItUp2020 wrote: Apr 21, 2022 8:19 pm FWIW, I logged a case today w/ VMware and received the same workaround, but it lacked specifics on how to run the command.

1. Power off the SQL VM
2. Disable the bloom filter by setting the config option
/config/SE/intOpts/BFEnabled to 0
3. Turn on the VM
@BackItUp2020
Can you please share your Vmware support case number? We have the same behavior and I am opening a case with Vmware. Thanks!
BackItUp2020
Enthusiast
Posts: 54
Liked: 3 times
Joined: Mar 24, 2020 6:36 pm
Full Name: M.S.
Contact:

Re: SQL Server data inconsistencies with snapshot - ESX 7U2, latest patches

Post by BackItUp2020 » 1 person likes this post

These forum controls are being janky. I was trying to quote you, but I don't have the option.

Case: 22323585704
juanmachado
Enthusiast
Posts: 29
Liked: 8 times
Joined: Nov 18, 2020 3:59 pm
Full Name: Juan Machado
Contact:

Re: SQL Server data inconsistencies with snapshot - ESX 7U2, latest patches

Post by juanmachado »

Thank you @BackItUp2020 !

I also got a case with vmware now SR 22324270104. They confirmed this is a bug, the root cause is not yet know, as per VMWARE PR 2927179.

Here are the host cli commands you are looking for if you need to disable the BLOOM filter while this bug is resolved:

Set the following config option. This disables the sesparse bloom filter feature.
vsish -e set /config/SE/intOpts/BFEnabled 0

Retrieve the value of config option BFEnabled
vsish -e get /config/SE/intOpts/BFEnabled
kfeps
Novice
Posts: 4
Liked: 3 times
Joined: Apr 13, 2022 7:49 am
Full Name: Kenny Fortuin
Contact:

Re: SQL Server data inconsistencies with snapshot - ESX 7U2, latest patches

Post by kfeps »

Adding to this thread, we too have been experiencing this issue for several months. Our VMware SR is 22320596404 for the sake of others to reference. The same workaround is suggested as in this thread, with the addition to automate the command to run at boot of the ESXi hosts.

Add the command to /etc/rc.local.d/local.sh to be run automatically on boot. (For full details, see https://kb.vmware.com/s/article/2043564?lang=en_US).

VMware support confirm the reproduction of the issue in their lab environment. We are awaiting the official patch release date. I will update with anything new once received back from VMware support.
Andreas Neufert
VP, Product Management
Posts: 6707
Liked: 1401 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert
Location: Germany
Contact:

Re: SQL Server data inconsistencies with snapshot - ESX 7U2, latest patches

Post by Andreas Neufert »

I am curious, is this issue when you use VMware Tools quiescence or on Crash consistent snapshots as well?
mkretzer
Veeam Legend
Posts: 1140
Liked: 387 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: SQL Server data inconsistencies with snapshot - ESX 7U2, latest patches

Post by mkretzer »

@Andreas Neufert
Irrelevant. Purely a bug in the IO logic / the bloom filter when reading snapshots
kfeps
Novice
Posts: 4
Liked: 3 times
Joined: Apr 13, 2022 7:49 am
Full Name: Kenny Fortuin
Contact:

Re: SQL Server data inconsistencies with snapshot - ESX 7U2, latest patches

Post by kfeps » 2 people like this post

Latest update:
No official patch release date, however VMware support has a tentative release timeframe aiming for Q3. The engineering team is targeting the change for patch release 7.0 P05. Nothing sooner announced as yet unfortunately and the workaround is still suggested for any Production workloads experiencing the issue. I'm hoping there are no long term effects on the SQL servers experiencing this issue. I have raised this concern with VMware support as the tentative release timeframe is quite far out.
kfeps
Novice
Posts: 4
Liked: 3 times
Joined: Apr 13, 2022 7:49 am
Full Name: Kenny Fortuin
Contact:

Re: SQL Server data inconsistencies with snapshot - ESX 7U2, latest patches

Post by kfeps » 1 person likes this post

Official VMware KB article published for this issue:
https://kb.vmware.com/s/article/88201
Workaround still suggested for Production workloads experiencing the issue and patch ETA is Q3 2022.
mkretzer
Veeam Legend
Posts: 1140
Liked: 387 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: SQL Server data inconsistencies with snapshot - ESX 7U2, latest patches

Post by mkretzer »

The update which disables the bloom filter just came out!
PetrM
Veeam Software
Posts: 3229
Liked: 520 times
Joined: Aug 28, 2013 8:23 am
Full Name: Petr Makarov
Location: Prague, Czech Republic
Contact:

Re: SQL Server data inconsistencies with snapshot - ESX 7U2, latest patches

Post by PetrM »

Hello,

I guess you're talking about ESXi 7.0 Update 3f ?

Thanks!
kfeps
Novice
Posts: 4
Liked: 3 times
Joined: Apr 13, 2022 7:49 am
Full Name: Kenny Fortuin
Contact:

Re: SQL Server data inconsistencies with snapshot - ESX 7U2, latest patches

Post by kfeps »

Looks like VMware still doing investigations though as there is not a true fix to this issue.

As per release notes of ESXi 7.0 Update 3f
This issue is resolved in this release. The fix disables the use of Bloom Filter functionality until the root cause of the issue is resolved.
Post Reply

Who is online

Users browsing this forum: No registered users and 93 guests