SQL Server data inconsistencies with snapshot - ESX 7U2, latest patches

[mkretzer]

@Ioannis.T Bloom filter is currently the suspect for causing the data inconsistency. We are talking about a bug here which needs (and will be) fixed by VMware. Disabling the bloom filter is the current workaround. Since the issue seems to be highly IO pattern dependend and the inconsistency does not seem to be persistent after snapshot removal it does not make alot of sense to disable it without having any issue...

[BackItUp2020]

FWIW, I logged a case today w/ VMware and received the same workaround, but it lacked specifics on how to run the command.

1. Power off the SQL VM
2. Disable the bloom filter by setting the config option
/config/SE/intOpts/BFEnabled to 0
3. Turn on the VM

[mkretzer]

No, just set the option, all will be fine without any restarts whatsoever!

[juanmachado]

FWIW, I logged a case today w/ VMware and received the same workaround, but it lacked specifics on how to run the command.

1. Power off the SQL VM
2. Disable the bloom filter by setting the config option
/config/SE/intOpts/BFEnabled to 0
3. Turn on the VM

@BackItUp2020
Can you please share your Vmware support case number? We have the same behavior and I am opening a case with Vmware. Thanks!

[BackItUp2020]

These forum controls are being janky. I was trying to quote you, but I don't have the option.

Case: 22323585704

[juanmachado]

Thank you @BackItUp2020 !

I also got a case with vmware now SR 22324270104. They confirmed this is a bug, the root cause is not yet know, as per VMWARE PR 2927179.

Here are the host cli commands you are looking for if you need to disable the BLOOM filter while this bug is resolved:

Set the following config option. This disables the sesparse bloom filter feature.
vsish -e set /config/SE/intOpts/BFEnabled 0

Retrieve the value of config option BFEnabled
vsish -e get /config/SE/intOpts/BFEnabled

[kfeps]

Adding to this thread, we too have been experiencing this issue for several months. Our VMware SR is 22320596404 for the sake of others to reference. The same workaround is suggested as in this thread, with the addition to automate the command to run at boot of the ESXi hosts.

Add the command to /etc/rc.local.d/local.sh to be run automatically on boot. (For full details, see https://kb.vmware.com/s/article/2043564?lang=en_US).

VMware support confirm the reproduction of the issue in their lab environment. We are awaiting the official patch release date. I will update with anything new once received back from VMware support.

[Andreas Neufert]

I am curious, is this issue when you use VMware Tools quiescence or on Crash consistent snapshots as well?

[mkretzer]

@Andreas Neufert
Irrelevant. Purely a bug in the IO logic / the bloom filter when reading snapshots

[kfeps]

Latest update:
No official patch release date, however VMware support has a tentative release timeframe aiming for Q3. The engineering team is targeting the change for patch release 7.0 P05. Nothing sooner announced as yet unfortunately and the workaround is still suggested for any Production workloads experiencing the issue. I'm hoping there are no long term effects on the SQL servers experiencing this issue. I have raised this concern with VMware support as the tentative release timeframe is quite far out.

[kfeps]

Official VMware KB article published for this issue:
https://kb.vmware.com/s/article/88201
Workaround still suggested for Production workloads experiencing the issue and patch ETA is Q3 2022.

[mkretzer]

The update which disables the bloom filter just came out!

[PetrM]

Hello,

I guess you're talking about ESXi 7.0 Update 3f ?

Thanks!

[kfeps]

Looks like VMware still doing investigations though as there is not a true fix to this issue.

As per release notes of ESXi 7.0 Update 3f
This issue is resolved in this release. The fix disables the use of Bloom Filter functionality until the root cause of the issue is resolved.