VeeamFCT disables ODX

[jakesterpdx]

I've spent the last 3 days banging my head against this, so hopefully I save someone else the same headache. Here's the deal: if you're trying to use ODX on your SAN (for Storage Migrations, or even just simple file copy/move operations), VeeamFCT will disable ODX (http://msdn.microsoft.com/en-us/library ... 33784.aspx). You can check to see if you're impacted by running an elevanted PowerShell command of:

Fltmc instances -v <volume name>

Where <volume name> is the name of your volume. Assuming you're running a CSV, it would probably be something like:

Fltmc instances -v c:\clusterstorage\volume1

If you get back results like this where the SprtFtrs value is 0, then you won't be able to use ODX:
Filter Altitude Instance Name Frame SprtFtrs VlStatus
-------------------- ------------ ---------------------- ----- -------- --------
VeeamFCT 405100 TopInstance 0 00000000

So, the obvious two follow-ups are:
1. Why in the name of all that is holy wasn't this in the release notes somewhere? I had to use up a valuable support case with Microsoft to finally figure it out, and
2. When is this going to be fixed? I'm pretty sure that VeeamFCT is a requisite for allowing my nodes to have their VMs backed up, so I'm in a bad place: either no ODX (which is a fantastic feature that I paid dearly for in my EqualLogic array) or finding another backup solution, and I've been very, very happy with Veeam up until this point.

[jake1110]

This is very bad news for us. We actually just disabled ODX on our cluster by recommendation from Microsoft. We had a MS Premier call with Microsoft as well due to issues we were seeing with ODX. One of the recommendations was to completely disable ODX until they get a hotfix built, but if Veeam is breaking ODX as well, I won't even bother enabling it until I hear it's been resolved in Veeam.

[jakesterpdx]

Yikes - what other issues with ODX were you seeing? Have you applied the new-new hotfix http://support.microsoft.com/kb/2813630 for it yet?
I really don't want to lose ODX - it cuts Storage Migration times in half, and, instead of flooding the iSCSI network during the transfer, hardly even touches it.

[jake1110]

Performance-wise, our transfer times haven't been too adversely affected but we think it's because we're on a 10gb network. We too, use Dell Equallogic SANs.

We were experiencing very odd performance fluctuations. Every so often, the entire cluster would just act up and become dirt slow. It's something that we have been working with MS on for a long time and we've had trouble pinpointing exactly what's happened. We do work a lot with MS though on numerous things, considering that we always early adopt their products and wind up testing for them, lol. Apparently, the ODX woes are more widespread than they expected, so we're filling out some detailed forms for them that outlines our infrastructure to help them get a better idea of what's going on.

Guess I'll wait to see how Veeam responds before enabling it. Hopefully this is resolved soon.

[andersonts]

Jake (Harris) have you opened up a ticket with us yet on this? If so can you provide the case number? If not can you please get one open and then provide the case number? Jake Ernst same thing when you get the chance please. We just want to make sure these are tracked internally as well. Thanks!

[jakesterpdx]

Mine is case # 00187675. By the way, folks to really take kindly to others putting the full names on public forum without first getting permission.

[andersonts]

Hi Jake if you look to the right of anyone's post you will see the following... looks like you have supplied this information while registering on the forum. I certainly apologize for any inconvenience. Thanks for the case number!

[jakesterpdx]

Oh! Looky there. Never noticed it. Sorry about that!

[andersonts]

No worries sir!

[patrick.shea]

You should be able to change what shows up in the Full Name section in the user control panel

[Gostev]

Hi everyone, I just wanted to provide the update on our research progress. While we do not specifically disable ODX, it looks like every mini-filter driver needs to specifically “opt-in” to support ODX, this is something we are not doing today. This part is very simple, however we are now focused on researching internals of ODX technology to see potential impact on our change tracking. If the research results are positive, we will include support for ODX in Patch 3 for version 6.5, or in version 7.0 (depending on the amount of work required). Thank you for your patience.

[jakesterpdx]

I'm happy to report that Patch 3 for Backup and Recovery 6.5 resolves this issue, and installing the patch is incredibly easy and simple.

Which brings me to the question: Is there some way to mark a post as "resolved" or something?

[Vitaliy S.]

Just did it

[Gostev]

Actually, Patch #3 was NOT supposed to add ODX support. What you are seeing is unexpected, and most likely has nothing to deal with the patch. I have forwarded your support case ID above to our support manager requesting that they take a look at your environment more closely.

At this time, you don't want to have ODX enabled. Our research have showed that ODX brakes some B&R functionality. Supporting ODX requires some fairly significant changes in the code for a few features. Thus, we have decided to postpone that, not to affect Patch #3 delivery schedule.

Right now, we are planning to introduce ODX support as a fix for Patch #3. We are currently working on this, and we will follow up with everyone who has a support case on ODX open as soon as the fix is ready. You will need to have Patch #3 installed before applying the fix.

[jakesterpdx]

Hmmm - you guys need to get your signals uncrossed. Here's the message I got from Cody in Tier 3 support:

"We received the hotfix patch today which should fix the ODX/Veeam FCT issue. You should be able to download this and install, when you open the console, it will ask to upgrade the host components as well. Let us know if you have any issues."

I installed it, and ODX is working. That said, my backup job that usually takes 5 hours took 12 last night, so...

[Ben Milligan]

Hi Jake, my name is Ben Milligan, I am the Director of North American Customer Support with Veeam. Cody reports to me. I greatly apologize for this miscommunication, and take responsibility for it. It certainly was unintended and we are dedicated to getting this working for you. We will continue working very hard on this and hopefully have a full resolution in the near future. Our engineer has reopened this case to ensure we are following up and through.

[jakesterpdx]

Thanks - I would appreciate a response ASAP, as on the one hand this seems to have resolved my issues, but on the other if I'm actually at risk for corruption, it's not worth it!

[Ben Milligan]

I agree 100% and it is of up most priority.

[jakesterpdx]

Can I get an update on this, please?

[Gostev]

The coding is done from R&D perspective, and the fix was submitted into QC for testing. Current estimate for release date is mid-May time frame. Since you have the support case open, you will be notified on the fix availability immediately by your support engineer.

[jakesterpdx]

I'm confused - ODX seems to be working fine now that I've applied Update 3. What fix is being submitted exactly?

[Vitaliy S.]

It might be just a coincidence (patch #3). The fix for the ODX is now validated by our QC team.

[Gostev]

What fix is being submitted exactly?

What is being fixed are things that enabled ODX brakes in our product.

[Ben Milligan]

To anyone else that experiences this problem, please open a case with our support (http://www.veeam.com/support.html). We now have a fix available for this problem and will gladly assist!

[jakesterpdx]

I deployed this yesterday and it went smoothly. Since it's a hotfix of sorts, there are a number of steps to follow, but it's not too bad.