Issue with Windows Server 2012 R2 Update (April 2014 rollup)

[rskinner]

Having installed the recently released Windows Server 2012 R2 Update, AKA Windows 8.1 Update 1 KB 2919355, my Veeam Hyper-V backups failed with errors as shown below.

Code: Select all

03/04/2014 20:58:28 :: Processing 'SERVERNAME' Error: Client error: The system cannot find the file specified.
Failed to open file [\\?\Volume{36db85be-bb6a-11e3-80ce-000af72f0333}\SERVERNAMEWINDOWS SERVER 2012 R2 STANDARD_DISK_1_117A5A0B-5976-4766-B256-3707A2D8CFA7-AutoRecovery.avhdx] in readonly mode.

The engineer (Anatoly) was excellent and was able to prove that the error wasn't related to Veeam by running Windows backup commands using PowerShell. Following the uninstall of KB 2919355, the only one of the six hot fixes comprising the Windows Server 2012 R2 Update listed in Add Remove Programs, I was able to run backups as before. If you're thinking of installing this update, ensure you test it well.

For reference the configuration is:

4x Hyper-V 2012 R2 hosts
Dell EqualLogic SAN 6.0.7
HIT Kit 4.7 EPA
Veeam Backup & Replication 7.0.0.839

[m.novelli]

Any info if this bug raise only on Hyper-V hosts or also on backup/proxy servers? I've migrated many Veeam Backup servers to Windows 2012 R2 in the last months and they have Windows Updated active

Thanks!

[Gostev]

Hi Marco, you are fine. This bug is specific to Hyper-V virtual machine backup from Hyper-V 2012 R2 (once the update is installed, Hyper-V starts to return a different path to one of the VM files). Thanks!

[m.novelli]

[Susanb]

Are you guys working with Microsoft to resolve this? This update has to be installed otherwise these servers won't get future security updates starting in May. We have 30 days to deploy that patch.

If you don't have contacts with Microsoft, let me know. There's a time sensitivity to this update.

[m.novelli]

Don't worry, Veeam has strict partnership with Microsoft and is super fast in hotfix release (Veeam, not Microsoft)

BTW we should speak about a workaround, not an hotfix, because the bug has been introduced by Microsoft and not by Veeam

[JRH]

Hi,

Just to add that we have the same issue on our 2012 R2 cluster (with the new updates) and currently have a support ticket open with Veeam.

Has anyone had any update or ETA from Microsoft?

[rskinner]

I haven't opened a call with MS. The problem is that I've had to put the cluster into production, so I can no longer carry out testing etc.

JRH - what storage are you using? I'm just wondering if it's related to the Dell EqualLogic VSS provider.

[Gostev]

All, our devs worked on Saturday to produce the hot fix. We will now need a couple of days for full regressive testing, and thus are hoping to make it available by the Patch Tuesday next week (which is when this will hit Windows Update). Thanks!

[JRH]

Hi,

It's not an EqualLogic issue as I'm running a Nimble SAN with a software VSS - I've had to uninstall the windows update, but now have working backups again, hopefully Veeam release the fix soon and I will reinstall the windows update.

[Susanb]

Kudos to the Veeam folks for being very responsive to this issue.

[Shralok]

This thread was a great find. I will be holding off on this KB 2919355 until there is a resolution for Veeam. Thanks to all, especially devs for working on Saturday

[Gostev]

QC has signed off the hot fix for distribution, so you should be able to obtain it through support at this time, and I am sure they will create a KB article tomorrow. Thanks!

[kasacompanies]

Can I get a link to the KB article? I can't seem to find it. Thanks

[Vitaliy S.]

There is no KB yet, I believe our support team is still working on it. If you want to get the hotfix, then contact our technical team directly for now.

[Gostev]

Just a head up, our QC has found another issue with this update. This second issue is completely unrelated to the previous one, and has to deal with Windows 2008 and earlier VMs.

Quick summary of all known issues so far:
1. Backing up Windows 2008R2 and later VMs on Hyper-V 2012 R2 hosts fails after update. Hotfix for this issue is already available from Veeam support.
2. Restoring Windows 2008 and earlier backups made on updated Hyper-V 2012 R2 hosts fails. No solution at the moment (my contact at Microsoft has just confirmed the issue present).

I will keep providing updates on the situation in this topic. You can subscribe to notifications by clicking the link at the bottom left of this page.

[Mrdoff]

Any news yet? I started a support chat and asked about the KB but they only gave me this link...

[Gostev]

KB was taken offline to add a note about issue #2, but now it is back online > http://www.veeam.com/kb1863

[Gostev]

Hot fix for the 2nd issue has been implemented, and put into automated test lab for overnight stress testing. If all goes well, we will start pilot deployments tomorrow morning.

[Gostev]

Download in the KB article above has been updated to include fixes for both issues.

[aeccles]

The KB is missing the file path on step 5. It's: C:\Program Files\Veeam\Backup and Replication\Backup

Also, the KB doesn't state what type of backup is required in Step 7. Is an incremental sufficient or does it need to be a full?

[Gostev]

Incremental run is sufficient. I have asked support to update the article. Thank you!

[aeccles]

Great thanks, Gostev

[KelKnowles]

Have applied the hotfix. Works like a charm.

[frankive]

For me the path is C:\Program Files\Veeam\Backup and Replication\
and not C:\Program Files\Veeam\Backup and Replication\backup
why is that?

[foggy]

Frank, what path are you referring to? The fixed DLL files should be placed in the C:\Program Files\Veeam\Backup and Replication\Backup folder, there are no DLL files in the parent directory.

[sebastien]

Hi,
I have the same issue on a server but can't download the hotfix. The ftp link provided in the KB (http://www.veeam.com/kb1863) doesn't work for me...

[foggy]

Sebastien, this hotfix was included into Veeam B&R patch 4, please consider upgrading.

[sebastien]

Thanks a lot, it worked perfectly !!