I/O device error / Cannot get file size - 9.5 Related?

[BGA-Robert]

Hello All,

Veeam support request 01993924

I've rolled out our first 9.5 upgrade. First night, we had two VM's error on the nightly backups.

Error messages look like this:
"Error: The request could not be performed because of an I/O device error. WINAPI: Cannot get file size. File: [Device '\\.\PhysicalDrive8']. Failed to upload disk. Agent failed to process method {DataTransfer.SyncDisk}."

The two VM's succeeded on retry.

The next night, all were successful, but the following night, 3 VM's failed in the nightly backup with similar errors. All three succeeded on retry.

Veeam support picked this out of the logs:
[01.12.2016 19:09:24] <45> Error WINAPI: Cannot get file size. File: [Device '\\.\PhysicalDrive5']. (Veeam.Backup.Common.CCppComponentException)
[01.12.2016 19:09:24] <45> Error in c++: Error code: 0x0000045d

Support is still looking at it.

But I'm wondering if anybody has seen this as a 9.5 upgrade related issue.
It pretty certainly started after the 9.5 upgrade. Prior I've had no errors backing these VM's up.

These are all Windows servers, running under VMware.

My major difficulty at this point is we need proof of success on the 9.5 upgrade cycle before deploying to more critical environments.
So I need a root cause determination and some proof at this point that I'm good to go on upgrade to 9.5 for the rest of our clients.

Thanks!

[Gostev]

Well, standard Windows API function failing to get something from the file system really does not look like a Veeam issue, but a lot like a corrupt file system issue... but let's see what support hears back from the devs.

Answering your last question, after the current number of downloads you can be fairly confident that 9.5 is a solid release and does not have major engine issues. Based on our experience with the previous releases, we know most of the real issues always come out en masse during the very first week and result in dozens of identical support cases created in a matter of a few days.

[BGA-Robert]

Support found that the error was on the Veeam server, acting as the Veeam proxy.

We found a couple of matching event log entries on the Veeam server indicating the LSI_SAS driver was having a problem

Code: Select all

Log Name:      System
Source:        LSI_SAS
Date:          12/10/2016 7:07:32 PM
Event ID:      11
Description:
The driver detected a controller error on \Device\RaidPort0.

Code: Select all

Log Name:      System
Source:        LSI_SAS
Date:          12/10/2016 7:07:29 PM
Event ID:      129
Description:
Reset to device, \Device\RaidPort0, was issued.

VMware had a match on the error:
https://kb.vmware.com/selfservice/micro ... Id=2063346
But our driver was newer 1.34.3.82 vs. what was referenced in the KB, 1.34.02

Since VMware should be providing that driver, we simply uninstalled VMware tools and re-installed, with a reboot.

The problem was a bit random and intermittent, so we're taking a wait-and-see to see if this fixes the issue. But we've gotten 5 backups since with no errors.

[BGA-Robert]

Just a quick update that this issue persists for us. We've updated the VMware environment, thinking it may be a VMware bug. No change.

The errors occur pretty randomly, but somewhat infrequently, but they do continue to occur. And only on this one site (ours - LOL). Since installing 9.5 in December, we've had jobs fail 14 times, across a random assortment of about 1/2 dozen VMs. All always succeed on retry.

We've actually had a rash of Veeam issues and this one is not our highest priority. I hope to come back to it soon and get it resolved.

[Gostev]

Thanks for the update!

Since 9.5 is times more efficient than previous versions at data processing, the extra load on the environment may cause things to break where they are thin (like malfunctioning hardware or drivers). On the other hand, there are also benefits in detecting these issues early instead of having them silently doing bad things to your backups.

[DaveWatkins]

Can you not switch over to the paravirtual driver for VMWare? That'd take the LSI driver out of the equation entirely.

It's a slightly messy process of adding it and a new disk onto it then booting to get he driver loaded for it before you can change the boot drives controller, but it's documented in plenty of places on how to do the switch

[Nadziratelb]

Hello
And we have this same problem on random virtual machines.
Total virtual machines 500, errors appear regularly after installing version 9.5

03.02.2017 1:28:38 :: Error: The request could not be performed because of an I/O device error.
WINAPI: Cannot get file size. File: [Device '\\.\PhysicalDrive1'].
Failed to upload disk.
Agent failed to process method {DataTransfer.SyncDisk}.

[Didi7]

Hello,

we also have random problems with errors like ...

Code: Select all

*******************************************************************************************************************
Processing XXX Error: The request could not be performed because of an I/O device error. WINAPI: Cannot get file size. File: [Device '\\.\PhysicalDrive5']. Failed to upload disk. Agent failed to process method {DataTransfer.SyncDisk}. 
Processing XXX Error: The request could not be performed because of an I/O device error. WINAPI: Cannot get file size. File: [Device '\\.\PhysicalDrive10']. Failed to upload disk. Agent failed to process method {DataTransfer.SyncDisk}. 

or

Code: Select all

Error: The request could not be performed because of an I/O device error. WINAPI: Cannot get file size. File: [Device '\\.\PhysicalDrive2']. Failed to upload disk. Agent failed to process method {DataTransfer.SyncDisk}. Exception from server: The request could not be performed because of an I/O device error. WINAPI: Cannot get file size. File: [Device '\\.\PhysicalDrive2']. Unable to retrieve next block transmission command. Number of already processed blocks: [9431]. Failed to download disk. 

*******************************************************************************************************************

The corresponding VMs were retried and everything worked fine then.

The days before everything worked fine as well, even the full backup on friday.

More than one week before, we got similar Errors like above and the following ...

Code: Select all

Unable to release guest. Error: Failed to process 'GetVssSnapshotException' command. Failed to obtain error description for VSS snapshot job. There is no job state for the specified job ID: [{9580d99e-0e42-46b1-b649-efd0c70a34d2}].
Error: Failed to process 'GetVssSnapshotException' command. Failed to obtain error description for VSS snapshot job. There is no job state for the specified job ID: [{9580d99e-0e42-46b1-b649-efd0c70a34d2}].

*******************************************************************************************************************

The errors started to appear with VBR 9.5 here as well. Something is fishy here!

Regards,
Didi7

[Didi7]

Today everything worked find agagin! Will post constantly now, as I suppose it has something to do with v9.5 or v9.5 Update 1.

With VBR 9.0 I never had those Errors!

Regards,
Didi7

[BGA-Robert]

We continue to see this error intermittently, maybe about once a week. Different jobs / different VMs. I've been busy with other Veeam support requests. When I get caught up, I'll re-engage support on this issue.

[Didi7]

I can confirm the same behavior here ...

Errors intermittently, maybe about once a week, different jobs, different VMs

and supposedly beginning with VBR 9.5 or VBR 9.5 Update 1. Not sure!

Needs to be further investigated here as well.

Regards,
Didi7

[Nadziratelb]

At us such error began to appear after installation 9.5.
Update 1 did not correct this error.

[Mike Resseler]

To everyone that has this issue. Do you all have a support case logged? If there are more we can use those. The more logs can be investigated the better to find the final root cause

Thanks
Mike

[Nadziratelb]

I applied the logs to the case № 02058418
The case is closed, but the problem remains.

[crackocain]

Case #02110634 was created.

Timed out waiting for asynchronous device I/O completion

We are experiencing this issue Wan Accelerator Replication jobs or backup jobs.

[Didi7]

The word from Gostev:
Just before the weekend, VMware has issued this rush post regarding an issue with VMware Tools versions 9.10.0 up to 10.1.5 that causes network packets to drop due to the fact that all network traffic is serviced by a single guest CPU. It immediately occurred to me that the issue may be especially impactful on virtual backup proxies, because they pump hundreds of MB per second through the network stack – while their CPU is already very busy with all the data processing. So, I thought that this bug could potentially be the reason for those intermittent backup job failures some customers have been having for a while now (the issue where job retry always helps)? I guess we'll find out soon, as soon as VMware Tools are updated.

*************************************************************************************************

Interesting! I just wonder, why this hasn't been a problem with Veeam B&R 9.0 ?

Regards,
Didi7

[Didi7]

Hello again,

just for your information. Here in our environment, VMware Tools 9.4.10 build-2068191 from ESXi 5.5 Update 2 is used since November 2014 and we never had such problems with Veeam B&R 7, 8 and 9 but with 9.5 !!!

Regards,
Didi7

[Didi7]

Severe I/O errors again this weekend. 3 VMs of 12 in one backup job reported ...

Error: The request could not be performed because of an I/O device error

They were retried by VBR and worked again!

Really fishy atm!

Regards,
Didi7

[nielsengelen]

Did you already contacted support in regards to this? Maybe there is something else going on which may become clear by reading the logs. Please post the support case ID as well for future reference.

[Bredero]

This seems eerily like an intermittent issue we have had with exchange backups at a customer. Physical Backup Proxies, RSS disabled, support case ID# 02092977.

Support initially blames UAC and antivirus but that seems unlikely. Resources for the backup server (which is a VM) were increased and issues seem to be less.

Job usually completes succesfully but truncation of transaction logs still fails frequently.

[Nadziratelb]

We tried to remove the antivirus and disable UAC, did not help.
The problem manifests itself only on large tasks, from 50 to 70 VM.
Support advised only to reduce the number of machines in the jobs.

[bavosod]

Please, any news about this issue? i have had the same error even in jobs with just one VM included.

Timed out waiting for asynchronous device I/O completion

Tks...

[Mike Resseler]

Daniel,

Do you have a support case? Please post the case ID here. The more our teams can investigate. Plus, it is rather important to know that if this specific issue gets to support multiple times, it will get escalated to the QC team faster

Thanks
Mike

[foggy]

In fact there's a hotfix for the issue that might potentially be causing this behavior, so contacting support is indeed the most expected step (you may refer to issue 96612, bug 96610).

[Didi7]

Hello,

more severe errors this weekend. Gets worser from week to week. As soon as I find time, I will open a case. Retries didn't work for all VMs this time!

Ok, in this case a problem with VMware Tools in one guest [Guest processing skipped (check VMware Tools status)].

Regards,
Didi7

[Didi7]

@foggy:
Is the problem or bug still VMware Tools related, as was pointed out by Gostev in one of his recent mail blogs?

Regards
Didi7

[foggy]

The mentioned bug is related to repository components experiencing very high load. Not necessarily the one you're seeing, but behavior looks somewhat similar.

[P4Tr1ck]

hey community,
we are getting the same message "Timed out waiting for asynchronous device I/O completion". in backup summary it is stated as a warning status, no error.
we are seeing the warning nearly every backup cycle. just one or two vms each night but concerning different vms, not always the same ones.
this behaviour is new since upgrading veeam b&r 9.0.0.1715 to 9.5.0.823

any ideas on this?

[Shestakov]

Hello P4Tr1ck,
Please contact the support team to check if above-mentioned hot fix covers your case.
Thanks!

[AndyGDIT]

Did this fix come in V9.5 U2? We havent installed it yet, but are seeing the warnings for "Timed out waiting for asynchronous device I/O completion" on some backups.

Andy