Comprehensive data protection for all workloads
BGA-Robert
Service Provider
Posts: 60
Liked: 8 times
Joined: Feb 03, 2016 5:06 pm
Full Name: Robert Wakefield
Contact:

I/O device error / Cannot get file size - 9.5 Related?

Post by BGA-Robert » 1 person likes this post

Hello All,

Veeam support request 01993924

I've rolled out our first 9.5 upgrade. First night, we had two VM's error on the nightly backups.

Error messages look like this:
"Error: The request could not be performed because of an I/O device error. WINAPI: Cannot get file size. File: [Device '\\.\PhysicalDrive8']. Failed to upload disk. Agent failed to process method {DataTransfer.SyncDisk}."

The two VM's succeeded on retry.

The next night, all were successful, but the following night, 3 VM's failed in the nightly backup with similar errors. All three succeeded on retry.

Veeam support picked this out of the logs:
[01.12.2016 19:09:24] <45> Error WINAPI: Cannot get file size. File: [Device '\\.\PhysicalDrive5']. (Veeam.Backup.Common.CCppComponentException)
[01.12.2016 19:09:24] <45> Error in c++: Error code: 0x0000045d

Support is still looking at it.

But I'm wondering if anybody has seen this as a 9.5 upgrade related issue.
It pretty certainly started after the 9.5 upgrade. Prior I've had no errors backing these VM's up.

These are all Windows servers, running under VMware.

My major difficulty at this point is we need proof of success on the 9.5 upgrade cycle before deploying to more critical environments.
So I need a root cause determination and some proof at this point that I'm good to go on upgrade to 9.5 for the rest of our clients.

Thanks!
Gostev
Chief Product Officer
Posts: 31816
Liked: 7302 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: I/O device error / Cannot get file size - 9.5 Related?

Post by Gostev »

Well, standard Windows API function failing to get something from the file system really does not look like a Veeam issue, but a lot like a corrupt file system issue... but let's see what support hears back from the devs.

Answering your last question, after the current number of downloads you can be fairly confident that 9.5 is a solid release and does not have major engine issues. Based on our experience with the previous releases, we know most of the real issues always come out en masse during the very first week and result in dozens of identical support cases created in a matter of a few days.
BGA-Robert
Service Provider
Posts: 60
Liked: 8 times
Joined: Feb 03, 2016 5:06 pm
Full Name: Robert Wakefield
Contact:

Re: I/O device error / Cannot get file size - 9.5 Related?

Post by BGA-Robert » 1 person likes this post

Support found that the error was on the Veeam server, acting as the Veeam proxy.

We found a couple of matching event log entries on the Veeam server indicating the LSI_SAS driver was having a problem

Code: Select all

Log Name:      System
Source:        LSI_SAS
Date:          12/10/2016 7:07:32 PM
Event ID:      11
Description:
The driver detected a controller error on \Device\RaidPort0.

Code: Select all

Log Name:      System
Source:        LSI_SAS
Date:          12/10/2016 7:07:29 PM
Event ID:      129
Description:
Reset to device, \Device\RaidPort0, was issued.
VMware had a match on the error:
https://kb.vmware.com/selfservice/micro ... Id=2063346
But our driver was newer 1.34.3.82 vs. what was referenced in the KB, 1.34.02

Since VMware should be providing that driver, we simply uninstalled VMware tools and re-installed, with a reboot.

The problem was a bit random and intermittent, so we're taking a wait-and-see to see if this fixes the issue. But we've gotten 5 backups since with no errors.
BGA-Robert
Service Provider
Posts: 60
Liked: 8 times
Joined: Feb 03, 2016 5:06 pm
Full Name: Robert Wakefield
Contact:

Re: I/O device error / Cannot get file size - 9.5 Related?

Post by BGA-Robert »

Just a quick update that this issue persists for us. We've updated the VMware environment, thinking it may be a VMware bug. No change.

The errors occur pretty randomly, but somewhat infrequently, but they do continue to occur. And only on this one site (ours - LOL). Since installing 9.5 in December, we've had jobs fail 14 times, across a random assortment of about 1/2 dozen VMs. All always succeed on retry.

We've actually had a rash of Veeam issues and this one is not our highest priority. I hope to come back to it soon and get it resolved.
Gostev
Chief Product Officer
Posts: 31816
Liked: 7302 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: I/O device error / Cannot get file size - 9.5 Related?

Post by Gostev »

Thanks for the update!

Since 9.5 is times more efficient than previous versions at data processing, the extra load on the environment may cause things to break where they are thin (like malfunctioning hardware or drivers). On the other hand, there are also benefits in detecting these issues early instead of having them silently doing bad things to your backups.
DaveWatkins
Veteran
Posts: 370
Liked: 97 times
Joined: Dec 13, 2015 11:33 pm
Contact:

Re: I/O device error / Cannot get file size - 9.5 Related?

Post by DaveWatkins »

Can you not switch over to the paravirtual driver for VMWare? That'd take the LSI driver out of the equation entirely.

It's a slightly messy process of adding it and a new disk onto it then booting to get he driver loaded for it before you can change the boot drives controller, but it's documented in plenty of places on how to do the switch
Nadziratelb
Novice
Posts: 6
Liked: never
Joined: Aug 06, 2014 11:38 pm
Full Name: Alexander
Contact:

Re: I/O device error / Cannot get file size - 9.5 Related?

Post by Nadziratelb »

Hello
And we have this same problem on random virtual machines.
Total virtual machines 500, errors appear regularly after installing version 9.5 :(
03.02.2017 1:28:38 :: Error: The request could not be performed because of an I/O device error.
WINAPI: Cannot get file size. File: [Device '\\.\PhysicalDrive1'].
Failed to upload disk.
Agent failed to process method {DataTransfer.SyncDisk}.
Didi7
Veteran
Posts: 511
Liked: 68 times
Joined: Oct 17, 2014 8:09 am
Location: Hypervisor
Contact:

Re: I/O device error / Cannot get file size - 9.5 Related?

Post by Didi7 »

Hello,

we also have random problems with errors like ...

Code: Select all

*******************************************************************************************************************
Processing XXX Error: The request could not be performed because of an I/O device error. WINAPI: Cannot get file size. File: [Device '\\.\PhysicalDrive5']. Failed to upload disk. Agent failed to process method {DataTransfer.SyncDisk}. 
Processing XXX Error: The request could not be performed because of an I/O device error. WINAPI: Cannot get file size. File: [Device '\\.\PhysicalDrive10']. Failed to upload disk. Agent failed to process method {DataTransfer.SyncDisk}. 
or

Code: Select all

Error: The request could not be performed because of an I/O device error. WINAPI: Cannot get file size. File: [Device '\\.\PhysicalDrive2']. Failed to upload disk. Agent failed to process method {DataTransfer.SyncDisk}. Exception from server: The request could not be performed because of an I/O device error. WINAPI: Cannot get file size. File: [Device '\\.\PhysicalDrive2']. Unable to retrieve next block transmission command. Number of already processed blocks: [9431]. Failed to download disk. 

*******************************************************************************************************************
The corresponding VMs were retried and everything worked fine then.

The days before everything worked fine as well, even the full backup on friday.

More than one week before, we got similar Errors like above and the following ...

Code: Select all

Unable to release guest. Error: Failed to process 'GetVssSnapshotException' command. Failed to obtain error description for VSS snapshot job. There is no job state for the specified job ID: [{9580d99e-0e42-46b1-b649-efd0c70a34d2}].
Error: Failed to process 'GetVssSnapshotException' command. Failed to obtain error description for VSS snapshot job. There is no job state for the specified job ID: [{9580d99e-0e42-46b1-b649-efd0c70a34d2}].

*******************************************************************************************************************
The errors started to appear with VBR 9.5 here as well. Something is fishy here!

Regards,
Didi7
Using the most recent Veeam B&R in many different environments now and counting!
Didi7
Veteran
Posts: 511
Liked: 68 times
Joined: Oct 17, 2014 8:09 am
Location: Hypervisor
Contact:

Re: I/O device error / Cannot get file size - 9.5 Related?

Post by Didi7 »

Today everything worked find agagin! Will post constantly now, as I suppose it has something to do with v9.5 or v9.5 Update 1.

With VBR 9.0 I never had those Errors!

Regards,
Didi7
Using the most recent Veeam B&R in many different environments now and counting!
BGA-Robert
Service Provider
Posts: 60
Liked: 8 times
Joined: Feb 03, 2016 5:06 pm
Full Name: Robert Wakefield
Contact:

Re: I/O device error / Cannot get file size - 9.5 Related?

Post by BGA-Robert »

We continue to see this error intermittently, maybe about once a week. Different jobs / different VMs. I've been busy with other Veeam support requests. When I get caught up, I'll re-engage support on this issue.
Didi7
Veteran
Posts: 511
Liked: 68 times
Joined: Oct 17, 2014 8:09 am
Location: Hypervisor
Contact:

Re: I/O device error / Cannot get file size - 9.5 Related?

Post by Didi7 »

I can confirm the same behavior here ...

Errors intermittently, maybe about once a week, different jobs, different VMs

and supposedly beginning with VBR 9.5 or VBR 9.5 Update 1. Not sure!

Needs to be further investigated here as well.

Regards,
Didi7
Using the most recent Veeam B&R in many different environments now and counting!
Nadziratelb
Novice
Posts: 6
Liked: never
Joined: Aug 06, 2014 11:38 pm
Full Name: Alexander
Contact:

Re: I/O device error / Cannot get file size - 9.5 Related?

Post by Nadziratelb »

At us such error began to appear after installation 9.5.
Update 1 did not correct this error.
Mike Resseler
Product Manager
Posts: 8191
Liked: 1322 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: I/O device error / Cannot get file size - 9.5 Related?

Post by Mike Resseler »

To everyone that has this issue. Do you all have a support case logged? If there are more we can use those. The more logs can be investigated the better to find the final root cause

Thanks
Mike
Nadziratelb
Novice
Posts: 6
Liked: never
Joined: Aug 06, 2014 11:38 pm
Full Name: Alexander
Contact:

Re: I/O device error / Cannot get file size - 9.5 Related?

Post by Nadziratelb »

I applied the logs to the case № 02058418
The case is closed, but the problem remains.
crackocain
Service Provider
Posts: 248
Liked: 28 times
Joined: Dec 14, 2015 8:20 pm
Full Name: Mehmet Istanbullu
Location: Türkiye
Contact:

Re: I/O device error / Cannot get file size - 9.5 Related?

Post by crackocain »

Case #02110634 was created.

Timed out waiting for asynchronous device I/O completion

We are experiencing this issue Wan Accelerator Replication jobs or backup jobs.
VMCA v12
Didi7
Veteran
Posts: 511
Liked: 68 times
Joined: Oct 17, 2014 8:09 am
Location: Hypervisor
Contact:

Re: I/O device error / Cannot get file size - 9.5 Related?

Post by Didi7 »

The word from Gostev:
Just before the weekend, VMware has issued this rush post regarding an issue with VMware Tools versions 9.10.0 up to 10.1.5 that causes network packets to drop due to the fact that all network traffic is serviced by a single guest CPU. It immediately occurred to me that the issue may be especially impactful on virtual backup proxies, because they pump hundreds of MB per second through the network stack – while their CPU is already very busy with all the data processing. So, I thought that this bug could potentially be the reason for those intermittent backup job failures some customers have been having for a while now (the issue where job retry always helps)? I guess we'll find out soon, as soon as VMware Tools are updated.

*************************************************************************************************

Interesting! I just wonder, why this hasn't been a problem with Veeam B&R 9.0 ?

Regards,
Didi7
Using the most recent Veeam B&R in many different environments now and counting!
Didi7
Veteran
Posts: 511
Liked: 68 times
Joined: Oct 17, 2014 8:09 am
Location: Hypervisor
Contact:

Re: I/O device error / Cannot get file size - 9.5 Related?

Post by Didi7 »

Hello again,

just for your information. Here in our environment, VMware Tools 9.4.10 build-2068191 from ESXi 5.5 Update 2 is used since November 2014 and we never had such problems with Veeam B&R 7, 8 and 9 but with 9.5 !!!

Regards,
Didi7
Using the most recent Veeam B&R in many different environments now and counting!
Didi7
Veteran
Posts: 511
Liked: 68 times
Joined: Oct 17, 2014 8:09 am
Location: Hypervisor
Contact:

Re: I/O device error / Cannot get file size - 9.5 Related?

Post by Didi7 »

Severe I/O errors again this weekend. 3 VMs of 12 in one backup job reported ...

Error: The request could not be performed because of an I/O device error

They were retried by VBR and worked again!

Really fishy atm!

Regards,
Didi7
Using the most recent Veeam B&R in many different environments now and counting!
nielsengelen
Product Manager
Posts: 5797
Liked: 1215 times
Joined: Jul 15, 2013 11:09 am
Full Name: Niels Engelen
Contact:

Re: I/O device error / Cannot get file size - 9.5 Related?

Post by nielsengelen »

Did you already contacted support in regards to this? Maybe there is something else going on which may become clear by reading the logs. Please post the support case ID as well for future reference.
Personal blog: https://foonet.be
GitHub: https://github.com/nielsengelen
Bredero
Novice
Posts: 3
Liked: never
Joined: Apr 05, 2015 11:12 am
Contact:

Re: I/O device error / Cannot get file size - 9.5 Related?

Post by Bredero »

This seems eerily like an intermittent issue we have had with exchange backups at a customer. Physical Backup Proxies, RSS disabled, support case ID# 02092977.

Support initially blames UAC and antivirus but that seems unlikely. Resources for the backup server (which is a VM) were increased and issues seem to be less.

Job usually completes succesfully but truncation of transaction logs still fails frequently.
Nadziratelb
Novice
Posts: 6
Liked: never
Joined: Aug 06, 2014 11:38 pm
Full Name: Alexander
Contact:

Re: I/O device error / Cannot get file size - 9.5 Related?

Post by Nadziratelb »

We tried to remove the antivirus and disable UAC, did not help.
The problem manifests itself only on large tasks, from 50 to 70 VM.
Support advised only to reduce the number of machines in the jobs.
bavosod
Novice
Posts: 3
Liked: never
Joined: Mar 21, 2017 11:53 pm
Full Name: Daniel Bavoso
Contact:

Re: I/O device error / Cannot get file size - 9.5 Related?

Post by bavosod »

Please, any news about this issue? i have had the same error even in jobs with just one VM included.

Timed out waiting for asynchronous device I/O completion

Tks...
Mike Resseler
Product Manager
Posts: 8191
Liked: 1322 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: I/O device error / Cannot get file size - 9.5 Related?

Post by Mike Resseler »

Daniel,

Do you have a support case? Please post the case ID here. The more our teams can investigate. Plus, it is rather important to know that if this specific issue gets to support multiple times, it will get escalated to the QC team faster

Thanks
Mike
foggy
Veeam Software
Posts: 21139
Liked: 2141 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: I/O device error / Cannot get file size - 9.5 Related?

Post by foggy »

In fact there's a hotfix for the issue that might potentially be causing this behavior, so contacting support is indeed the most expected step (you may refer to issue 96612, bug 96610).
Didi7
Veteran
Posts: 511
Liked: 68 times
Joined: Oct 17, 2014 8:09 am
Location: Hypervisor
Contact:

Re: I/O device error / Cannot get file size - 9.5 Related?

Post by Didi7 »

Hello,

more severe errors this weekend. Gets worser from week to week. As soon as I find time, I will open a case. Retries didn't work for all VMs this time!

Ok, in this case a problem with VMware Tools in one guest [Guest processing skipped (check VMware Tools status)].

Regards,
Didi7
Using the most recent Veeam B&R in many different environments now and counting!
Didi7
Veteran
Posts: 511
Liked: 68 times
Joined: Oct 17, 2014 8:09 am
Location: Hypervisor
Contact:

Re: I/O device error / Cannot get file size - 9.5 Related?

Post by Didi7 »

@foggy:
Is the problem or bug still VMware Tools related, as was pointed out by Gostev in one of his recent mail blogs?

Regards
Didi7
Using the most recent Veeam B&R in many different environments now and counting!
foggy
Veeam Software
Posts: 21139
Liked: 2141 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: I/O device error / Cannot get file size - 9.5 Related?

Post by foggy »

The mentioned bug is related to repository components experiencing very high load. Not necessarily the one you're seeing, but behavior looks somewhat similar.
P4Tr1ck
Novice
Posts: 5
Liked: 2 times
Joined: Apr 27, 2017 6:09 am
Contact:

Re: I/O device error / Cannot get file size - 9.5 Related?

Post by P4Tr1ck »

hey community,
we are getting the same message "Timed out waiting for asynchronous device I/O completion". in backup summary it is stated as a warning status, no error.
we are seeing the warning nearly every backup cycle. just one or two vms each night but concerning different vms, not always the same ones.
this behaviour is new since upgrading veeam b&r 9.0.0.1715 to 9.5.0.823

any ideas on this?
Shestakov
Veteran
Posts: 7328
Liked: 781 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: I/O device error / Cannot get file size - 9.5 Related?

Post by Shestakov »

Hello P4Tr1ck,
Please contact the support team to check if above-mentioned hot fix covers your case.
Thanks!
AndyGDIT
Enthusiast
Posts: 68
Liked: 9 times
Joined: Nov 14, 2011 3:15 pm
Full Name: Andrew Frye
Contact:

Re: I/O device error / Cannot get file size - 9.5 Related?

Post by AndyGDIT »

Did this fix come in V9.5 U2? We havent installed it yet, but are seeing the warnings for "Timed out waiting for asynchronous device I/O completion" on some backups.

Andy
Post Reply

Who is online

Users browsing this forum: Google [Bot], Semrush [Bot] and 55 guests