-
- Service Provider
- Posts: 60
- Liked: 8 times
- Joined: Feb 03, 2016 5:06 pm
- Full Name: Robert Wakefield
- Contact:
I/O device error / Cannot get file size - 9.5 Related?
Hello All,
Veeam support request 01993924
I've rolled out our first 9.5 upgrade. First night, we had two VM's error on the nightly backups.
Error messages look like this:
"Error: The request could not be performed because of an I/O device error. WINAPI: Cannot get file size. File: [Device '\\.\PhysicalDrive8']. Failed to upload disk. Agent failed to process method {DataTransfer.SyncDisk}."
The two VM's succeeded on retry.
The next night, all were successful, but the following night, 3 VM's failed in the nightly backup with similar errors. All three succeeded on retry.
Veeam support picked this out of the logs:
[01.12.2016 19:09:24] <45> Error WINAPI: Cannot get file size. File: [Device '\\.\PhysicalDrive5']. (Veeam.Backup.Common.CCppComponentException)
[01.12.2016 19:09:24] <45> Error in c++: Error code: 0x0000045d
Support is still looking at it.
But I'm wondering if anybody has seen this as a 9.5 upgrade related issue.
It pretty certainly started after the 9.5 upgrade. Prior I've had no errors backing these VM's up.
These are all Windows servers, running under VMware.
My major difficulty at this point is we need proof of success on the 9.5 upgrade cycle before deploying to more critical environments.
So I need a root cause determination and some proof at this point that I'm good to go on upgrade to 9.5 for the rest of our clients.
Thanks!
Veeam support request 01993924
I've rolled out our first 9.5 upgrade. First night, we had two VM's error on the nightly backups.
Error messages look like this:
"Error: The request could not be performed because of an I/O device error. WINAPI: Cannot get file size. File: [Device '\\.\PhysicalDrive8']. Failed to upload disk. Agent failed to process method {DataTransfer.SyncDisk}."
The two VM's succeeded on retry.
The next night, all were successful, but the following night, 3 VM's failed in the nightly backup with similar errors. All three succeeded on retry.
Veeam support picked this out of the logs:
[01.12.2016 19:09:24] <45> Error WINAPI: Cannot get file size. File: [Device '\\.\PhysicalDrive5']. (Veeam.Backup.Common.CCppComponentException)
[01.12.2016 19:09:24] <45> Error in c++: Error code: 0x0000045d
Support is still looking at it.
But I'm wondering if anybody has seen this as a 9.5 upgrade related issue.
It pretty certainly started after the 9.5 upgrade. Prior I've had no errors backing these VM's up.
These are all Windows servers, running under VMware.
My major difficulty at this point is we need proof of success on the 9.5 upgrade cycle before deploying to more critical environments.
So I need a root cause determination and some proof at this point that I'm good to go on upgrade to 9.5 for the rest of our clients.
Thanks!
-
- Chief Product Officer
- Posts: 31816
- Liked: 7302 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: I/O device error / Cannot get file size - 9.5 Related?
Well, standard Windows API function failing to get something from the file system really does not look like a Veeam issue, but a lot like a corrupt file system issue... but let's see what support hears back from the devs.
Answering your last question, after the current number of downloads you can be fairly confident that 9.5 is a solid release and does not have major engine issues. Based on our experience with the previous releases, we know most of the real issues always come out en masse during the very first week and result in dozens of identical support cases created in a matter of a few days.
Answering your last question, after the current number of downloads you can be fairly confident that 9.5 is a solid release and does not have major engine issues. Based on our experience with the previous releases, we know most of the real issues always come out en masse during the very first week and result in dozens of identical support cases created in a matter of a few days.
-
- Service Provider
- Posts: 60
- Liked: 8 times
- Joined: Feb 03, 2016 5:06 pm
- Full Name: Robert Wakefield
- Contact:
Re: I/O device error / Cannot get file size - 9.5 Related?
Support found that the error was on the Veeam server, acting as the Veeam proxy.
We found a couple of matching event log entries on the Veeam server indicating the LSI_SAS driver was having a problem
VMware had a match on the error:
https://kb.vmware.com/selfservice/micro ... Id=2063346
But our driver was newer 1.34.3.82 vs. what was referenced in the KB, 1.34.02
Since VMware should be providing that driver, we simply uninstalled VMware tools and re-installed, with a reboot.
The problem was a bit random and intermittent, so we're taking a wait-and-see to see if this fixes the issue. But we've gotten 5 backups since with no errors.
We found a couple of matching event log entries on the Veeam server indicating the LSI_SAS driver was having a problem
Code: Select all
Log Name: System
Source: LSI_SAS
Date: 12/10/2016 7:07:32 PM
Event ID: 11
Description:
The driver detected a controller error on \Device\RaidPort0.
Code: Select all
Log Name: System
Source: LSI_SAS
Date: 12/10/2016 7:07:29 PM
Event ID: 129
Description:
Reset to device, \Device\RaidPort0, was issued.
https://kb.vmware.com/selfservice/micro ... Id=2063346
But our driver was newer 1.34.3.82 vs. what was referenced in the KB, 1.34.02
Since VMware should be providing that driver, we simply uninstalled VMware tools and re-installed, with a reboot.
The problem was a bit random and intermittent, so we're taking a wait-and-see to see if this fixes the issue. But we've gotten 5 backups since with no errors.
-
- Service Provider
- Posts: 60
- Liked: 8 times
- Joined: Feb 03, 2016 5:06 pm
- Full Name: Robert Wakefield
- Contact:
Re: I/O device error / Cannot get file size - 9.5 Related?
Just a quick update that this issue persists for us. We've updated the VMware environment, thinking it may be a VMware bug. No change.
The errors occur pretty randomly, but somewhat infrequently, but they do continue to occur. And only on this one site (ours - LOL). Since installing 9.5 in December, we've had jobs fail 14 times, across a random assortment of about 1/2 dozen VMs. All always succeed on retry.
We've actually had a rash of Veeam issues and this one is not our highest priority. I hope to come back to it soon and get it resolved.
The errors occur pretty randomly, but somewhat infrequently, but they do continue to occur. And only on this one site (ours - LOL). Since installing 9.5 in December, we've had jobs fail 14 times, across a random assortment of about 1/2 dozen VMs. All always succeed on retry.
We've actually had a rash of Veeam issues and this one is not our highest priority. I hope to come back to it soon and get it resolved.
-
- Chief Product Officer
- Posts: 31816
- Liked: 7302 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: I/O device error / Cannot get file size - 9.5 Related?
Thanks for the update!
Since 9.5 is times more efficient than previous versions at data processing, the extra load on the environment may cause things to break where they are thin (like malfunctioning hardware or drivers). On the other hand, there are also benefits in detecting these issues early instead of having them silently doing bad things to your backups.
Since 9.5 is times more efficient than previous versions at data processing, the extra load on the environment may cause things to break where they are thin (like malfunctioning hardware or drivers). On the other hand, there are also benefits in detecting these issues early instead of having them silently doing bad things to your backups.
-
- Veteran
- Posts: 370
- Liked: 97 times
- Joined: Dec 13, 2015 11:33 pm
- Contact:
Re: I/O device error / Cannot get file size - 9.5 Related?
Can you not switch over to the paravirtual driver for VMWare? That'd take the LSI driver out of the equation entirely.
It's a slightly messy process of adding it and a new disk onto it then booting to get he driver loaded for it before you can change the boot drives controller, but it's documented in plenty of places on how to do the switch
It's a slightly messy process of adding it and a new disk onto it then booting to get he driver loaded for it before you can change the boot drives controller, but it's documented in plenty of places on how to do the switch
-
- Novice
- Posts: 6
- Liked: never
- Joined: Aug 06, 2014 11:38 pm
- Full Name: Alexander
- Contact:
Re: I/O device error / Cannot get file size - 9.5 Related?
Hello
And we have this same problem on random virtual machines.
Total virtual machines 500, errors appear regularly after installing version 9.5
And we have this same problem on random virtual machines.
Total virtual machines 500, errors appear regularly after installing version 9.5
03.02.2017 1:28:38 :: Error: The request could not be performed because of an I/O device error.
WINAPI: Cannot get file size. File: [Device '\\.\PhysicalDrive1'].
Failed to upload disk.
Agent failed to process method {DataTransfer.SyncDisk}.
-
- Veteran
- Posts: 511
- Liked: 68 times
- Joined: Oct 17, 2014 8:09 am
- Location: Hypervisor
- Contact:
Re: I/O device error / Cannot get file size - 9.5 Related?
Hello,
we also have random problems with errors like ...
or
The corresponding VMs were retried and everything worked fine then.
The days before everything worked fine as well, even the full backup on friday.
More than one week before, we got similar Errors like above and the following ...
The errors started to appear with VBR 9.5 here as well. Something is fishy here!
Regards,
Didi7
we also have random problems with errors like ...
Code: Select all
*******************************************************************************************************************
Processing XXX Error: The request could not be performed because of an I/O device error. WINAPI: Cannot get file size. File: [Device '\\.\PhysicalDrive5']. Failed to upload disk. Agent failed to process method {DataTransfer.SyncDisk}.
Processing XXX Error: The request could not be performed because of an I/O device error. WINAPI: Cannot get file size. File: [Device '\\.\PhysicalDrive10']. Failed to upload disk. Agent failed to process method {DataTransfer.SyncDisk}.
Code: Select all
Error: The request could not be performed because of an I/O device error. WINAPI: Cannot get file size. File: [Device '\\.\PhysicalDrive2']. Failed to upload disk. Agent failed to process method {DataTransfer.SyncDisk}. Exception from server: The request could not be performed because of an I/O device error. WINAPI: Cannot get file size. File: [Device '\\.\PhysicalDrive2']. Unable to retrieve next block transmission command. Number of already processed blocks: [9431]. Failed to download disk.
*******************************************************************************************************************
The days before everything worked fine as well, even the full backup on friday.
More than one week before, we got similar Errors like above and the following ...
Code: Select all
Unable to release guest. Error: Failed to process 'GetVssSnapshotException' command. Failed to obtain error description for VSS snapshot job. There is no job state for the specified job ID: [{9580d99e-0e42-46b1-b649-efd0c70a34d2}].
Error: Failed to process 'GetVssSnapshotException' command. Failed to obtain error description for VSS snapshot job. There is no job state for the specified job ID: [{9580d99e-0e42-46b1-b649-efd0c70a34d2}].
*******************************************************************************************************************
Regards,
Didi7
Using the most recent Veeam B&R in many different environments now and counting!
-
- Veteran
- Posts: 511
- Liked: 68 times
- Joined: Oct 17, 2014 8:09 am
- Location: Hypervisor
- Contact:
Re: I/O device error / Cannot get file size - 9.5 Related?
Today everything worked find agagin! Will post constantly now, as I suppose it has something to do with v9.5 or v9.5 Update 1.
With VBR 9.0 I never had those Errors!
Regards,
Didi7
With VBR 9.0 I never had those Errors!
Regards,
Didi7
Using the most recent Veeam B&R in many different environments now and counting!
-
- Service Provider
- Posts: 60
- Liked: 8 times
- Joined: Feb 03, 2016 5:06 pm
- Full Name: Robert Wakefield
- Contact:
Re: I/O device error / Cannot get file size - 9.5 Related?
We continue to see this error intermittently, maybe about once a week. Different jobs / different VMs. I've been busy with other Veeam support requests. When I get caught up, I'll re-engage support on this issue.
-
- Veteran
- Posts: 511
- Liked: 68 times
- Joined: Oct 17, 2014 8:09 am
- Location: Hypervisor
- Contact:
Re: I/O device error / Cannot get file size - 9.5 Related?
I can confirm the same behavior here ...
Errors intermittently, maybe about once a week, different jobs, different VMs
and supposedly beginning with VBR 9.5 or VBR 9.5 Update 1. Not sure!
Needs to be further investigated here as well.
Regards,
Didi7
Errors intermittently, maybe about once a week, different jobs, different VMs
and supposedly beginning with VBR 9.5 or VBR 9.5 Update 1. Not sure!
Needs to be further investigated here as well.
Regards,
Didi7
Using the most recent Veeam B&R in many different environments now and counting!
-
- Novice
- Posts: 6
- Liked: never
- Joined: Aug 06, 2014 11:38 pm
- Full Name: Alexander
- Contact:
Re: I/O device error / Cannot get file size - 9.5 Related?
At us such error began to appear after installation 9.5.
Update 1 did not correct this error.
Update 1 did not correct this error.
-
- Product Manager
- Posts: 8191
- Liked: 1322 times
- Joined: Feb 08, 2013 3:08 pm
- Full Name: Mike Resseler
- Location: Belgium
- Contact:
Re: I/O device error / Cannot get file size - 9.5 Related?
To everyone that has this issue. Do you all have a support case logged? If there are more we can use those. The more logs can be investigated the better to find the final root cause
Thanks
Mike
Thanks
Mike
-
- Novice
- Posts: 6
- Liked: never
- Joined: Aug 06, 2014 11:38 pm
- Full Name: Alexander
- Contact:
Re: I/O device error / Cannot get file size - 9.5 Related?
I applied the logs to the case № 02058418
The case is closed, but the problem remains.
The case is closed, but the problem remains.
-
- Service Provider
- Posts: 248
- Liked: 28 times
- Joined: Dec 14, 2015 8:20 pm
- Full Name: Mehmet Istanbullu
- Location: Türkiye
- Contact:
Re: I/O device error / Cannot get file size - 9.5 Related?
Case #02110634 was created.
Timed out waiting for asynchronous device I/O completion
We are experiencing this issue Wan Accelerator Replication jobs or backup jobs.
Timed out waiting for asynchronous device I/O completion
We are experiencing this issue Wan Accelerator Replication jobs or backup jobs.
VMCA v12
-
- Veteran
- Posts: 511
- Liked: 68 times
- Joined: Oct 17, 2014 8:09 am
- Location: Hypervisor
- Contact:
Re: I/O device error / Cannot get file size - 9.5 Related?
The word from Gostev:
Just before the weekend, VMware has issued this rush post regarding an issue with VMware Tools versions 9.10.0 up to 10.1.5 that causes network packets to drop due to the fact that all network traffic is serviced by a single guest CPU. It immediately occurred to me that the issue may be especially impactful on virtual backup proxies, because they pump hundreds of MB per second through the network stack – while their CPU is already very busy with all the data processing. So, I thought that this bug could potentially be the reason for those intermittent backup job failures some customers have been having for a while now (the issue where job retry always helps)? I guess we'll find out soon, as soon as VMware Tools are updated.
*************************************************************************************************
Interesting! I just wonder, why this hasn't been a problem with Veeam B&R 9.0 ?
Regards,
Didi7
Just before the weekend, VMware has issued this rush post regarding an issue with VMware Tools versions 9.10.0 up to 10.1.5 that causes network packets to drop due to the fact that all network traffic is serviced by a single guest CPU. It immediately occurred to me that the issue may be especially impactful on virtual backup proxies, because they pump hundreds of MB per second through the network stack – while their CPU is already very busy with all the data processing. So, I thought that this bug could potentially be the reason for those intermittent backup job failures some customers have been having for a while now (the issue where job retry always helps)? I guess we'll find out soon, as soon as VMware Tools are updated.
*************************************************************************************************
Interesting! I just wonder, why this hasn't been a problem with Veeam B&R 9.0 ?
Regards,
Didi7
Using the most recent Veeam B&R in many different environments now and counting!
-
- Veteran
- Posts: 511
- Liked: 68 times
- Joined: Oct 17, 2014 8:09 am
- Location: Hypervisor
- Contact:
Re: I/O device error / Cannot get file size - 9.5 Related?
Hello again,
just for your information. Here in our environment, VMware Tools 9.4.10 build-2068191 from ESXi 5.5 Update 2 is used since November 2014 and we never had such problems with Veeam B&R 7, 8 and 9 but with 9.5 !!!
Regards,
Didi7
just for your information. Here in our environment, VMware Tools 9.4.10 build-2068191 from ESXi 5.5 Update 2 is used since November 2014 and we never had such problems with Veeam B&R 7, 8 and 9 but with 9.5 !!!
Regards,
Didi7
Using the most recent Veeam B&R in many different environments now and counting!
-
- Veteran
- Posts: 511
- Liked: 68 times
- Joined: Oct 17, 2014 8:09 am
- Location: Hypervisor
- Contact:
Re: I/O device error / Cannot get file size - 9.5 Related?
Severe I/O errors again this weekend. 3 VMs of 12 in one backup job reported ...
Error: The request could not be performed because of an I/O device error
They were retried by VBR and worked again!
Really fishy atm!
Regards,
Didi7
Error: The request could not be performed because of an I/O device error
They were retried by VBR and worked again!
Really fishy atm!
Regards,
Didi7
Using the most recent Veeam B&R in many different environments now and counting!
-
- Product Manager
- Posts: 5797
- Liked: 1215 times
- Joined: Jul 15, 2013 11:09 am
- Full Name: Niels Engelen
- Contact:
Re: I/O device error / Cannot get file size - 9.5 Related?
Did you already contacted support in regards to this? Maybe there is something else going on which may become clear by reading the logs. Please post the support case ID as well for future reference.
Personal blog: https://foonet.be
GitHub: https://github.com/nielsengelen
GitHub: https://github.com/nielsengelen
-
- Novice
- Posts: 3
- Liked: never
- Joined: Apr 05, 2015 11:12 am
- Contact:
Re: I/O device error / Cannot get file size - 9.5 Related?
This seems eerily like an intermittent issue we have had with exchange backups at a customer. Physical Backup Proxies, RSS disabled, support case ID# 02092977.
Support initially blames UAC and antivirus but that seems unlikely. Resources for the backup server (which is a VM) were increased and issues seem to be less.
Job usually completes succesfully but truncation of transaction logs still fails frequently.
Support initially blames UAC and antivirus but that seems unlikely. Resources for the backup server (which is a VM) were increased and issues seem to be less.
Job usually completes succesfully but truncation of transaction logs still fails frequently.
-
- Novice
- Posts: 6
- Liked: never
- Joined: Aug 06, 2014 11:38 pm
- Full Name: Alexander
- Contact:
Re: I/O device error / Cannot get file size - 9.5 Related?
We tried to remove the antivirus and disable UAC, did not help.
The problem manifests itself only on large tasks, from 50 to 70 VM.
Support advised only to reduce the number of machines in the jobs.
The problem manifests itself only on large tasks, from 50 to 70 VM.
Support advised only to reduce the number of machines in the jobs.
-
- Novice
- Posts: 3
- Liked: never
- Joined: Mar 21, 2017 11:53 pm
- Full Name: Daniel Bavoso
- Contact:
Re: I/O device error / Cannot get file size - 9.5 Related?
Please, any news about this issue? i have had the same error even in jobs with just one VM included.
Timed out waiting for asynchronous device I/O completion
Tks...
Timed out waiting for asynchronous device I/O completion
Tks...
-
- Product Manager
- Posts: 8191
- Liked: 1322 times
- Joined: Feb 08, 2013 3:08 pm
- Full Name: Mike Resseler
- Location: Belgium
- Contact:
Re: I/O device error / Cannot get file size - 9.5 Related?
Daniel,
Do you have a support case? Please post the case ID here. The more our teams can investigate. Plus, it is rather important to know that if this specific issue gets to support multiple times, it will get escalated to the QC team faster
Thanks
Mike
Do you have a support case? Please post the case ID here. The more our teams can investigate. Plus, it is rather important to know that if this specific issue gets to support multiple times, it will get escalated to the QC team faster
Thanks
Mike
-
- Veeam Software
- Posts: 21139
- Liked: 2141 times
- Joined: Jul 11, 2011 10:22 am
- Full Name: Alexander Fogelson
- Contact:
Re: I/O device error / Cannot get file size - 9.5 Related?
In fact there's a hotfix for the issue that might potentially be causing this behavior, so contacting support is indeed the most expected step (you may refer to issue 96612, bug 96610).
-
- Veteran
- Posts: 511
- Liked: 68 times
- Joined: Oct 17, 2014 8:09 am
- Location: Hypervisor
- Contact:
Re: I/O device error / Cannot get file size - 9.5 Related?
Hello,
more severe errors this weekend. Gets worser from week to week. As soon as I find time, I will open a case. Retries didn't work for all VMs this time!
Ok, in this case a problem with VMware Tools in one guest [Guest processing skipped (check VMware Tools status)].
Regards,
Didi7
more severe errors this weekend. Gets worser from week to week. As soon as I find time, I will open a case. Retries didn't work for all VMs this time!
Ok, in this case a problem with VMware Tools in one guest [Guest processing skipped (check VMware Tools status)].
Regards,
Didi7
Using the most recent Veeam B&R in many different environments now and counting!
-
- Veteran
- Posts: 511
- Liked: 68 times
- Joined: Oct 17, 2014 8:09 am
- Location: Hypervisor
- Contact:
Re: I/O device error / Cannot get file size - 9.5 Related?
@foggy:
Is the problem or bug still VMware Tools related, as was pointed out by Gostev in one of his recent mail blogs?
Regards
Didi7
Is the problem or bug still VMware Tools related, as was pointed out by Gostev in one of his recent mail blogs?
Regards
Didi7
Using the most recent Veeam B&R in many different environments now and counting!
-
- Veeam Software
- Posts: 21139
- Liked: 2141 times
- Joined: Jul 11, 2011 10:22 am
- Full Name: Alexander Fogelson
- Contact:
Re: I/O device error / Cannot get file size - 9.5 Related?
The mentioned bug is related to repository components experiencing very high load. Not necessarily the one you're seeing, but behavior looks somewhat similar.
-
- Novice
- Posts: 5
- Liked: 2 times
- Joined: Apr 27, 2017 6:09 am
- Contact:
Re: I/O device error / Cannot get file size - 9.5 Related?
hey community,
we are getting the same message "Timed out waiting for asynchronous device I/O completion". in backup summary it is stated as a warning status, no error.
we are seeing the warning nearly every backup cycle. just one or two vms each night but concerning different vms, not always the same ones.
this behaviour is new since upgrading veeam b&r 9.0.0.1715 to 9.5.0.823
any ideas on this?
we are getting the same message "Timed out waiting for asynchronous device I/O completion". in backup summary it is stated as a warning status, no error.
we are seeing the warning nearly every backup cycle. just one or two vms each night but concerning different vms, not always the same ones.
this behaviour is new since upgrading veeam b&r 9.0.0.1715 to 9.5.0.823
any ideas on this?
-
- Veteran
- Posts: 7328
- Liked: 781 times
- Joined: May 21, 2014 11:03 am
- Full Name: Nikita Shestakov
- Location: Prague
- Contact:
Re: I/O device error / Cannot get file size - 9.5 Related?
Hello P4Tr1ck,
Please contact the support team to check if above-mentioned hot fix covers your case.
Thanks!
Please contact the support team to check if above-mentioned hot fix covers your case.
Thanks!
-
- Enthusiast
- Posts: 68
- Liked: 9 times
- Joined: Nov 14, 2011 3:15 pm
- Full Name: Andrew Frye
- Contact:
Re: I/O device error / Cannot get file size - 9.5 Related?
Did this fix come in V9.5 U2? We havent installed it yet, but are seeing the warnings for "Timed out waiting for asynchronous device I/O completion" on some backups.
Andy
Andy
Who is online
Users browsing this forum: Google [Bot], Semrush [Bot] and 55 guests