I/O device error / Cannot get file size - 9.5 Related?

Availability for the Always-On Enterprise

I/O device error / Cannot get file size - 9.5 Related?

Veeam Logoby BGA-Robert » Wed Dec 07, 2016 12:45 am 1 person likes this post

Hello All,

Veeam support request 01993924

I've rolled out our first 9.5 upgrade. First night, we had two VM's error on the nightly backups.

Error messages look like this:
"Error: The request could not be performed because of an I/O device error. WINAPI: Cannot get file size. File: [Device '\\.\PhysicalDrive8']. Failed to upload disk. Agent failed to process method {DataTransfer.SyncDisk}."

The two VM's succeeded on retry.

The next night, all were successful, but the following night, 3 VM's failed in the nightly backup with similar errors. All three succeeded on retry.

Veeam support picked this out of the logs:
[01.12.2016 19:09:24] <45> Error WINAPI: Cannot get file size. File: [Device '\\.\PhysicalDrive5']. (Veeam.Backup.Common.CCppComponentException)
[01.12.2016 19:09:24] <45> Error in c++: Error code: 0x0000045d

Support is still looking at it.

But I'm wondering if anybody has seen this as a 9.5 upgrade related issue.
It pretty certainly started after the 9.5 upgrade. Prior I've had no errors backing these VM's up.

These are all Windows servers, running under VMware.

My major difficulty at this point is we need proof of success on the 9.5 upgrade cycle before deploying to more critical environments.
So I need a root cause determination and some proof at this point that I'm good to go on upgrade to 9.5 for the rest of our clients.

Thanks!
BGA-Robert
Service Provider
 
Posts: 22
Liked: 3 times
Joined: Wed Feb 03, 2016 5:06 pm
Full Name: Robert Wakefield

Re: I/O device error / Cannot get file size - 9.5 Related?

Veeam Logoby Gostev » Wed Dec 07, 2016 1:05 am

Well, standard Windows API function failing to get something from the file system really does not look like a Veeam issue, but a lot like a corrupt file system issue... but let's see what support hears back from the devs.

Answering your last question, after the current number of downloads you can be fairly confident that 9.5 is a solid release and does not have major engine issues. Based on our experience with the previous releases, we know most of the real issues always come out en masse during the very first week and result in dozens of identical support cases created in a matter of a few days.
Gostev
Veeam Software
 
Posts: 21254
Liked: 2318 times
Joined: Sun Jan 01, 2006 1:01 am
Full Name: Anton Gostev

Re: I/O device error / Cannot get file size - 9.5 Related?

Veeam Logoby BGA-Robert » Fri Dec 16, 2016 6:35 pm 1 person likes this post

Support found that the error was on the Veeam server, acting as the Veeam proxy.

We found a couple of matching event log entries on the Veeam server indicating the LSI_SAS driver was having a problem

Code: Select all
Log Name:      System
Source:        LSI_SAS
Date:          12/10/2016 7:07:32 PM
Event ID:      11
Description:
The driver detected a controller error on \Device\RaidPort0.


Code: Select all
Log Name:      System
Source:        LSI_SAS
Date:          12/10/2016 7:07:29 PM
Event ID:      129
Description:
Reset to device, \Device\RaidPort0, was issued.


VMware had a match on the error:
https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2063346
But our driver was newer 1.34.3.82 vs. what was referenced in the KB, 1.34.02

Since VMware should be providing that driver, we simply uninstalled VMware tools and re-installed, with a reboot.

The problem was a bit random and intermittent, so we're taking a wait-and-see to see if this fixes the issue. But we've gotten 5 backups since with no errors.
BGA-Robert
Service Provider
 
Posts: 22
Liked: 3 times
Joined: Wed Feb 03, 2016 5:06 pm
Full Name: Robert Wakefield

Re: I/O device error / Cannot get file size - 9.5 Related?

Veeam Logoby BGA-Robert » Tue Jan 31, 2017 6:42 pm

Just a quick update that this issue persists for us. We've updated the VMware environment, thinking it may be a VMware bug. No change.

The errors occur pretty randomly, but somewhat infrequently, but they do continue to occur. And only on this one site (ours - LOL). Since installing 9.5 in December, we've had jobs fail 14 times, across a random assortment of about 1/2 dozen VMs. All always succeed on retry.

We've actually had a rash of Veeam issues and this one is not our highest priority. I hope to come back to it soon and get it resolved.
BGA-Robert
Service Provider
 
Posts: 22
Liked: 3 times
Joined: Wed Feb 03, 2016 5:06 pm
Full Name: Robert Wakefield

Re: I/O device error / Cannot get file size - 9.5 Related?

Veeam Logoby Gostev » Tue Jan 31, 2017 7:22 pm

Thanks for the update!

Since 9.5 is times more efficient than previous versions at data processing, the extra load on the environment may cause things to break where they are thin (like malfunctioning hardware or drivers). On the other hand, there are also benefits in detecting these issues early instead of having them silently doing bad things to your backups.
Gostev
Veeam Software
 
Posts: 21254
Liked: 2318 times
Joined: Sun Jan 01, 2006 1:01 am
Full Name: Anton Gostev

Re: I/O device error / Cannot get file size - 9.5 Related?

Veeam Logoby DaveWatkins » Tue Jan 31, 2017 9:26 pm

Can you not switch over to the paravirtual driver for VMWare? That'd take the LSI driver out of the equation entirely.

It's a slightly messy process of adding it and a new disk onto it then booting to get he driver loaded for it before you can change the boot drives controller, but it's documented in plenty of places on how to do the switch
DaveWatkins
Expert
 
Posts: 241
Liked: 60 times
Joined: Sun Dec 13, 2015 11:33 pm

Re: I/O device error / Cannot get file size - 9.5 Related?

Veeam Logoby Nadziratelb » Fri Feb 03, 2017 1:37 am

Hello
And we have this same problem on random virtual machines.
Total virtual machines 500, errors appear regularly after installing version 9.5 :(
03.02.2017 1:28:38 :: Error: The request could not be performed because of an I/O device error.
WINAPI: Cannot get file size. File: [Device '\\.\PhysicalDrive1'].
Failed to upload disk.
Agent failed to process method {DataTransfer.SyncDisk}.
Nadziratelb
Novice
 
Posts: 5
Liked: never
Joined: Wed Aug 06, 2014 11:38 pm
Full Name: Alexander

Re: I/O device error / Cannot get file size - 9.5 Related?

Veeam Logoby Didi7 » Wed Mar 15, 2017 9:54 am

Hello,

we also have random problems with errors like ...
Code: Select all
*******************************************************************************************************************
Processing XXX Error: The request could not be performed because of an I/O device error. WINAPI: Cannot get file size. File: [Device '\\.\PhysicalDrive5']. Failed to upload disk. Agent failed to process method {DataTransfer.SyncDisk}.
Processing XXX Error: The request could not be performed because of an I/O device error. WINAPI: Cannot get file size. File: [Device '\\.\PhysicalDrive10']. Failed to upload disk. Agent failed to process method {DataTransfer.SyncDisk}.

or
Code: Select all
Error: The request could not be performed because of an I/O device error. WINAPI: Cannot get file size. File: [Device '\\.\PhysicalDrive2']. Failed to upload disk. Agent failed to process method {DataTransfer.SyncDisk}. Exception from server: The request could not be performed because of an I/O device error. WINAPI: Cannot get file size. File: [Device '\\.\PhysicalDrive2']. Unable to retrieve next block transmission command. Number of already processed blocks: [9431]. Failed to download disk.

*******************************************************************************************************************

The corresponding VMs were retried and everything worked fine then.

The days before everything worked fine as well, even the full backup on friday.

More than one week before, we got similar Errors like above and the following ...
Code: Select all
Unable to release guest. Error: Failed to process 'GetVssSnapshotException' command. Failed to obtain error description for VSS snapshot job. There is no job state for the specified job ID: [{9580d99e-0e42-46b1-b649-efd0c70a34d2}].
Error: Failed to process 'GetVssSnapshotException' command. Failed to obtain error description for VSS snapshot job. There is no job state for the specified job ID: [{9580d99e-0e42-46b1-b649-efd0c70a34d2}].

*******************************************************************************************************************

The errors started to appear with VBR 9.5 here as well. Something is fishy here!

Regards,
Didi7
Didi7
Expert
 
Posts: 206
Liked: 15 times
Joined: Fri Oct 17, 2014 8:09 am

Re: I/O device error / Cannot get file size - 9.5 Related?

Veeam Logoby Didi7 » Thu Mar 16, 2017 11:01 am

Today everything worked find agagin! Will post constantly now, as I suppose it has something to do with v9.5 or v9.5 Update 1.

With VBR 9.0 I never had those Errors!

Regards,
Didi7
Didi7
Expert
 
Posts: 206
Liked: 15 times
Joined: Fri Oct 17, 2014 8:09 am

Re: I/O device error / Cannot get file size - 9.5 Related?

Veeam Logoby BGA-Robert » Thu Mar 16, 2017 3:08 pm

We continue to see this error intermittently, maybe about once a week. Different jobs / different VMs. I've been busy with other Veeam support requests. When I get caught up, I'll re-engage support on this issue.
BGA-Robert
Service Provider
 
Posts: 22
Liked: 3 times
Joined: Wed Feb 03, 2016 5:06 pm
Full Name: Robert Wakefield

Re: I/O device error / Cannot get file size - 9.5 Related?

Veeam Logoby Didi7 » Thu Mar 16, 2017 3:50 pm

I can confirm the same behavior here ...

Errors intermittently, maybe about once a week, different jobs, different VMs

and supposedly beginning with VBR 9.5 or VBR 9.5 Update 1. Not sure!

Needs to be further investigated here as well.

Regards,
Didi7
Didi7
Expert
 
Posts: 206
Liked: 15 times
Joined: Fri Oct 17, 2014 8:09 am

Re: I/O device error / Cannot get file size - 9.5 Related?

Veeam Logoby Nadziratelb » Fri Mar 24, 2017 2:29 am

At us such error began to appear after installation 9.5.
Update 1 did not correct this error.
Nadziratelb
Novice
 
Posts: 5
Liked: never
Joined: Wed Aug 06, 2014 11:38 pm
Full Name: Alexander

Re: I/O device error / Cannot get file size - 9.5 Related?

Veeam Logoby Mike Resseler » Fri Mar 24, 2017 6:23 am

To everyone that has this issue. Do you all have a support case logged? If there are more we can use those. The more logs can be investigated the better to find the final root cause

Thanks
Mike
Mike Resseler
Veeam Software
 
Posts: 2997
Liked: 354 times
Joined: Fri Feb 08, 2013 3:08 pm
Location: Belgium, the land of the fries, the beer, the chocolate and the diamonds...
Full Name: Mike Resseler

Re: I/O device error / Cannot get file size - 9.5 Related?

Veeam Logoby Nadziratelb » Mon Mar 27, 2017 5:34 am

I applied the logs to the case № 02058418
The case is closed, but the problem remains.
Nadziratelb
Novice
 
Posts: 5
Liked: never
Joined: Wed Aug 06, 2014 11:38 pm
Full Name: Alexander

Re: I/O device error / Cannot get file size - 9.5 Related?

Veeam Logoby crackocain » Mon Mar 27, 2017 8:15 am

Case #02110634 was created.

Timed out waiting for asynchronous device I/O completion

We are experiencing this issue Wan Accelerator Replication jobs or backup jobs.
EMC, IBM Storage Specialist - VMCEv9 - Dosbil - Turkey
crackocain
Veeam ProPartner
 
Posts: 48
Liked: 7 times
Joined: Mon Dec 14, 2015 8:20 pm
Location: Turkey
Full Name: Mehmet Istanbullu

Next

Return to Veeam Backup & Replication



Who is online

Users browsing this forum: Bing [Bot], Yahoo [Bot] and 22 guests