Availability for the Always-On Enterprise
BGA-Robert
Service Provider
Posts: 48
Liked: 7 times
Joined: Feb 03, 2016 5:06 pm
Full Name: Robert Wakefield
Contact:

I/O device error / Cannot get file size - 9.5 Related?

Post by BGA-Robert » Dec 07, 2016 12:45 am 1 person likes this post

Hello All,

Veeam support request 01993924

I've rolled out our first 9.5 upgrade. First night, we had two VM's error on the nightly backups.

Error messages look like this:
"Error: The request could not be performed because of an I/O device error. WINAPI: Cannot get file size. File: [Device '\\.\PhysicalDrive8']. Failed to upload disk. Agent failed to process method {DataTransfer.SyncDisk}."

The two VM's succeeded on retry.

The next night, all were successful, but the following night, 3 VM's failed in the nightly backup with similar errors. All three succeeded on retry.

Veeam support picked this out of the logs:
[01.12.2016 19:09:24] <45> Error WINAPI: Cannot get file size. File: [Device '\\.\PhysicalDrive5']. (Veeam.Backup.Common.CCppComponentException)
[01.12.2016 19:09:24] <45> Error in c++: Error code: 0x0000045d

Support is still looking at it.

But I'm wondering if anybody has seen this as a 9.5 upgrade related issue.
It pretty certainly started after the 9.5 upgrade. Prior I've had no errors backing these VM's up.

These are all Windows servers, running under VMware.

My major difficulty at this point is we need proof of success on the 9.5 upgrade cycle before deploying to more critical environments.
So I need a root cause determination and some proof at this point that I'm good to go on upgrade to 9.5 for the rest of our clients.

Thanks!

Gostev
Veeam Software
Posts: 22813
Liked: 2807 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: I/O device error / Cannot get file size - 9.5 Related?

Post by Gostev » Dec 07, 2016 1:05 am

Well, standard Windows API function failing to get something from the file system really does not look like a Veeam issue, but a lot like a corrupt file system issue... but let's see what support hears back from the devs.

Answering your last question, after the current number of downloads you can be fairly confident that 9.5 is a solid release and does not have major engine issues. Based on our experience with the previous releases, we know most of the real issues always come out en masse during the very first week and result in dozens of identical support cases created in a matter of a few days.

BGA-Robert
Service Provider
Posts: 48
Liked: 7 times
Joined: Feb 03, 2016 5:06 pm
Full Name: Robert Wakefield
Contact:

Re: I/O device error / Cannot get file size - 9.5 Related?

Post by BGA-Robert » Dec 16, 2016 6:35 pm 1 person likes this post

Support found that the error was on the Veeam server, acting as the Veeam proxy.

We found a couple of matching event log entries on the Veeam server indicating the LSI_SAS driver was having a problem

Code: Select all

Log Name:      System
Source:        LSI_SAS
Date:          12/10/2016 7:07:32 PM
Event ID:      11
Description:
The driver detected a controller error on \Device\RaidPort0.

Code: Select all

Log Name:      System
Source:        LSI_SAS
Date:          12/10/2016 7:07:29 PM
Event ID:      129
Description:
Reset to device, \Device\RaidPort0, was issued.
VMware had a match on the error:
https://kb.vmware.com/selfservice/micro ... Id=2063346
But our driver was newer 1.34.3.82 vs. what was referenced in the KB, 1.34.02

Since VMware should be providing that driver, we simply uninstalled VMware tools and re-installed, with a reboot.

The problem was a bit random and intermittent, so we're taking a wait-and-see to see if this fixes the issue. But we've gotten 5 backups since with no errors.

BGA-Robert
Service Provider
Posts: 48
Liked: 7 times
Joined: Feb 03, 2016 5:06 pm
Full Name: Robert Wakefield
Contact:

Re: I/O device error / Cannot get file size - 9.5 Related?

Post by BGA-Robert » Jan 31, 2017 6:42 pm

Just a quick update that this issue persists for us. We've updated the VMware environment, thinking it may be a VMware bug. No change.

The errors occur pretty randomly, but somewhat infrequently, but they do continue to occur. And only on this one site (ours - LOL). Since installing 9.5 in December, we've had jobs fail 14 times, across a random assortment of about 1/2 dozen VMs. All always succeed on retry.

We've actually had a rash of Veeam issues and this one is not our highest priority. I hope to come back to it soon and get it resolved.

Gostev
Veeam Software
Posts: 22813
Liked: 2807 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: I/O device error / Cannot get file size - 9.5 Related?

Post by Gostev » Jan 31, 2017 7:22 pm

Thanks for the update!

Since 9.5 is times more efficient than previous versions at data processing, the extra load on the environment may cause things to break where they are thin (like malfunctioning hardware or drivers). On the other hand, there are also benefits in detecting these issues early instead of having them silently doing bad things to your backups.

DaveWatkins
Expert
Posts: 316
Liked: 82 times
Joined: Dec 13, 2015 11:33 pm
Contact:

Re: I/O device error / Cannot get file size - 9.5 Related?

Post by DaveWatkins » Jan 31, 2017 9:26 pm

Can you not switch over to the paravirtual driver for VMWare? That'd take the LSI driver out of the equation entirely.

It's a slightly messy process of adding it and a new disk onto it then booting to get he driver loaded for it before you can change the boot drives controller, but it's documented in plenty of places on how to do the switch

Nadziratelb
Novice
Posts: 5
Liked: never
Joined: Aug 06, 2014 11:38 pm
Full Name: Alexander
Contact:

Re: I/O device error / Cannot get file size - 9.5 Related?

Post by Nadziratelb » Feb 03, 2017 1:37 am

Hello
And we have this same problem on random virtual machines.
Total virtual machines 500, errors appear regularly after installing version 9.5 :(
03.02.2017 1:28:38 :: Error: The request could not be performed because of an I/O device error.
WINAPI: Cannot get file size. File: [Device '\\.\PhysicalDrive1'].
Failed to upload disk.
Agent failed to process method {DataTransfer.SyncDisk}.

Didi7
Expert
Posts: 275
Liked: 19 times
Joined: Oct 17, 2014 8:09 am
Contact:

Re: I/O device error / Cannot get file size - 9.5 Related?

Post by Didi7 » Mar 15, 2017 9:54 am

Hello,

we also have random problems with errors like ...

Code: Select all

*******************************************************************************************************************
Processing XXX Error: The request could not be performed because of an I/O device error. WINAPI: Cannot get file size. File: [Device '\\.\PhysicalDrive5']. Failed to upload disk. Agent failed to process method {DataTransfer.SyncDisk}. 
Processing XXX Error: The request could not be performed because of an I/O device error. WINAPI: Cannot get file size. File: [Device '\\.\PhysicalDrive10']. Failed to upload disk. Agent failed to process method {DataTransfer.SyncDisk}. 
or

Code: Select all

Error: The request could not be performed because of an I/O device error. WINAPI: Cannot get file size. File: [Device '\\.\PhysicalDrive2']. Failed to upload disk. Agent failed to process method {DataTransfer.SyncDisk}. Exception from server: The request could not be performed because of an I/O device error. WINAPI: Cannot get file size. File: [Device '\\.\PhysicalDrive2']. Unable to retrieve next block transmission command. Number of already processed blocks: [9431]. Failed to download disk. 

*******************************************************************************************************************
The corresponding VMs were retried and everything worked fine then.

The days before everything worked fine as well, even the full backup on friday.

More than one week before, we got similar Errors like above and the following ...

Code: Select all

Unable to release guest. Error: Failed to process 'GetVssSnapshotException' command. Failed to obtain error description for VSS snapshot job. There is no job state for the specified job ID: [{9580d99e-0e42-46b1-b649-efd0c70a34d2}].
Error: Failed to process 'GetVssSnapshotException' command. Failed to obtain error description for VSS snapshot job. There is no job state for the specified job ID: [{9580d99e-0e42-46b1-b649-efd0c70a34d2}].

*******************************************************************************************************************
The errors started to appear with VBR 9.5 here as well. Something is fishy here!

Regards,
Didi7
Using the most recent Veeam B&R in 10 different environments now and counting!

Didi7
Expert
Posts: 275
Liked: 19 times
Joined: Oct 17, 2014 8:09 am
Contact:

Re: I/O device error / Cannot get file size - 9.5 Related?

Post by Didi7 » Mar 16, 2017 11:01 am

Today everything worked find agagin! Will post constantly now, as I suppose it has something to do with v9.5 or v9.5 Update 1.

With VBR 9.0 I never had those Errors!

Regards,
Didi7
Using the most recent Veeam B&R in 10 different environments now and counting!

BGA-Robert
Service Provider
Posts: 48
Liked: 7 times
Joined: Feb 03, 2016 5:06 pm
Full Name: Robert Wakefield
Contact:

Re: I/O device error / Cannot get file size - 9.5 Related?

Post by BGA-Robert » Mar 16, 2017 3:08 pm

We continue to see this error intermittently, maybe about once a week. Different jobs / different VMs. I've been busy with other Veeam support requests. When I get caught up, I'll re-engage support on this issue.

Didi7
Expert
Posts: 275
Liked: 19 times
Joined: Oct 17, 2014 8:09 am
Contact:

Re: I/O device error / Cannot get file size - 9.5 Related?

Post by Didi7 » Mar 16, 2017 3:50 pm

I can confirm the same behavior here ...

Errors intermittently, maybe about once a week, different jobs, different VMs

and supposedly beginning with VBR 9.5 or VBR 9.5 Update 1. Not sure!

Needs to be further investigated here as well.

Regards,
Didi7
Using the most recent Veeam B&R in 10 different environments now and counting!

Nadziratelb
Novice
Posts: 5
Liked: never
Joined: Aug 06, 2014 11:38 pm
Full Name: Alexander
Contact:

Re: I/O device error / Cannot get file size - 9.5 Related?

Post by Nadziratelb » Mar 24, 2017 2:29 am

At us such error began to appear after installation 9.5.
Update 1 did not correct this error.

Mike Resseler
Veeam Software
Posts: 4686
Liked: 498 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: I/O device error / Cannot get file size - 9.5 Related?

Post by Mike Resseler » Mar 24, 2017 6:23 am

To everyone that has this issue. Do you all have a support case logged? If there are more we can use those. The more logs can be investigated the better to find the final root cause

Thanks
Mike

Nadziratelb
Novice
Posts: 5
Liked: never
Joined: Aug 06, 2014 11:38 pm
Full Name: Alexander
Contact:

Re: I/O device error / Cannot get file size - 9.5 Related?

Post by Nadziratelb » Mar 27, 2017 5:34 am

I applied the logs to the case № 02058418
The case is closed, but the problem remains.

crackocain
Service Provider
Posts: 90
Liked: 7 times
Joined: Dec 14, 2015 8:20 pm
Full Name: Mehmet Istanbullu
Location: Turkey
Contact:

Re: I/O device error / Cannot get file size - 9.5 Related?

Post by crackocain » Mar 27, 2017 8:15 am

Case #02110634 was created.

Timed out waiting for asynchronous device I/O completion

We are experiencing this issue Wan Accelerator Replication jobs or backup jobs.
EMC, IBM Storage Specialist - VMCEv9 - Dosbil - Turkey

Post Reply

Who is online

Users browsing this forum: apopkov, Bing [Bot], DonZoomik, foggy and 67 guests