Comprehensive data protection for all workloads
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: I/O device error / Cannot get file size - 9.5 Related?

Post by foggy »

It didn't, as far as I know. Please contact support to get the hotfix.
P4Tr1ck
Novice
Posts: 5
Liked: 2 times
Joined: Apr 27, 2017 6:09 am
Contact:

Re: I/O device error / Cannot get file size - 9.5 Related?

Post by P4Tr1ck » 2 people like this post

we've updated our B&R to 9.5 U2
we didn't get any more warnings since updating (one week ago)
pco
Novice
Posts: 6
Liked: 1 time
Joined: Nov 03, 2011 11:10 am
Full Name: Philippe Combes
Contact:

Re: I/O device error / Cannot get file size - 9.5 Related?

Post by pco »

Hello,

Same behavior here, replication between 2 sites.
Multiple VMs, only 2 VMs got this issue.
Do you have an update or a suggested fix?

Thanks
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: I/O device error / Cannot get file size - 9.5 Related?

Post by foggy »

Hi Philippe, what Veeam B&R version are you at?
pco
Novice
Posts: 6
Liked: 1 time
Joined: Nov 03, 2011 11:10 am
Full Name: Philippe Combes
Contact:

Re: I/O device error / Cannot get file size - 9.5 Related?

Post by pco »

Hello,

Veeam backup and replication 9.5 update2 / 9.5.0.1038
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: I/O device error / Cannot get file size - 9.5 Related?

Post by foggy »

Please contact support for a closer look at whether the hotfix is applicable.
pco
Novice
Posts: 6
Liked: 1 time
Joined: Nov 03, 2011 11:10 am
Full Name: Philippe Combes
Contact:

Re: I/O device error / Cannot get file size - 9.5 Related?

Post by pco » 1 person likes this post

Hello,

Case opened 02261738 since Monday 31/07.

Regards
Philippe
Jackf
Service Provider
Posts: 19
Liked: 4 times
Joined: Mar 29, 2015 9:26 pm
Full Name: Jack Firth
Contact:

Re: I/O device error / Cannot get file size - 9.5 Related?

Post by Jackf »

Hi,

I am experiencing exactly the same issue.

My Support ref is:-

Case # 02348457

Please Help!

I have built a brand new Repo server on 2016 and tried performing a new full backup however the issue still persists.
BGA-Robert
Service Provider
Posts: 60
Liked: 8 times
Joined: Feb 03, 2016 5:06 pm
Full Name: Robert Wakefield
Contact:

Re: I/O device error / Cannot get file size - 9.5 Related?

Post by BGA-Robert »

I know this is a year old post, but we still see intermittent failures with "Cannot get file size".
I've sorta just gotten use to seeing them and knowing that they will succeed on retry. But this does need to be fixed.

Here's the whole error message from last night:

Code: Select all

3/8/2018 8:11:08 PM :: Error: The request could not be performed because of an I/O device error.
WINAPI: Cannot get file size. File: [Device '\\.\PhysicalDrive7'].
Failed to upload disk.
Agent failed to process method {DataTransfer.SyncDisk}.
I've opened a new ticket for this: 02661167
BGA-Robert
Service Provider
Posts: 60
Liked: 8 times
Joined: Feb 03, 2016 5:06 pm
Full Name: Robert Wakefield
Contact:

Re: I/O device error / Cannot get file size - 9.5 Related?

Post by BGA-Robert »

Support hasn't been much help. I'm getting the impression that they haven't seen much of this issue. I may be getting the error randomly, but I do see it on a regular basis, maybe a few times a month. Surely support has some history. I've asked Zach a few times if there's any similar tickets, but he hasn't responded to that. I have been asked for logs and to test a job in network mode. Which doesn't seem like it would prove anything since the error is sooo random. Meaning if I put a job in network mode and it doesn't fail, it would not likely have failed anyhow. Frankly, this error first occurred the day after I installed 9.5 for the first time a year ago and is clearly a 9.5 bug. Support also mentioned it could be a resource issue where the VMware datastore is getting swamped. Even if that's true, Veeam should handle that better and AT LEAST report that. Not "cannot get file size"
sdunkin
Lurker
Posts: 1
Liked: never
Joined: Dec 02, 2016 4:47 pm
Full Name: Sherry Dunkin
Contact:

Re: I/O device error / Cannot get file size - 9.5 Related?

Post by sdunkin »

I have been on 9.5 for a while with no issues. We did however just upgrade vmware from 5.5 to 6.5 and now for the first time I see this error. The vm having the error is actually the backup server, which is not the proxy - I have a another vm for that. We use Nimble storage. Since this is the first time I have ever seen this issue I was wondering if the issue I'm having is similar to these posts and if it ever went away for folks.
BGA-Robert
Service Provider
Posts: 60
Liked: 8 times
Joined: Feb 03, 2016 5:06 pm
Full Name: Robert Wakefield
Contact:

Re: I/O device error / Cannot get file size - 9.5 Related?

Post by BGA-Robert » 3 people like this post

I was escalated to Level II and this issue appears to be resolved.

Looking back at my notes from last time, I uninstalled and reinstalled the VMtools on the Veeam server, since VMware provides the SCSI controller driver. That Veeam server hasn't had the "cannot get file size error" since.

David in Level II looked thru the event logs of the Veeam server currently having these errors and found some matching errors in the System log:

Code: Select all

Log Name:      System
Source:        LSI_SAS
Date:          3/25/2018 4:05:01 PM
Event ID:      11
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      SVR-BACKUP-01.xxxxx.loc
Description:
The driver detected a controller error on \Device\RaidPort0.
Still sorta point to VMware as the culprit.

This time, we decided to switch to a different SCSI controller:
https://kb.vmware.com/s/article/1010398

So I migrated the Veeam server in question from the LSI_SAS SCSI controller to the Paravirtual SCSI controller. This is a resource definition in the VMware settings for the Veeam server. The switch is a little tricky as it's entirely possible to blue screen the server on startup due to a missing driver.

My feeling is that this issue was caused by a bug in VMware that Veeam started tripping with 9.5. The above solution is more of a work-around than a fix as we're simply side-stepping the bug by using a different driver.

FYI, I did test the Paravirtual driver on a couple of test boxes. The first one had no issues after installing using the above KB. The second instance was nothing but blue screens. On that instance, I finally manually downloaded the Paravirtual driver from the VMware tools download directly from VMware and just installed it. After both those experiences, I was able to get that driver into production with no problems.

The Paravirtual driver documentation does say that it's a more efficient driver (less CPU, more throughput). Considering the amount of data moved by the Veeam server, that could be a win too...
dlewellen
Service Provider
Posts: 3
Liked: never
Joined: Sep 05, 2013 5:20 pm
Full Name: Daniel Lewellen
Contact:

Re: I/O device error / Cannot get file size - 9.5 Related?

Post by dlewellen »

So I recently noticed this same issue. Brand new Veeam proxy VM on server 2016. KB2442 has a similar error but my logs didn't match the exact error as this KB showed, and the proxy had the Microsoft Visual C++ 2008 Redistributable Package installed so I skipped this as a fix. I updated VMware tools to the latest version but still had the same problem.

https://www.veeam.com/kb2442

Repair steps:
1. Removed proxy from job
2. Uninstalled Proxy
3. Uninstalled C++ Redistribution packs from Proxy VM
4. Reinstalled C++ Redistribution pack per KB2442,
5. Reinstalled proxy
6. Added proxy to job

Code: Select all

Summary of backup job with single file server in job: 
6/10/2018 :: Unable to hot add source disk, failing over to network mode...  
6/11/2018 :: Backup ran without issues
6/12/2018 :: Backup ran without issues
6/13/2018 :: Backup ran without issues
6/14/2018 :: Backup ran without issues 
6/15/2018 :: Unable to hot add source disk, failing over to network mode...  
6/17/2018 :: Unable to hot add source disk, failing over to network mode...  
6/17/2018 :: Unable to hot add source disk, failing over to network mode...  
6/18/2018 :: Backup ran without issues
6/19/2018 :: Backup ran without issues
6/20/2018 :: Backup ran without issues
6/21/2018 :: Backup ran without issues
6/22/2018 :: Unable to hot add source disk, failing over to network mode...  
6/23/2018 :: Backup ran without issues
6/24/2018 :: Backup ran without issues
6/25/2018 :: Backup ran without issues
6/26/2018 :: Unable to hot add source disk, failing over to network mode...  
6/27/2018 :: Unable to hot add source disk, failing over to network mode...  

Test incrementals today: 

6/28/2018 11:11:18 AM :: Unable to hot add source disk, failing over to network mode...  

Completed Repair steps above.

6/28/2018 11:47:23 AM :: Test completed without issues. 
6/28/2018 12:05:14 PM :: Test completed without issues. 
6/28/2018 12:14:36 PM :: Test completed without issues. 
6/28/2018 1:30:44 PM :: Test completed without issues. 
I'll update if there is any change but this appears to have resolved the issue.
Post Reply

Who is online

Users browsing this forum: VAAirGap706 and 271 guests