I/O device error / Cannot get file size - 9.5 Related?

Availability for the Always-On Enterprise

Re: I/O device error / Cannot get file size - 9.5 Related?

Veeam Logoby Didi7 » Mon Mar 27, 2017 11:50 am

The word from Gostev:
Just before the weekend, VMware has issued this rush post regarding an issue with VMware Tools versions 9.10.0 up to 10.1.5 that causes network packets to drop due to the fact that all network traffic is serviced by a single guest CPU. It immediately occurred to me that the issue may be especially impactful on virtual backup proxies, because they pump hundreds of MB per second through the network stack – while their CPU is already very busy with all the data processing. So, I thought that this bug could potentially be the reason for those intermittent backup job failures some customers have been having for a while now (the issue where job retry always helps)? I guess we'll find out soon, as soon as VMware Tools are updated.

*************************************************************************************************

Interesting! I just wonder, why this hasn't been a problem with Veeam B&R 9.0 ?

Regards,
Didi7
Using Veeam Backup & Replication 9.5 Update 2 on every backup server here!
Didi7
Expert
 
Posts: 241
Liked: 16 times
Joined: Fri Oct 17, 2014 8:09 am

Re: I/O device error / Cannot get file size - 9.5 Related?

Veeam Logoby Didi7 » Wed Mar 29, 2017 6:01 am

Hello again,

just for your information. Here in our environment, VMware Tools 9.4.10 build-2068191 from ESXi 5.5 Update 2 is used since November 2014 and we never had such problems with Veeam B&R 7, 8 and 9 but with 9.5 !!!

Regards,
Didi7
Using Veeam Backup & Replication 9.5 Update 2 on every backup server here!
Didi7
Expert
 
Posts: 241
Liked: 16 times
Joined: Fri Oct 17, 2014 8:09 am

Re: I/O device error / Cannot get file size - 9.5 Related?

Veeam Logoby Didi7 » Sun Apr 02, 2017 9:43 am

Severe I/O errors again this weekend. 3 VMs of 12 in one backup job reported ...

Error: The request could not be performed because of an I/O device error

They were retried by VBR and worked again!

Really fishy atm!

Regards,
Didi7
Using Veeam Backup & Replication 9.5 Update 2 on every backup server here!
Didi7
Expert
 
Posts: 241
Liked: 16 times
Joined: Fri Oct 17, 2014 8:09 am

Re: I/O device error / Cannot get file size - 9.5 Related?

Veeam Logoby vmniels » Sun Apr 02, 2017 9:50 am

Did you already contacted support in regards to this? Maybe there is something else going on which may become clear by reading the logs. Please post the support case ID as well for future reference.
VCP-DCV
Veeam Certified Architect (VMCA)
http://foonet.be
vmniels
Veeam Software
 
Posts: 1633
Liked: 362 times
Joined: Mon Jul 15, 2013 11:09 am
Full Name: Niels Engelen

Re: I/O device error / Cannot get file size - 9.5 Related?

Veeam Logoby Bredero » Tue Apr 04, 2017 11:19 am

This seems eerily like an intermittent issue we have had with exchange backups at a customer. Physical Backup Proxies, RSS disabled, support case ID# 02092977.

Support initially blames UAC and antivirus but that seems unlikely. Resources for the backup server (which is a VM) were increased and issues seem to be less.

Job usually completes succesfully but truncation of transaction logs still fails frequently.
Bredero
Lurker
 
Posts: 1
Liked: never
Joined: Sun Apr 05, 2015 11:12 am

Re: I/O device error / Cannot get file size - 9.5 Related?

Veeam Logoby Nadziratelb » Tue Apr 04, 2017 9:38 pm

We tried to remove the antivirus and disable UAC, did not help.
The problem manifests itself only on large tasks, from 50 to 70 VM.
Support advised only to reduce the number of machines in the jobs.
Nadziratelb
Novice
 
Posts: 5
Liked: never
Joined: Wed Aug 06, 2014 11:38 pm
Full Name: Alexander

Re: I/O device error / Cannot get file size - 9.5 Related?

Veeam Logoby bavosod » Wed Apr 05, 2017 11:32 pm

Please, any news about this issue? i have had the same error even in jobs with just one VM included.

Timed out waiting for asynchronous device I/O completion

Tks...
bavosod
Novice
 
Posts: 3
Liked: never
Joined: Tue Mar 21, 2017 11:53 pm
Full Name: Daniel Bavoso

Re: I/O device error / Cannot get file size - 9.5 Related?

Veeam Logoby Mike Resseler » Thu Apr 06, 2017 4:45 am

Daniel,

Do you have a support case? Please post the case ID here. The more our teams can investigate. Plus, it is rather important to know that if this specific issue gets to support multiple times, it will get escalated to the QC team faster

Thanks
Mike
Mike Resseler
Veeam Software
 
Posts: 3382
Liked: 384 times
Joined: Fri Feb 08, 2013 3:08 pm
Location: Belgium, the land of the fries, the beer, the chocolate and the diamonds...
Full Name: Mike Resseler

Re: I/O device error / Cannot get file size - 9.5 Related?

Veeam Logoby foggy » Thu Apr 06, 2017 11:42 am

In fact there's a hotfix for the issue that might potentially be causing this behavior, so contacting support is indeed the most expected step (you may refer to issue 96612, bug 96610).
foggy
Veeam Software
 
Posts: 15295
Liked: 1133 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: I/O device error / Cannot get file size - 9.5 Related?

Veeam Logoby Didi7 » Mon Apr 10, 2017 10:41 am

Hello,

more severe errors this weekend. Gets worser from week to week. As soon as I find time, I will open a case. Retries didn't work for all VMs this time!

Ok, in this case a problem with VMware Tools in one guest [Guest processing skipped (check VMware Tools status)].

Regards,
Didi7
Using Veeam Backup & Replication 9.5 Update 2 on every backup server here!
Didi7
Expert
 
Posts: 241
Liked: 16 times
Joined: Fri Oct 17, 2014 8:09 am

Re: I/O device error / Cannot get file size - 9.5 Related?

Veeam Logoby Didi7 » Mon Apr 10, 2017 10:47 am

@foggy:
Is the problem or bug still VMware Tools related, as was pointed out by Gostev in one of his recent mail blogs?

Regards
Didi7
Using Veeam Backup & Replication 9.5 Update 2 on every backup server here!
Didi7
Expert
 
Posts: 241
Liked: 16 times
Joined: Fri Oct 17, 2014 8:09 am

Re: I/O device error / Cannot get file size - 9.5 Related?

Veeam Logoby foggy » Fri Apr 14, 2017 5:52 pm

The mentioned bug is related to repository components experiencing very high load. Not necessarily the one you're seeing, but behavior looks somewhat similar.
foggy
Veeam Software
 
Posts: 15295
Liked: 1133 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: I/O device error / Cannot get file size - 9.5 Related?

Veeam Logoby P4Tr1ck » Thu Apr 27, 2017 6:20 am

hey community,
we are getting the same message "Timed out waiting for asynchronous device I/O completion". in backup summary it is stated as a warning status, no error.
we are seeing the warning nearly every backup cycle. just one or two vms each night but concerning different vms, not always the same ones.
this behaviour is new since upgrading veeam b&r 9.0.0.1715 to 9.5.0.823

any ideas on this?
P4Tr1ck
Novice
 
Posts: 5
Liked: 2 times
Joined: Thu Apr 27, 2017 6:09 am

Re: I/O device error / Cannot get file size - 9.5 Related?

Veeam Logoby Shestakov » Thu Apr 27, 2017 9:32 am

Hello P4Tr1ck,
Please contact the support team to check if above-mentioned hot fix covers your case.
Thanks!
Shestakov
Veeam Software
 
Posts: 5141
Liked: 430 times
Joined: Wed May 21, 2014 11:03 am
Location: Saint Petersburg
Full Name: Nikita Shestakov

Re: I/O device error / Cannot get file size - 9.5 Related?

Veeam Logoby AndyGDIT » Wed May 31, 2017 12:03 pm

Did this fix come in V9.5 U2? We havent installed it yet, but are seeing the warnings for "Timed out waiting for asynchronous device I/O completion" on some backups.

Andy
AndyGDIT
Enthusiast
 
Posts: 34
Liked: 1 time
Joined: Mon Nov 14, 2011 3:15 pm
Full Name: Andrew Frye

PreviousNext

Return to Veeam Backup & Replication



Who is online

Users browsing this forum: Are, Bing [Bot], Google [Bot], Google Feedfetcher, Yahoo [Bot] and 1 guest