CNfsViDiskFast::Read error

[rferrigno]

I'm getting a lot of these all of the sudden and I'm not sure why. Has anyone seen this? V8.0.0.917
It doesn't happen on all the vms but a good amount.

1/5/2015 5:16:20 PM :: Error: CNfsViDiskFast::Read error: %s
Failed to upload disk.
Agent failed to process method {DataTransfer.SyncDisk}.

Veeam Support - Case # 00718038

[ckbrou]

I get this same error from time to time with v8. Any resolution yet? I had it over the weekend on 2 VMs, ran an active full again and got the same error on the two VMs again.

[Vitaliy S.]

OP's ticket is in the research/resting stage, so in order to have a quicker resolution, I would also recommend contacting our support team. Thanks!

[tbrownfield]

We're also receiving this error on one of our mission critical VMs after upgrading to v8. Support had me install the latest patch and a hotfix but it hasn't resolved the issue. Case #00723486.

[tehfiskk]

same here. waiting for support to contact me.

case#00729949

[Mattias]

This happened to me to. In my case there was with NFS and NetApp storage integration.

I got a "custom made" veeam agent that i installed on the server and all proxies then the error did go away.

You can probably refer to Case # 00693600 (my case number) at veeam support and they can see the issue and what file they sent me. The latest info i have is that its not fixed in patch 1 (as you notice) but should be fixed in patch 2.

Regards Mattias

[ckbrou]

Thanks for the uddate. I also am using NFS and NetApp storage integration but my issue has gone away by itself. I know what to look for if it comes back though!

[cpurn]

HI

We have just upgraded to v8 with P1 and are consistently encountering this issue with one VM.

I have raised a support case #00740997 and has been provided an updated VeeamAgent.exe file, restarted the Veeam server (which also has proxy role) and re-run the job. The job still bombs out with the same error message.

Has anyone had any similar experience? This is critical for us as the issue is impacting our exchange server.

thanks

[foggy]

Christian, that means that your particular issue has different nature from the one that was supposed to be fixed by the updated file. Please continue investigating with your support engineer.

[cpurn]

HI Foggy

The case has been escalated from Level 1 (Australia / Level 1) to the next tier, I have had not much progress in the last 3 days from Australia Support Engineer regarding the case and had to insist an escalation of this issue which is now happening.

I will provide feedback about my progress once (and the resolution in the end).

Cheers

[yatesy]

I am suffering from a similar issue also, since changing to Storage Snapshots. Any help would be gratefully received! V8.0.0.917
Also experiencing slower speeds since changing to Storage Snapshots...

05/02/2015 22:50:25 :: Processing servername Error: CNfsViDiskFast::Read error: %s
Failed to upload disk.
Agent failed to process method {DataTransfer.SyncDisk}.
Exception from server: CNfsViDiskFast::Read error: %s
Unable to retrieve next block transmission command. Number of already processed blocks: [71904].
Failed to download disk.

Veeam Support - Case # 00744433

[rferrigno]

This seems to be an issue where the storage snapshots cannot handle the same load as hot add backups. It's strange because it's the same storage but that seems to be the case. If I turn off parallel processing it works but takes way too long so now I am capping it to 5 jobs at a time per datastore and that seems to be working. I'll let it run for a few days and post the results.

[rferrigno]

By the way updated Veeamagent.exe did not work in my case.

[Scouserste81]

I work with Yatesy, posted above.

We have also had the error and been given an updated Veeam agent however are still seeing the same error.

We are using netapp storage snapshots on NFS presented volumes.

When the system runs over 10Gbps network the system reads data a great speeds, but once the error occurs the speed slows to a crawl.

We have also found restarting the Veeam agent service on the proxies speeds the system up on a retry however errors again.

There does appear to be a correlation to the error and virtual machines that are set for application aware, however we are also suspicious that the issue can be caused by to many jobs queueing waiting for Veeam infrastructure.

At the moment the application aware correlation is the strongest Link as we backed up over 2 TB without issue on a file server(non app aware) but all servers that are app aware have failed and failed multiple times.

[cpurn]

Hi All

Just thought I might share my experience with this issue.

In our environment, the new veeam agent makes no difference with the error. Tried fiddling with the parallel and concurrent by dropping / increasing the number and this made no difference.

In the end the fix the fix was to turn OFF Storage Integration / Use Storage Snapshots and our backup has been running consistently with similar speed from v7.

Looking forward for Dev to provide a fix to this issue, this feature in fact was the most appealing factor for us to upgrade to v8 and we thought we had waited *long enough* to go through the early stages release bug fixes.

[veremin]

Hi, Christian, what were the result of support investigation? Did you notify them that updated agent hadn't made any difference? I'm wondering since merely can disabling storage integration be considered as a fix. Thanks.

[cpurn]

Hi, Christian, what were the result of support investigation? Did you notify them that updated agent hadn't made any difference? I'm wondering since merely can disabling storage integration be considered as a fix. Thanks.

Hi V.Eremin

Support initially provided Fix_41359 (VeeamAgent.exe update), we had a remote session support and the outcome from the 1st webex session support was:
- turn off storage snapshot - to switch back to v7 mode.
- turn off parallel processing
- reduce max number of operation on repository

The idea was to start the backup again using the low baseline above and incrementally "tweak" the 3 parameters above.

I went on for 1.5 weeks with the same job and concluded in the end the parallel processing and max number operation on repository do not trigger the backup instability but when the storage snapshot is switched on - the backup bombs out straight away. I was made aware that the patch #41359 is the only available solution available for this particular problem at this point in time.

I informed support (message below) with the findings and posted my last post in this forum thread.

---
HI Mark

I have been doing some tweaking with the backup configuration, it seems to
me that Storage Snapshot feature is unstable and is the cause of all
issues.

Tried Storage Snapshot technology on different settings, i.e. no parallel
processing, concurrent max tasks, etc and none of the makes any difference.

I'm now running the conventional backup and stability is good. Certainly
looking forward for future patch with this bug and hopefully The Dev team
can resolve the issue.

Thanks
---

I received a reply back the following day that patch #41359 requires that both x86 and x64 VeeamAgent.exe to be replaced on all veeam servers (and proxy), initially only x64 veeamagent.exe was replaced in our environment as all of our Veeam Server + proxy run WIndows Server 2008 R2 (x64).

My last night job ran successfully 100% with Storage Snapshot feature switched on, I have provided this feedback to support.

I hope this would help others with similar issue.

PS: Veeam Support case #00740997

[veremin]

Thank you, Christian, for coming back and providing such a detailed response. Surely it will help others.

[murg235@lni.wa.gov]

I just opened 00834957 for this issue as well and it has started doing this since we went to the Storage Snapshots. I will update when I know more.

[vee789]

Hello Christian
Do you know why disabling the storage integration helped in your case?
I mean storage integrations has many advantages. Would be too bad to have to turn it off... Any news on this from your side?

[foggy]

As far as I can get, the hotfix that Christian has got from support allowed him to "run successfully 100% with Storage Snapshot feature switched on". Also, this hotfix was included into Veeam B&R Update 2, so if you have it installed, no need to worry about this issue.