Veeam 8 possible CBT affecting issues

VMware specific discussions

Veeam 8 possible CBT affecting issues

Veeam Logoby KevinK » Thu Jan 15, 2015 11:45 am

Since moving to Veeam 8 (and Patch 1 consecutively) 3 of our largest file servers have had CBT problems, spread over two different sites.

The first one was an incremental that failed last Wednesday with the following error;

Code: Select all
Error: Transmission pipeline hanged, aborting process. ChannelError: ConnectionReset


At the time all other backups going through that Veeam server appeared to be servered;

Code: Select all
Error: An unexpected network error occurred. Failed to write data to the file [\\XXXXXXXXXXXXXXXXXXXXXXXXXXXX\XXXXXXXXX_Daily2015-01-05T180105.vbk]. Failed to download disk. An existing connection was forcibly closed by the remote host Failed to upload disk. Agent failed to process method {DataTransfer.SyncDisk}.


Afterewards I was unable to consolidate the file server and VMware pinpointed this down to a file lock. Restarted the management agents on the host allowed me to consolidate the VM.

I performed a retry and the same thing happened.

To rule out the incremental, I created a new backup job and performed a full. This was successful.

Performing a full on the datastore job will had previously failed also worked, and the next incremental was successful too.

Yesterday a similar issue affected two file servers in another site, both hosted on two separate datastores.

Code: Select all
Processing xxxxxfs04 Error: The specified network name is no longer available. Failed to write data to the file [\\XXXXXXXXXXXXXX\XXXXXXXXX_Daily\XXXXXXXX_Daily2015-01-12T190132.vbk]. Failed to download disk. An existing connection was forcibly closed by the remote host Failed to upload disk. Agent failed to process method {DataTransfer.SyncDisk}.


I have now kicked of a full backup to attempt to clear the issue.

Both Veeam backup servers are physical with fibre to the SAN and 10Gbps fibre network.
The Data Domain destinations are also on 10Gbps fibre.
We are confident that there were no network issues at the time of these events.
The file servers are 6TB+ each with multiple vDisks.

Anyone else experienced similar?

Cheers

KK
KevinK
Enthusiast
 
Posts: 28
Liked: 10 times
Joined: Wed Apr 24, 2013 9:18 am
Full Name: Kevin Kissack

Re: Veeam 8 possible CBT issues

Veeam Logoby foggy » Thu Jan 15, 2015 12:00 pm

Why do you consider this is related to CBT? Have you contacted support already?
foggy
Veeam Software
 
Posts: 15081
Liked: 1110 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Veeam 8 possible CBT issues

Veeam Logoby KevinK » Thu Jan 15, 2015 12:14 pm

Sorry I should have said CBT affecting - as I've managed to self resolve the issue, I thought I'd comment on here in case others were having the same problem.

If it happens again, I'll raise a support ticket.
KevinK
Enthusiast
 
Posts: 28
Liked: 10 times
Joined: Wed Apr 24, 2013 9:18 am
Full Name: Kevin Kissack

Re: Veeam 8 possible CBT affecting issues

Veeam Logoby foggy » Thu Jan 15, 2015 12:19 pm

Make also sure to update the thread with the ticket ID in this case, for further reference. Thanks.
foggy
Veeam Software
 
Posts: 15081
Liked: 1110 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Veeam 8 possible CBT affecting issues

Veeam Logoby KevinK » Mon Jan 19, 2015 9:57 am 1 person likes this post

Case #00730062
KevinK
Enthusiast
 
Posts: 28
Liked: 10 times
Joined: Wed Apr 24, 2013 9:18 am
Full Name: Kevin Kissack

Re: Veeam 8 possible CBT affecting issues

Veeam Logoby bretesq » Mon Jan 19, 2015 3:32 pm

Please update this as you receive support from Veeam. We are experiencing the same exact issue as well. Thanks!
bretesq
Service Provider
 
Posts: 5
Liked: 1 time
Joined: Mon Jan 19, 2015 3:19 pm
Full Name: Bret Esquivel

Re: Veeam 8 possible CBT affecting issues

Veeam Logoby KevinK » Tue Jan 20, 2015 2:00 pm

Veeam asked me to do the following, but never provided a reason.

Backup Infrastructure -> Microsoft Windows -> Find repository server for this job -> Properties -> Credentials -> Ports -> Choose "Run server on this side"

I applied the setting and it made no difference. We're seeing sporadic issues propagating over our two biggest sites now - both have physical backup servers with dual fibre connections to the SAN and Network.

I'd like to see this escalated as the large file servers which can take over 10 hours to backup are being missed.

Again this was faultless till V8.
KevinK
Enthusiast
 
Posts: 28
Liked: 10 times
Joined: Wed Apr 24, 2013 9:18 am
Full Name: Kevin Kissack

Re: Veeam 8 possible CBT affecting issues

Veeam Logoby jabroni » Tue Jan 20, 2015 4:56 pm

KevinK wrote:Again this was faultless till V8.


I have also been seemingly randomly having the same problem since V8

2015-01-19 21:02:45 :: Processing XXXXX Error: The specified network name is no longer available.
Failed to write data to the file [\\XXXXX\vol_VeeamBackup\backups\Tier 1 VM Backup\Tier 1 VM Backup2015-01-19T210127.vib].
Failed to download disk.
An existing connection was forcibly closed by the remote host
Failed to upload disk.
Agent failed to process method {DataTransfer.SyncDisk}.



KevinK wrote:Veeam asked me to do the following, but never provided a reason.

Backup Infrastructure -> Microsoft Windows -> Find repository server for this job -> Properties -> Credentials -> Ports -> Choose "Run server on this side"


I'm assuming this might be relevant because you are using a Windows SMB share for the backup repo correct?

In my case we are using a NetApp 8.2.2P1 7-Mode CIFS share as the backup repo target.

I cannot reproduce the error consistently and it seems to occur at random. There are no errors in the Veeam (or proxy) server event log nor on the Netapp or on the switch logs.

Not sure that it's relevant but we are using a 4x1GBps link agg for our networking.
jabroni
Novice
 
Posts: 4
Liked: never
Joined: Tue Jan 20, 2015 4:08 pm

Re: Veeam 8 possible CBT affecting issues

Veeam Logoby KevinK » Tue Jan 20, 2015 6:20 pm

Veeam asked me to do the following;

create the DataMoverLocalFastPath (DWORD) registry value under HKLM\SOFTWARE\Veeam\Veeam Backup and Replication, and set it to the following value: 2


From the patch 1 notes
Performance enhancements
Added experimental support for direct data movers communication when both are running on the same server (for example, when backing up to a local storage on backup proxy server). If your local backup jobs report Network as the bottleneck, and you see high load on some backup proxy server NICs when the data was supposed to stay local to the server, your jobs performance is likely to benefit from this behavior modifier. To enable alternative data exchange path, create the DataMoverLocalFastPath (DWORD) registry value under HKLM\SOFTWARE\Veeam\Veeam Backup and Replication, and set it to the following values:
0: Default behavior (no optimizations)
1: Data exchange through TCP socket on the loopback interface (faster)
2: Data exchange through shared memory (fastest)


Since doing so the throughput of the backup server has increased and remained stable, hovering around 500MB/s (@70% CPU) at the Data Domain [below]

Image

The reported bottleneck has now moved to source rather than network.

I'll let you know if its improved the backup reliability.
KevinK
Enthusiast
 
Posts: 28
Liked: 10 times
Joined: Wed Apr 24, 2013 9:18 am
Full Name: Kevin Kissack

Re: Veeam 8 possible CBT affecting issues

Veeam Logoby davidkillingsworth » Tue Jan 20, 2015 8:02 pm

I suddenly have this issue after upgrading from Veeam 7 to Veeam 8 Patch 1. Here's the full details of my setup and issues and what I have tried.

vmware-vsphere-f24/unable-to-retrieve-next-block-transmission-command-t13109-75.html#p133856

I tried what KevinK suggests:
Created DataMoverLocalFastPath (DWORD) registry value under HKLM\SOFTWARE\Veeam\Veeam Backup and Replication, and set it to the following value: 2

I did this on both the repository and proxy servers, and it did help. The backup job wrote 46% of the disk, then it failed. Before, it was only writing a GB or two.

I'm dead in the water.
davidkillingsworth
Enthusiast
 
Posts: 29
Liked: 4 times
Joined: Mon Nov 21, 2011 5:41 am
Location: Hong Kong

Re: Veeam 8 possible CBT affecting issues

Veeam Logoby KevinK » Tue Jan 20, 2015 9:24 pm

On the performance front the good new continues in a second site. I've activated full backups on all the jobs that failed and I'm seeing similar results - 500+MB/s @85% CPU on the DD.

Image

No failures yet. Perhaps this is masking an issue :)
KevinK
Enthusiast
 
Posts: 28
Liked: 10 times
Joined: Wed Apr 24, 2013 9:18 am
Full Name: Kevin Kissack

Re: Veeam 8 possible CBT affecting issues

Veeam Logoby KevinK » Tue Jan 20, 2015 10:57 pm

All jobs have now failed for that site. Veeam backup server is physical.

Code: Select all
Error: Shared memory connection was closed. Failed to upload disk. Agent failed to process method {DataTransfer.SyncDisk}. Exception from server: An unexpected network error occurred. Failed to write data to the file [\\xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx13_Daily2015-01-20T185000.vbk]. Failed to download disk.


Code: Select all
20/01/2015 22:44:13 :: Error: An unexpected network error occurred.
Failed to write data to the file [\\xxxxxxxxxxxxxxxx\xxxxxxxxx_Daily2015-01-20T185000.vbk].
Failed to backup text locally. Backup: [veeamfs:0:2039bbaa-43db-4f1d-a950-5f73189fcce2 (vm-92223)\summary.xml@\\xxxxxxxxxxxxxxxxxxxxxxxxxxx_Daily2015-01-20T185000.vbkVBK: 'veeamfs:0:2039bbaa-43db-4f1d-a950-5f73189fcce2 (vm-92223)\summary.xml@\\xxxxxxxxxxxxxxxxxStore13_Dail


I don't think this is an issue with the Data Domain as Jabroni uses Netapp.
KevinK
Enthusiast
 
Posts: 28
Liked: 10 times
Joined: Wed Apr 24, 2013 9:18 am
Full Name: Kevin Kissack

Re: Veeam 8 possible CBT affecting issues

Veeam Logoby bretesq » Wed Jan 21, 2015 11:55 pm

We are using a Data Domain as the target; Not having this issue on any other client servers running Veeam 8 Patch 1, just the one with the DD610 as a target.

It is sporadic on which VMs fail, some will work on a retry, while others never work. Same error "ChannelError: ConnectionReset". Is there a supported way to downgrade from patch 1? We need to get these backups rolling again.
bretesq
Service Provider
 
Posts: 5
Liked: 1 time
Joined: Mon Jan 19, 2015 3:19 pm
Full Name: Bret Esquivel

Re: Veeam 8 possible CBT affecting issues

Veeam Logoby KevinK » Thu Jan 22, 2015 10:09 am

EMC couldn't find any issues on the Data Domain, so it's back with Veeam support.

We're missing business critical backups for primary file servers at the moment.
KevinK
Enthusiast
 
Posts: 28
Liked: 10 times
Joined: Wed Apr 24, 2013 9:18 am
Full Name: Kevin Kissack

Re: Veeam 8 possible CBT affecting issues

Veeam Logoby KevinK » Thu Jan 22, 2015 10:58 pm 2 people like this post

Today with Veeam we disabled the IP4 Checksum offload on the 10Gbps NICs. Also on one of the servers/sites we retried a constantly failing job at the same time as using a Veeam supplied pftest.exe to write high amounts of data to the Data Domain.

All todays jobs and the remaining failed job complete successfully, even with the fptest.exe hammering the DD with 100MB/s traffic.

I will monitor over the next few days - perhaps the offload settings helped or it was a fluke. Stay tuned!
KevinK
Enthusiast
 
Posts: 28
Liked: 10 times
Joined: Wed Apr 24, 2013 9:18 am
Full Name: Kevin Kissack

Next

Return to VMware vSphere



Who is online

Users browsing this forum: Bing [Bot] and 57 guests