Issues backing up sophos UTM

[sonicasuk]

Case # 00516615

Veeam version 7.0.0.771

I wonder if anyone could help me with this error that i recieve when trying to backup a Sophos UTM Appliance.

Things ive checked \ done so far.

checked the forums and support site for fixes.
Rebooted everything, Vm, ESX host, Veeam backup server.

DNS = is good no issues here am able to resolve the appliance name
Firewall = Ports are open from the backup server to the Vm and the other way around.
Storage = no issues here am able to conntect to the data store and download the .vmx file to my desktop.
Ping = can ping between vm and Veeam backup server both directions. there are no losses when pinging the servers.

I have seen the backup running sometimes it reaches 40% complete and then i recieve the following error :-

13-2-2014 9:40:35 :: Processing SERVERNAME Error: Client error: VDDK error: 10041633552057.NBD_ERR_NETWORK_CONNECT
Exception from server: VDDK error: 10041633552057.NBD_ERR_NETWORK_CONNECT
Unable to retrieve next block transmission command. Number of already processed blocks: [2692].
Failed to perform data backup.

Other times the backup failes without starting with the following message.

SERVERNAME Error: Client error: Cannot get service content. Soap fault. TimeoutDetail: 'connect failed in tcp_connect()', endpoint: 'https://IP address:443/sdk' SOAP connection is not available. Connection ID: [IP ADDRESS]. Failed to create NFC download stream. NFC path: [nfc://conn:IP ADDRESS,nfchost:host-50,stg:datastore-30@SERVERNAME/SERVERNAME.vmx].

Many thanks in advance.

Kind regards

Sonic

[veremin]

As mentioned in the adjacent thread, sometimes you might get the said error when existing network is unstable. Even though you've checked it already, it might be worth testing network again, this time, verifying all backup connections present in your environment (backup management, backup repository, backup proxy, vcenter, etc.).

Also, I'm wondering what product version you're using.

Thanks.

[sonicasuk]

Hi Vladimir,

We are using Veeam version 7.0.0.771 as mentioned above

The strange thing is that i have another backup job for the same enviroment and this backup runs without issues.
I will have another look at the network and see if i can find any issues,

Very strange.

Kind regards
sonic

[veremin]

We are using Veeam version 7.0.0.771

It must have slipped out of my attention.

I will have another look at the network and see if i can find any issues,

I believe the support team will be able to confirm your environment and find root cause of the said behavior after analyzing backup logs.

Thanks.

[sonicasuk]

No worries it can happen. Time for coffee perhaps

Thanks for the update ill wait for contact with the support team.

when we resolve this issue ill post the resolution here if any resolution is needed.

Many thanks.

Kind regards
Sonic

[sonicasuk]

retried the backup while running a ping to

1) the vm
2) the esx hosts
3) the veeam backup server
4) the vcenter server

backup failes at 40% and the ping carries on with out a problem

[Vitaliy S.]

Do you backup this VM in network mode? Have you tried hotadd? Do you see the same issue?

[veremin]

If you're unable to find the root cause of the said behaviour, then, it's a time to open a ticket with our support team, letting them investigate it directly. Thanks.

[sonicasuk]

Hi Gents,

i do have a support case open at this moment and the guys \ girls are looking into the logs.

i havent tried the hot add option, is it possible to give me a little more information over this? ( is this with the different proxy servers that i have selected at the moment automatic selection is selected.)

i did have an issue with one of the esx hosts, this issue has been solved but the issue with the backup still persists.

Many thanks for the reply.

Kind regards
Sonic

[Vitaliy S.]

In order to use hotadd backup mode you should have a proxy server on the host that has access to the storage where the VM you're backing up is located. Can you please verify what backup mode is currently used? You can check that by looking at the corresponding backup job session.

[sonicasuk]

Hi Safaarov,
At this moment we are not using the hotadd option, i was thinking about this option but came to the conclusion that as the other backups from this Esx cluster work perfectly.
however if there is no other way ill install a proxy on the host and see if this helps.

Kind regards
Sonic

[Vitaliy S.]

Sonic, please keep us updated on this. BTW, what did you support engineer say after reviewing all log files?

[sonicasuk]

Hi,

I have had a Phone conversation with the support engineer today.

we looked at quiet a few things
1) config of veeam
2) Proxy config ( no hotadd option at the moment)
3) connection between host and veeam sever
4) jumbo frames off on network adapter.
5) added 1 host as a stand alone host within Veeam ( still didnt work)
6) migrated the guest vm to a different data store.
7) from server a we were able to download the VMWare log, from the UTM we were unable to download the log file, even though both servers reside on the same datastore.

The engineer was very helpful and knowledgable however this issue looks like a new issue.

"normally we dont see an error message that long "10041633552057.NBD_ERR_NETWORK_CONNECT" so perhaps its a new issue idk.

However we are not giving up on this one ive mailed my results to the engineer and will see whats next.

I begining to think ( but am not sure) that there may be an issue on the underlying storage, is it possible that the folder on the Lun for server A has different rights to the folder for server B. ( just clutching at straws now :lol)

Ill have a look at the underlying storage to see if i can find anything,

Kind regards
Sonic

[sonicasuk]

Hi all,

well ive just had some feed back from the support team.
It seems that the server is failing at the same block each time, this can indicate that the vmdk is corrupt. if this is so then there is nothing to be done for this server.

An idea i had was to clone the server and try a backup with the clone, if this works then we can be 100% sure that the issue is with VMWare and not Veeam

as soon as ive tested this ill post back my results.
Kind regards
Sonic

[sonicasuk]

well i created a clone of the server. which should then create a new VMDK file.

after doing this i created a new backup job and ran it today.

unfortunatly i recieve the same issue, the server gets to about 40% and then failes with the error Error: Client error: VDDK error: 10041633552057.NBD_ERR_NETWORK_CONNECT
Exception from server: VDDK error: 10041633552057.NBD_ERR_NETWORK_CONNECT
Unable to retrieve next block transmission command. Number of already processed blocks: [3598].
Failed to perform data backup.

I have updated my case with the support guys and thought i would post my findings upto now here.

Kind regards
sonic

[sonicasuk]

looked at this issue again with support.
we decided to install another proxy on the enviroment.
the backup is getting further however we now have to ajust the firewall to allow all the communication.
so it looks like its solved.
I will post back when i have a sucessfull backup.
Kind regards
Sonic

[sonicasuk]

ARRGGHHHHH. I have recieved a new support engineer because the engineer that i was dealing with is not available today.

I have been busy with this issue for a month now and the first question the engineer asked was have i moved the vm to another datastore.

Is there a call registration system available for the engineers because i dont want to have to repeat everything that ive already done.

sorry but i had to vent a little frustration over this.
thanks.

[Vitaliy S.]

Please reference to this thread and to your existing support ticket (yes we do track all email conversations in the support system) in order not to answer all these questions again.

[sonicasuk]

ok... this is very strange.

ive just noticed that the job has started working... i have no idea why we havent changed anything regarding configuration netowrking etc etc.

so im confused that it is all of a sudden working. im also very happy that this is working as it should now but very curious as to why.

I have a dial in session planned for tomorrow with one of the support eningneers perhaps then he can shed some light on why its working.

perhaps swearing and cursing the job has helped

[sonicasuk]

Well the issue is back.
the machine failes with the same error :

end of file unable to retrieve next block transmission command. ill open another support ticket and start the trouble shooting process all over again

[MartinM]

It seems like I'm encountering the same issue as you. Trying to backup a VM running SBS2011 on ESXi, through Veeam. I'm also getting "Unable to retrieve next block transmission command", and I'm getting it at the exact same block every single time. My other VM's are backed up without issues, but these aren't very big in physical size. My SBS2011 is around 500 gigs, so it takes a while to reach the error (almost at the end of the job, every time).

I also have a support ticket running, and so far the supporter asked me to try and up the memory for the VM from 4 to 8 gigs. I just did that, and I'll see how it works out tonight when the job runs again - unless the support comes up with something before that.

I'll try and keep you posted on this. I'm thinking, that if I get a resolve on my problem, it might help you as well (and vice versa)

Kind regards
Martin

[Vitaliy S.]

Martin, try changing backup mode or running active full against this VM, should help to nail down the issue. Thanks!

[MartinM]

I have an update on our problem. We also encountered the "Unable to retrieve next block transmission command", and at the exact same block every single time. We've been through a LOT of support work, and got our ticket elevated to the "Insane Veeam Ninja SWAT Team" - they couldn't get it working either. Not that strange though, since the problem was a physical one on our server.

One of our VMDK's (a big 320gig one) had a bad block, and as if that wasn't enough, the bad block was right in the middle of our Exchange DB.

What I did to fix it was:

• - Make a new virtual disk, big enough to keep all data form the broken disk.
  - Mount it in the 2008R2 disk management.
  - Robocopy (google it, seriously!) all files from the broken disk to the new one.
  - Created a new Exchange DB on the new disk and moved all mailboxes to this DB. Surprisingly this didn't fail, even though 2 of the mailboxes was a bit bugged from the bad block.
  - Change the shares (that was obviously shared from the broken disk) in group policy, to point to the new disk.

Voila! Everything worked with a insane minimum of downtime, and the backup now blasts through every night at around 140mb/sec.

[Vitaliy S.]

"Insane Veeam Ninja SWAT Team"

You just made my day Good to know that you're back on track with this VM.

[rgiovannelli]

Hi all,
my customer has this problem. Several VM's backups fail with this error:

Code: Select all

"Error: Client error: VDDK error: 1245540515841.NBD_ERR_GENERIC Exception from server: VDDK error: 1245540515841.NBD_ERR_GENERIC Unable to retrieve next block transmission command. Number of already processed blocks: [0]. Failed to perform data backup."

The customer Veem backup server version is 7.0.0.871.

Thanks in advance

Roberto

[veremin]

Hi, Roberto,

Kindly, make your customer update his product installation to the supported version and open a case with our support team for further investigation.

Thanks.