On Restore - "existing connection was forcibly closed"

[bhendin]

I'm having an issue restoring a large(?) 500GB disk to a VM.

I have tried twice now and each time this is what I get:

Code: Select all

2/11/2015 10:19:25 AM          Starting restore job
2/11/2015 10:19:26 AM          Locking required backup files
2/11/2015 10:20:18 AM          Starting restore agents on server "10.10.10.10"
2/11/2015 10:20:15 AM          Queued for processing at 2/11/2015 10:20:15 AM
2/11/2015 10:20:15 AM          Required backup infrastructure resources have been assigned
2/11/2015 10:20:17 AM          Preparing next VM for processing
2/11/2015 10:20:17 AM          Using target proxy VMware Backup Proxy [nbd]
2/11/2015 12:44:32 PM Error    Powering off restored VM
2/11/2015 12:44:47 PM          Restoring disk Hard Disk (500.0 GB) 62%
2/11/2015 12:44:48 PM Error    Restore job failed Error: Client error: An existing connection was forcibly closed by the remote host
                               Cannot process [restoreDiskContent] command. Target descriptor is [vddk://<vddkConnSpec><viConn name="10.10.10.10" authdPort="902" vicPort="443" /><vmxPath vmRef="2" datacenterRef="ha-datacenter" datacenterInventoryPath="ha-datacenter" snapshotRef="2-snapshot-4" datastoreName="vm" path="server01.xxx.xxx/server01.xxx.xxx.vmx" /><vmdkPath datastoreName="vm" path="server01.xxx.xxx/server01.xxx.xxx

I did this backup yesterday and tested it by mounting it for an individual file restore and was able to recover files from it.
The only think "strange" about my operation is that the backup repository is stored on an external USB drive since we are kind of in a disaster recovery mode and needed to move this off, this was our only option.

Each time the failure occurs it is at 62%. Since it takes over 2 hours to get this far, I'm not thrilled about having to keep trying.

Right now, I am migrating data from the local veeam server to free up enough space in the local drives to copy this 500GB backup locally to veeam. This should eliminate any issue with the USB drive being a source of the problem.

We had another 250GB drive backed up and restored the same way without issue (though not on the USB), and about 60GB of other full VMs restore (*from* the USB drive).

I'm hoping someone can tell me what's up so when my 500GB is copied locally I can do whatever else might be necessary to minimize this issue. The network infrastructure is all very direct in this environment, and the fact that it timed out at 62% each time doesn't seem to be random.

[foggy]

Ben, I strongly recommend to involve technical support. Please post your case ID for further reference, if you want to keep this thread open.

As to the issue itself, I would try to use another proxy server for restore. Since it is currently using nbd, please try restore through hotadd.

[bhendin]

So, I have some more information which isn't exactly looking good.

I copied the 500GB backup to a local drive on the veeam server.
The job failed again with the same error at the same point.
I then attempted to do a "file restore" instead of a "disk restore" coping over the vmdk files. Here is the log:

Code: Select all

2/12/2015 12:19:32 AM          VM files restore started
2/12/2015 12:20:36 AM          File restored: server01.xxx.xxx_1.vmdk
2/12/2015 1:56:48 AM Error    Failed to restore file: server01.xxx.xxx_1-flat.vmdk Error: Client error: An existing connection was forcibly closed by the remote host
                              Cannot process [restore] command.
                              Exception from server: Failed to decompress LZ4 block: Incorrect decompression result or length (result: '-865272', expected length: '1048576').
                              Unable to receive file.
2/12/2015 1:56:51 AM          VM files restore completed

From everything I am finding, this basically means a corrupt backup file. This essentially puts me up a creek.

My environment here is extremely simple - standalone ESXi 5.5 server with local storage only (about 3TB).
Secondary physical server hosts vsphere and all Veeam backup components and also contains local storage.
Backups go directly to this server.

- Veeam reported NO ERRORS during the backup of this host.
- The drive that it was backed up to was not used for anything after making this backup.
- I was able to do a file level test restore (which would assume Veeam would validate the file as valid)
- The drives that this file was stored in have logged no disk issues and show as clean

The size of these backups makes doing actual FULL test restores prohibitive. My understanding is that since Veeam 6.5 there have been safeguards to protect/warn against corruption.
I need some better information as to how this happened, as I don't think i can trust my data to Veeam anymore when for all intents and purposes it says it is fine, but instead it is corrupted.

I've installed a trial of 8.0 onto another temporary system and am going to hook up the USB and try one more restore, but it isn't looking good.

[foggy]

Though it is not necessarily corrupt, there are too many players involved in the backup process to solely blame Veeam B&R (think network, storage...). Anyway, as mentioned, technical issues like that should be investigated via support, as they require thorough debug logs review.

[bhendin]

I've just completed another failed restore (same decompress error).

In this test, the Veeam server was hooked directly to the ESXi server via network cable.

I find it very unlikely that anything regarding the network is the issue, especially since the failure occurs at the same point each time.

While storage may be the culprit, the fact is that none of the storage devices are reporting errors of any kind (again, this are all simple local storage disks which show no corruption).
The job of the backup engine should be to validate that backup files written are valid - otherwise (as in this case), they are worthless.

I do enjoy the overall simplicity of this software, but it seems it can't be trusted to validate backups (by any process other than doing a complete restore to see).

I don't have time to deal with support, I need to recover data from older (non-veeam) backups and determine how much data loss we have, as well as look for a more reliable solution for the future.

[veremin]

Barely can this sort of issues be handled properly without seeing debug logs and actual infrastructure. Thus, opening a ticket sounds like the best idea (feel free to open it with due severity). Thanks.