Veeam Community Edition - Cannot restore backups

[Sbyrd]

I recently setup the free Community edition as a test environment and it is backing up around 6 VMs.

The backup server is itself running on one of the ESX servers and All my backup jobs complete successfully.

Today I went to test a restore of one of my smallest VMs and each time it fails when restoring the disk with an error similar to this.

Code: Select all

7/12/2021 10:22:03 AM Error    Restore job failed Error: Failed to decompress LZ4 block: Bad crc (original crc: ace48749, current crc: 6e30e205).
                               Failed to download disk 'Device '\\.\PhysicalDrive2''.
                               Agent failed to process method {DataTransfer.SyncDisk}.
                               Exception from server: Shared memory connection was closed.
                               Failed to upload disk. Skipped arguments: [Ubn-GestioIP-flat.vmdk];

I then deleted the backups and successfully did a new full backup and when I did the restore I got the same error.

Previously I was using Unitrends Free and it would backup fine but would also fail to restore any backups while transferring the disk to the Host. However it's logs were less clear.
I thought maybe it was the application so I decided to setup Veeam instead to see if it would work.

Any help would be appreciated.

[veremin]

Typically this type of errors indicates problems with underlying storage (that backups reside on). But you can try to reach our support team to double check it. Thanks!

[Sbyrd]

So by underlying storage do you mean the VMware datastore or the virtual disk of the virtual server?

Backup server is a VM with an OS NTFS disk and a separate NTFS disk used as the backup location. The VM server then "resides" on an ISCSI datastore attached to the ESXI host. Datastore is a small Qnap NAS.

[Gostev]

By "underlying storage" he means storage where your backup files are stored. In your case, it returns a different data comparing to what was written to it back when a backup file was created. Such reliability issues are not uncommon for low end NAS devices: they are prone to data loss due to not using enterprise-grade RAID controllers with battery-backed write cache.

[Sbyrd]

Thanks. I'll try to investigate. Your explanation makes sense but I guess I did not expect Veeam to make a full backup and then the backup is corrupt as soon as it is done.

I guess I have to find out if the corruption is happening on write or read.

[Gostev]

Identifying a storage-based corruption would require re-reading the entire full backup back immediately after creation.

We actually do provide a health check functionality that does this, you can find it in the advanced job settings. But it's not scheduled to run daily by default, as it would double the backup window. While enterprise-grade storage rarely has these issues, they may only appear after the hardware gets really "tired" and worn out ICs start getting so called "sticky bits".

[Sbyrd]

So I just ran a new full job and immediately after running it I ran the command line validator and it failed when validating the vmdk with the same LZ4 error. So does that mean it is corrupting it on write or read?

Storage path: E:\Backups\Information Backup\Information BackupD2021-07-12T154348_945D.vbk
File 1: Ubn-GestioIP.vmx (2.9 KB)
File 2: Ubn-GestioIP.vmxf (150.0 B)
File 3: Ubn-GestioIP.nvram (8.5 KB)
File 4: Ubn-GestioIP.vmdk (614.0 B)
File 5: Ubn-GestioIP-flat.vmdk (16.0 GB)
File 6: FsAwareMeta:89f6c957-da07-4975-b609-cae991b2ac5f:2000 (0.0 B)
Validation progress
Ubn-GestioIP.vmx 100%
Ubn-GestioIP.vmxf 100%
Ubn-GestioIP.nvram 100%
Ubn-GestioIP.vmdk 100%
Ubn-GestioIP-flat.vmdk 23%
Statistics
VM count: Incomplete VM count: Failed VM count: Files count: Total size: Details:
1 0 1 5 16.0 GB Validation failed.
The following VMs are corrupted:
1. 'Ubn-GestioIP': File "Ubn-GestioIP-flat.vmdk" is corrupted. Failed to decompress LZ4 block: Incorrect decompression result or length (result: '-626428', expected length: '1048576'). --tr:Failed to calculate [md5] digest for the block. Source block size: [626468]. --tr:Client failed to process the command. Command: [checkFile]. --tr:event:3:

[Gostev]

This is actually impossible to say for sure, because it depends on the type of malfunction that is causing this. However, does it even matter? The storage is clearly unusable regardless.

By the way, you'd probably never know about the issue if your NAS was used to store images or videos, and perhaps even documents, because a few broken bytes are quite hard to notice in the media content, and there are no checksums for the corresponding applications to validate. Veeam is one of a few application that does not "trust" the storage and checksums everything that is being written, so that data could then be validated when it is read back by health check or restore processes.

[Sbyrd]

Got it. This NAS actually is the datastore for the veeam server itself and this Windows machine seems to run just fine, but I see what your saying.

[veremin]

So the same storage hosts both production data and its backup, right? Be aware that such configuration has certain drawbacks the main one being single point of failure (storage system). Thanks!

[Sbyrd]

No that is not the exact setup. I have two ESXI hosts and two datastores. Host 1 houses all the production servers with the servers living on datastore 1. Host 2 has the veeam server and is running off datastore 2.

This is a very small non-enterprise setup.

The idea is that if datastore 2 or host 2 fails it will not impact the production servers. If host 1 or datastore 1 fails then I use host 2 and/or datastore 2 to restore the servers while I get the "main" host and datastore back online.

[veremin]

But the backups are still located on VMFS, correct? So to get access to restore points you will always need an ESXi host and VMFS datastore configured, which might bring additional complexity to restoration procedure.

I understand the budget restrictions here, but if you have a decommissioned physical server, you can simply stuff it with bunch of directly attached disks - and this would make it a better backup target compared to VMFS datastore configured on top of malfunctioning NAS device.

Thanks!

[Sbyrd]

So I decided to test this theory as I have 2 Qnap NAS devices and two datastores and any backups written to either fail with the LZ4 error.

I decided then to create a new disk for the Veeam VM server but have this disk attached to the local storage datastore of the HP Proliant G6 server. Disks are 15k SAS 72GB drives in Raid 1.

If this succeeds on validation then I assume that the Qnaps are the issue. However if it fails with the same error then it is not my storage and must be something else. The QNAPs are connected via IP so could it be my network that is causing the corruption? This setup has been active and in use for over 3 years.

I was just struggling to see how I can have multiple Linux and Windows servers run just fine off the Qnaps and backups are written, but only the validation of the backups fails.

Anyway I did a backup of one server to the drive that lives on the host's local storage and it backed up and restored just fine. So either both my Qnap NAS work just fine for running servers but fail at making backup? So could the network itself be the source of my issues and not actually the drives of the NAS? Just wanting to know if that is possible.

[Sbyrd]

Well for now so I at least have working backups I have attached a large HDD drive via eSATA to one of the esx hosts and created a drive for the veeam server that is on that HDD.

Backups are written and validated successfully on that drive.

In the next 1-2 months I will be migrating off my existing vm environment to new hosts and an enterprise level SAN array.

[veremin]

Sounds like a solid move, but it still might be worth connecting HDD drives to separate physical machine and using it as a target repository instead of connecting it to ESXi host - should remove additional complexity during restore process. Thanks!

[guitarfish]

So by underlying storage do you mean the VMware datastore or the virtual disk of the virtual server?

Backup server is a VM with an OS NTFS disk and a separate NTFS disk used as the backup location. The VM server then "resides" on an ISCSI datastore attached to the ESXI host. Datastore is a small Qnap NAS.

I'm curious what model QNAP you are using? I have been using some TS-439 & 469 for years and I am able to restore without issue.