Zlib decompression error on restore

[empty]

I tried to restore a successfull Veeam-Backup and got always the following error:

---------------------------
Veeam Backup and FastSCP
---------------------------
Failed to restore item "B-AV-Mail"
Restore failed
Client error: Zlib decompression error: [-3].
Cannot process [restore] command.

Server error: Zlib decompression error: [-3].
Unable to retrieve next block transmission command. Number of already processed blocks: [8578].
Unable to receive file.
---------------------------
OK
---------------------------
It doesn't matter where I try to restore (HDD or local storage on ESX) or if I choose the entire VM or individual VM-Files. The only thing that changes is the "Number of already processed blocks". The also successfull Veeam-Backup one week earlier I can restore without any problem. What went wrong and how can I trust the Veeam-Backup?

Veeam Backup & Replication x64 4.1.2, ESX 4.0U2 with local storage

[Gostev]

Hello Matthias, for any technical issues please open a case with our support directly, as explained when you click New Topic on this forum. Our technical staff will require full log files in order to be able to troubleshoot most issues. Thank you!

[trustblomme]

Hello,

can you give an update on what was the cause of the error? Im testing VEEAM 6.5 and I'm having simular problems. All backup jobs are fine but I receive aan Zlib decompression error when I validate the full backup.

Thanks,

Steven

[Vitaliy S.]

Hi Steven, this is a pretty old thread (was created two years ago), so to get quicker assistance I would recommend to contact our support team directly. Thanks!

[Gostev]

In short, this means that your backup file was corrupted after it was created. Some block's content does not match its checksum that we store along for data verification purposes.

[munklarsen]

Hi' everyone...

Just recently is did a restore of a VM - it failed with the zlib decompression error. The job was using a backup repository which was mapped using CIFS. I tried to restore some other VMs from other jobs and they all failed with the same zlib error.

I then decided to create a new backup job (again with CIFS backing) and then restore immediately after it completed. It also failed with the zlib error. I did with multiple times with different VMs from different hosts to different shares (all CIFS) on the same Windows Server 2012, all resulted in the zlib error.

I then mapped the the same windows server 2012 by installing the veeam agent on it. Created a new job, backed up a VM and restored it to another host. Worked perfectly.

My question is: Does anyone else have any problems with CIFS shares on Windows Server 2012 backup repository servers?

[veremin]

Are you backing up to a remote CIFS repository or to a local one? If you’re backing up offsite over WAN connection, are you using additional proxy server or not?

Anyway, the best way of addressing technical issue is to open a ticket with our support team and let them investigate why your backup file is considered to be corrupted (zlib decompression error).

Hope this helps.
Thanks.

[munklarsen]

Support ticket: 00197647

The backup repository server is on a different subnet but in the same datacenter and connected via 10gbit.

All trafic are going straight from the hotadd/nbd proxy to the CIFS share.

[Gostev]

My question is: Does anyone else have any problems with CIFS shares on Windows Server 2012 backup repository servers?

Yes, we've seen some issues with data corruption during data transfers involving Windows Server 2012. This was one of he reasons why we've built inline network traffic validation in B&R version 6.5 (we've seen network traffic corruption in a few other scenario as well in the past years). Essentially, this functionality detects and automatically resends blocks that got corrupted during the network transfer.

Of course, this only works when two Veeam agents are talking to each other over the network (which is not the case when backing up to a CIFS share).

I then mapped the the same windows server 2012 by installing the veeam agent on it. Created a new job, backed up a VM and restored it to another host. Worked perfectly.

This is actually the recommended way to set up your backup repository for many considerations, including (but not limited) to what I have explained above.

[munklarsen]

Will this error be fixed in version 7?

The reason why we are using CIFS is because I have to use the backup repository server for multiple veeam installations and CIFS is the only way to keep my customers (who have access to the Veeam B&R Server) from being able to mount other customers backup folders and importing their backups.

[Gostev]

Will this error be fixed in version 7?

Actually, we believe the issue is in Windows Server 2012 itself, as we were able to confirm the issue without Veeam B&R present (by just doing regular file copies and then comparing source and target files). We were able to reproduce it a number of times on different hardware and in different copy scenarios, although it is one of those "come and go" issues (in other words, it would reproduce consistently during every test on one day, but then will go away for weeks in the same lab)... so, might have to deal with environment load etc.

When it reproduces, the pattern of data corruption is always the same no matter of hardware, data being transferred or test scenario, which really suggests this is a bug.

We have submitted the information about this issue to Microsoft as soon as we have discovered it.

The reason why we are using CIFS is because I have to use the backup repository server for multiple veeam installations and CIFS is the only way to keep my customers (who have access to the Veeam B&R Server) from being able to mount other customers backup folders and importing their backups.

If you have some Linux expertise, you can also achieve that with Linux-based backup repository. Generally, I always recommend backing up to agent-enabled backup repository for performance and reliability reasons. I don't like shares

[munklarsen]

Okay, very interesting. You might want to write this issue somewhere where it's easier to find...

Regarding CIFS. I don't like shares either but Windows Server 2012 was chosen due to its ability to create storage pools since Veeam can't automaticly distribute data across multiple disks. When we go into final production, we would probably have around 40TB of data and many many jobs.

But thanks for the information. I going to think of someway to redesign this.

[ravelin]

I have the same problem on Windows 2012 when backing up.
I just had an job fail for several days because I didnt see it. This is a big job, so I just cant restart it with a full. Is there really no way to fix it?

It started with this on the last client in job:

Code: Select all

Error: Client error: An existing connection was forcibly closed by the remote host Unable to retrieve next block transmission command. Number of already processed blocks: [97985]. 

The retry jobs and daily jobs after this then comes with this:

Code: Select all

Failed to delete oib '[vmname=VMNAME:creation_time=25-04-2013 22:19:22:alg=Syntethic:id=45ba47ff-146d-419f-9690-4e3c77fe4eef:point_id=bf235d3c-ac5c-472e-9628-bb11ff0b42d7:storage_id=cad87bd2-7e09-4c2f-b8a8-2cf8d5a8108b]' from 'D:\Kunder\Customername\VMNAME2013-04-25T220033.vbk', redo 'D:\Kunder\Customername\VMNAME2013-04-24T220036.vrb' Undir failed Client error: Zlib decompression error: [-3].

My support #00202011

[Gostev]

Usually there is really no way to fix it. However, I would not expect dropped connection to corrupt the backup, this is definitely unexpected. Happens all the time in our own deployment (remote backup across Atlantic), and never saw this causing any issues with the actual backup file in 3 years (I am receiving email reports for the job). Definitely worth close investigating because the reason could have been different, for example storage issues.

[ravelin]

Its ón a Windows 2012. I read another post where you claim that to be The problem.
Isnt it possible to check The file before you delete The old one?

[Gostev]

Isnt it possible to check The file before you delete The old one?

What do you mean?

[ravelin]

I mean, my file is corrupt. I do reverse incremental.
Now I have to start over. I thought Veeam wrote a full copy of the last full , and then when written succesfully (with the daily changes) it deleted the old file from the day before.
Shouldnt veeam check the new file before deleting the original?

[Gostev]

Reversed incremental backup does not really work the way you say it does (you may want to check out this forum or documentation), but the bottom line is that updates to backups files are transactional. Which is exactly why simple disconnect or server crash should never results in backup file corruption. Which is why, in turn, I suspect possible issues with the storage here...

[ravelin]

Hmmm, Its a hp dl380g8 with internal discs ón raid 6.
You didnt answer regarding The Windows 2012 bug you believed earlier could be The problem?

[Gostev]

No, since you are not backing up to a CIFS share... see my explanation on the previous page.

[kkk]

ticket id: 00616684

vsphere 5.1u2, vm&veeam server OS = win2k8 r2 sp1 x64.

have tried update to path 4. still failed.

error:
Restore job failed Error: Client error: Zlib decompression error: [-3].

[Vitaliy S.]

Please continue working with your support engineer on this issue and I would appreciate if you could update this topic with the resolution. Thanks!

[MartijnV]

We have problems restoring files from a VM.
Restore guest files (Windows) gives error 'No disks have been mounted. Either disks of this type aren't supported or some errors occured. See log file for details.'
Restore VM files fails with error zlib decompression error: [-3]

We use reversed incremental backup mode for this job, backup target is a Windows 2012 R2 storage server on the local LAN. Status of the job is 'Success' ...
This one VM seems to be the only one that is corrupt, other VMs in the backup job can be restored without problems

From what I've been reading in the knowledge base (http://www.veeam.com/kb1795) and on the forums the zlinb decompression error indicates data corruption in the backup file, typically caused by hardware malfunction during storage or transmission.

Is there any way to work around this issue so we can still restore files from this backup?
What is best practice to prevent data corruption?
Is it better to use normal Incrementals and make a new full backup every weekend?

[Vitaliy S.]

What is best practice to prevent data corruption?

I would recommend using SureBackup jobs for that, please see this topic for more info > Zlib Decompression Error - Why doesn't Veeam warn me?

[MartijnV]

Thanks Vitali, I will look into SureBackup.
Also I will look into changing the backupjobs from reversed incremental to forward incremental with weekly full.

Will a corruption in the backup file automatically carry over to the Backup Copy job?
This is what happened as I was testing with Backup Copy jobs but I assume that was because the backup file was already corrupt when I created the Backup Copy job.

[foggy]

Will a corruption in the backup file automatically carry over to the Backup Copy job?

Yes, since backup copy job creates a bit-identical copy of the backup.

[mobin.qasim]

Hi Guys,

I tried to restore entire VM including registration option and I was getting following error even after trying again and again.


"01/05/2015 12:18:41 Error Restoring disk '[TestDatastore] Test-VM/Test-VM-000002.vmdk' (60.0 GB)
01/05/2015 12:18:48 Error Restore job failed Error: Client error: Zlib decompression error: [-3].
Cannot process [restoreDiskContent] command. Target descriptor is [vddk://<vddkConnSpec><viConn name="Test-ESX" authdPort="902" vicPort="443" /><vmxPath vmRef="24" datacenterRef="ha-datacenter" datacenterInventoryPath="ha-datacenter" snapshotRef="24-snapshot-1" datastoreName="TestDatastore" path="Test-VM/Test-VM.vmx" /><vmdkPath datastoreName="TestDatastore" path="Test-VM/Test-VM-"

I found Veeam KB ID: 1795 which is something similar to error what I had. Just to add, "Enable Automatic Backup Integrity Checks" checkbox is enabled on the backup job, but Veeam didn't report any error. Also could someone please tell me how to know if Backup Integrity check is running when the checkbox is enabled. I can't see any Backup Integrity check message while each increment or during Full Synthetic backup.

Regards,

Mobin

[foggy]

Mobin, please review the thread above for details on the error you're receiving.

Also, the following threads should answer your questions regarding integrity checks:
Question on backup integrity
Enable Automatic Backup Integrity Checks ?

Thanks.

[Rob@KB]

Same problems here with Veeam 9 (fully patched) and Windows Server 2012R2 (also tried with Windows 10 as Veeam "Server")

12-5-2016 12:38:37 Error Restore job failed Error: Failed to decompress LZ4 block: incorrect block header.
Failed to restore file from local backup. VFS link: [SYNC01.vmdk]. Target file: [MemFs://frontend::CDataTransferCommandSet::RestoreText_{7c521933-cb24-4c74-9de0-070886155f35}]. CHMOD mask: [0].
Agent failed to process method {DataTransfer.RestoreText}.
Agent failed to process method {DataTransfer.RestoreText}.

Backup jobs all run succesfully.
Inconsistency checks are failing the first time and are succesfull the second try.
Still no succesfull restore.

Case 01766410

Seen the duration of this error i am wondering if there ever will be a fix in the future???

[foggy]

Every case is different, besides, typically this error is caused by storage hardware issues. So please continue working with your support engineer.