Comprehensive data protection for all workloads
Post Reply
empty
Lurker
Posts: 1
Liked: never
Joined: Jul 14, 2010 11:52 am
Full Name: Matthias Taeger
Contact:

Zlib decompression error on restore

Post by empty »

I tried to restore a successfull Veeam-Backup and got always the following error:

---------------------------
Veeam Backup and FastSCP
---------------------------
Failed to restore item "B-AV-Mail"
Restore failed
Client error: Zlib decompression error: [-3].
Cannot process [restore] command.

Server error: Zlib decompression error: [-3].
Unable to retrieve next block transmission command. Number of already processed blocks: [8578].
Unable to receive file.
---------------------------
OK
---------------------------
It doesn't matter where I try to restore (HDD or local storage on ESX) or if I choose the entire VM or individual VM-Files. The only thing that changes is the "Number of already processed blocks". The also successfull Veeam-Backup one week earlier I can restore without any problem. What went wrong and how can I trust the Veeam-Backup?

Veeam Backup & Replication x64 4.1.2, ESX 4.0U2 with local storage

Gostev
SVP, Product Management
Posts: 26679
Liked: 4268 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Zlib decompression error on restore

Post by Gostev »

Hello Matthias, for any technical issues please open a case with our support directly, as explained when you click New Topic on this forum. Our technical staff will require full log files in order to be able to troubleshoot most issues. Thank you!

trustblomme
Influencer
Posts: 22
Liked: 3 times
Joined: Nov 16, 2012 9:07 am

Re: Zlib decompression error on restore

Post by trustblomme »

Hello,

can you give an update on what was the cause of the error? Im testing VEEAM 6.5 and I'm having simular problems. All backup jobs are fine but I receive aan Zlib decompression error when I validate the full backup.

Thanks,

Steven

Vitaliy S.
Product Manager
Posts: 24243
Liked: 1859 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Zlib decompression error on restore

Post by Vitaliy S. »

Hi Steven, this is a pretty old thread (was created two years ago), so to get quicker assistance I would recommend to contact our support team directly. Thanks!

Gostev
SVP, Product Management
Posts: 26679
Liked: 4268 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Zlib decompression error on restore

Post by Gostev »

In short, this means that your backup file was corrupted after it was created. Some block's content does not match its checksum that we store along for data verification purposes.

munklarsen
Service Provider
Posts: 22
Liked: 8 times
Joined: Nov 15, 2012 11:02 pm
Full Name: Michael Munk Larsen
Contact:

[MERGED] Windows Server 2012 CIFS = zlib decompression error

Post by munklarsen »

Hi' everyone...

Just recently is did a restore of a VM - it failed with the zlib decompression error. The job was using a backup repository which was mapped using CIFS. I tried to restore some other VMs from other jobs and they all failed with the same zlib error.

I then decided to create a new backup job (again with CIFS backing) and then restore immediately after it completed. It also failed with the zlib error. I did with multiple times with different VMs from different hosts to different shares (all CIFS) on the same Windows Server 2012, all resulted in the zlib error.

I then mapped the the same windows server 2012 by installing the veeam agent on it. Created a new job, backed up a VM and restored it to another host. Worked perfectly.

My question is: Does anyone else have any problems with CIFS shares on Windows Server 2012 backup repository servers?

veremin
Product Manager
Posts: 17834
Liked: 1661 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Windows Server 2012 CIFS = zlib decompression error

Post by veremin »

Are you backing up to a remote CIFS repository or to a local one? If you’re backing up offsite over WAN connection, are you using additional proxy server or not?

Anyway, the best way of addressing technical issue is to open a ticket with our support team and let them investigate why your backup file is considered to be corrupted (zlib decompression error).

Hope this helps.
Thanks.

munklarsen
Service Provider
Posts: 22
Liked: 8 times
Joined: Nov 15, 2012 11:02 pm
Full Name: Michael Munk Larsen
Contact:

Re: Zlib decompression error on restore

Post by munklarsen »

Support ticket: 00197647

The backup repository server is on a different subnet but in the same datacenter and connected via 10gbit.

All trafic are going straight from the hotadd/nbd proxy to the CIFS share.

Gostev
SVP, Product Management
Posts: 26679
Liked: 4268 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: [MERGED] Windows Server 2012 CIFS = zlib decompression e

Post by Gostev »

munklarsen wrote:My question is: Does anyone else have any problems with CIFS shares on Windows Server 2012 backup repository servers?
Yes, we've seen some issues with data corruption during data transfers involving Windows Server 2012. This was one of he reasons why we've built inline network traffic validation in B&R version 6.5 (we've seen network traffic corruption in a few other scenario as well in the past years). Essentially, this functionality detects and automatically resends blocks that got corrupted during the network transfer.

Of course, this only works when two Veeam agents are talking to each other over the network (which is not the case when backing up to a CIFS share).
munklarsen wrote:I then mapped the the same windows server 2012 by installing the veeam agent on it. Created a new job, backed up a VM and restored it to another host. Worked perfectly.
This is actually the recommended way to set up your backup repository for many considerations, including (but not limited) to what I have explained above.

munklarsen
Service Provider
Posts: 22
Liked: 8 times
Joined: Nov 15, 2012 11:02 pm
Full Name: Michael Munk Larsen
Contact:

Re: Zlib decompression error on restore

Post by munklarsen »

Will this error be fixed in version 7?

The reason why we are using CIFS is because I have to use the backup repository server for multiple veeam installations and CIFS is the only way to keep my customers (who have access to the Veeam B&R Server) from being able to mount other customers backup folders and importing their backups.

Gostev
SVP, Product Management
Posts: 26679
Liked: 4268 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Zlib decompression error on restore

Post by Gostev »

munklarsen wrote:Will this error be fixed in version 7?
Actually, we believe the issue is in Windows Server 2012 itself, as we were able to confirm the issue without Veeam B&R present (by just doing regular file copies and then comparing source and target files). We were able to reproduce it a number of times on different hardware and in different copy scenarios, although it is one of those "come and go" issues (in other words, it would reproduce consistently during every test on one day, but then will go away for weeks in the same lab)... so, might have to deal with environment load etc.

When it reproduces, the pattern of data corruption is always the same no matter of hardware, data being transferred or test scenario, which really suggests this is a bug.

We have submitted the information about this issue to Microsoft as soon as we have discovered it.
munklarsen wrote:The reason why we are using CIFS is because I have to use the backup repository server for multiple veeam installations and CIFS is the only way to keep my customers (who have access to the Veeam B&R Server) from being able to mount other customers backup folders and importing their backups.
If you have some Linux expertise, you can also achieve that with Linux-based backup repository. Generally, I always recommend backing up to agent-enabled backup repository for performance and reliability reasons. I don't like shares :wink:

munklarsen
Service Provider
Posts: 22
Liked: 8 times
Joined: Nov 15, 2012 11:02 pm
Full Name: Michael Munk Larsen
Contact:

Re: Zlib decompression error on restore

Post by munklarsen »

Okay, very interesting. You might want to write this issue somewhere where it's easier to find...

Regarding CIFS. I don't like shares either but Windows Server 2012 was chosen due to its ability to create storage pools since Veeam can't automaticly distribute data across multiple disks. When we go into final production, we would probably have around 40TB of data and many many jobs.

But thanks for the information. I going to think of someway to redesign this.

ravelin
Veeam ProPartner
Posts: 48
Liked: 3 times
Joined: Apr 30, 2012 9:37 am
Full Name: Kenneth Westergaard
Contact:

Re: Zlib decompression error on restore

Post by ravelin »

I have the same problem on Windows 2012 when backing up.
I just had an job fail for several days because I didnt see it. This is a big job, so I just cant restart it with a full. Is there really no way to fix it?

It started with this on the last client in job:

Code: Select all

Error: Client error: An existing connection was forcibly closed by the remote host Unable to retrieve next block transmission command. Number of already processed blocks: [97985]. 
The retry jobs and daily jobs after this then comes with this:

Code: Select all

Failed to delete oib '[vmname=VMNAME:creation_time=25-04-2013 22:19:22:alg=Syntethic:id=45ba47ff-146d-419f-9690-4e3c77fe4eef:point_id=bf235d3c-ac5c-472e-9628-bb11ff0b42d7:storage_id=cad87bd2-7e09-4c2f-b8a8-2cf8d5a8108b]' from 'D:\Kunder\Customername\VMNAME2013-04-25T220033.vbk', redo 'D:\Kunder\Customername\VMNAME2013-04-24T220036.vrb' Undir failed Client error: Zlib decompression error: [-3].
My support #00202011

Gostev
SVP, Product Management
Posts: 26679
Liked: 4268 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Zlib decompression error on restore

Post by Gostev »

Usually there is really no way to fix it. However, I would not expect dropped connection to corrupt the backup, this is definitely unexpected. Happens all the time in our own deployment (remote backup across Atlantic), and never saw this causing any issues with the actual backup file in 3 years (I am receiving email reports for the job). Definitely worth close investigating because the reason could have been different, for example storage issues.

ravelin
Veeam ProPartner
Posts: 48
Liked: 3 times
Joined: Apr 30, 2012 9:37 am
Full Name: Kenneth Westergaard
Contact:

Re: Zlib decompression error on restore

Post by ravelin »

Its ón a Windows 2012. I read another post where you claim that to be The problem.
Isnt it possible to check The file before you delete The old one?

Gostev
SVP, Product Management
Posts: 26679
Liked: 4268 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Zlib decompression error on restore

Post by Gostev »

ravelin wrote:Isnt it possible to check The file before you delete The old one?
What do you mean?

ravelin
Veeam ProPartner
Posts: 48
Liked: 3 times
Joined: Apr 30, 2012 9:37 am
Full Name: Kenneth Westergaard
Contact:

Re: Zlib decompression error on restore

Post by ravelin »

I mean, my file is corrupt. I do reverse incremental.
Now I have to start over. I thought Veeam wrote a full copy of the last full , and then when written succesfully (with the daily changes) it deleted the old file from the day before.
Shouldnt veeam check the new file before deleting the original?

Gostev
SVP, Product Management
Posts: 26679
Liked: 4268 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Zlib decompression error on restore

Post by Gostev »

Reversed incremental backup does not really work the way you say it does (you may want to check out this forum or documentation), but the bottom line is that updates to backups files are transactional. Which is exactly why simple disconnect or server crash should never results in backup file corruption. Which is why, in turn, I suspect possible issues with the storage here...

ravelin
Veeam ProPartner
Posts: 48
Liked: 3 times
Joined: Apr 30, 2012 9:37 am
Full Name: Kenneth Westergaard
Contact:

Re: Zlib decompression error on restore

Post by ravelin »

Hmmm, Its a hp dl380g8 with internal discs ón raid 6.
You didnt answer regarding The Windows 2012 bug you believed earlier could be The problem?

Gostev
SVP, Product Management
Posts: 26679
Liked: 4268 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Zlib decompression error on restore

Post by Gostev »

No, since you are not backing up to a CIFS share... see my explanation on the previous page.

kkk
Lurker
Posts: 1
Liked: never
Joined: Aug 11, 2014 9:21 am
Contact:

[MERGED] VM restore failed

Post by kkk »

ticket id: 00616684

vsphere 5.1u2, vm&veeam server OS = win2k8 r2 sp1 x64.

have tried update to path 4. still failed.

error:
Restore job failed Error: Client error: Zlib decompression error: [-3].

Vitaliy S.
Product Manager
Posts: 24243
Liked: 1859 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Zlib decompression error on restore

Post by Vitaliy S. »

Please continue working with your support engineer on this issue and I would appreciate if you could update this topic with the resolution. Thanks!

MartijnV
Novice
Posts: 6
Liked: never
Joined: Jun 07, 2012 2:20 pm
Full Name: Martijn Verkuijlen
Contact:

[MERGED] zlib decompression error: [-3], backup corruption

Post by MartijnV »

We have problems restoring files from a VM.
Restore guest files (Windows) gives error 'No disks have been mounted. Either disks of this type aren't supported or some errors occured. See log file for details.'
Restore VM files fails with error zlib decompression error: [-3]

We use reversed incremental backup mode for this job, backup target is a Windows 2012 R2 storage server on the local LAN. Status of the job is 'Success' ...
This one VM seems to be the only one that is corrupt, other VMs in the backup job can be restored without problems

From what I've been reading in the knowledge base (http://www.veeam.com/kb1795) and on the forums the zlinb decompression error indicates data corruption in the backup file, typically caused by hardware malfunction during storage or transmission.

Is there any way to work around this issue so we can still restore files from this backup?
What is best practice to prevent data corruption?
Is it better to use normal Incrementals and make a new full backup every weekend?

Vitaliy S.
Product Manager
Posts: 24243
Liked: 1859 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Zlib decompression error on restore

Post by Vitaliy S. »

MartijnV wrote:What is best practice to prevent data corruption?
I would recommend using SureBackup jobs for that, please see this topic for more info > Zlib Decompression Error - Why doesn't Veeam warn me?

MartijnV
Novice
Posts: 6
Liked: never
Joined: Jun 07, 2012 2:20 pm
Full Name: Martijn Verkuijlen
Contact:

Re: Zlib decompression error on restore

Post by MartijnV »

Thanks Vitali, I will look into SureBackup.
Also I will look into changing the backupjobs from reversed incremental to forward incremental with weekly full.

Will a corruption in the backup file automatically carry over to the Backup Copy job?
This is what happened as I was testing with Backup Copy jobs but I assume that was because the backup file was already corrupt when I created the Backup Copy job.

foggy
Veeam Software
Posts: 19434
Liked: 1764 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Zlib decompression error on restore

Post by foggy »

MartijnV wrote:Will a corruption in the backup file automatically carry over to the Backup Copy job?
Yes, since backup copy job creates a bit-identical copy of the backup.

mobin.qasim
Enthusiast
Posts: 58
Liked: 1 time
Joined: Jun 19, 2013 7:23 pm
Full Name: Mobin Qasim
Contact:

[MERGED] Restore Entire VM (Including Registration) Failed!

Post by mobin.qasim »

Hi Guys,

I tried to restore entire VM including registration option and I was getting following error even after trying again and again.


"01/05/2015 12:18:41 Error Restoring disk '[TestDatastore] Test-VM/Test-VM-000002.vmdk' (60.0 GB)
01/05/2015 12:18:48 Error Restore job failed Error: Client error: Zlib decompression error: [-3].
Cannot process [restoreDiskContent] command. Target descriptor is [vddk://<vddkConnSpec><viConn name="Test-ESX" authdPort="902" vicPort="443" /><vmxPath vmRef="24" datacenterRef="ha-datacenter" datacenterInventoryPath="ha-datacenter" snapshotRef="24-snapshot-1" datastoreName="TestDatastore" path="Test-VM/Test-VM.vmx" /><vmdkPath datastoreName="TestDatastore" path="Test-VM/Test-VM-"


I found Veeam KB ID: 1795 which is something similar to error what I had. Just to add, "Enable Automatic Backup Integrity Checks" checkbox is enabled on the backup job, but Veeam didn't report any error. Also could someone please tell me how to know if Backup Integrity check is running when the checkbox is enabled. I can't see any Backup Integrity check message while each increment or during Full Synthetic backup.

Regards,

Mobin

foggy
Veeam Software
Posts: 19434
Liked: 1764 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Zlib decompression error on restore

Post by foggy »

Mobin, please review the thread above for details on the error you're receiving.

Also, the following threads should answer your questions regarding integrity checks:
Question on backup integrity
Enable Automatic Backup Integrity Checks ?

Thanks.

Rob@KB
Novice
Posts: 5
Liked: 1 time
Joined: May 12, 2016 10:51 am
Full Name: Rob Hulleman
Contact:

Re: Zlib decompression error on restore

Post by Rob@KB »

Same problems here with Veeam 9 (fully patched) and Windows Server 2012R2 (also tried with Windows 10 as Veeam "Server")

12-5-2016 12:38:37 Error Restore job failed Error: Failed to decompress LZ4 block: incorrect block header.
Failed to restore file from local backup. VFS link: [SYNC01.vmdk]. Target file: [MemFs://frontend::CDataTransferCommandSet::RestoreText_{7c521933-cb24-4c74-9de0-070886155f35}]. CHMOD mask: [0].
Agent failed to process method {DataTransfer.RestoreText}.
Agent failed to process method {DataTransfer.RestoreText}.

Backup jobs all run succesfully.
Inconsistency checks are failing the first time and are succesfull the second try.
Still no succesfull restore.

Case 01766410

Seen the duration of this error i am wondering if there ever will be a fix in the future???

foggy
Veeam Software
Posts: 19434
Liked: 1764 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Zlib decompression error on restore

Post by foggy »

Every case is different, besides, typically this error is caused by storage hardware issues. So please continue working with your support engineer.

Post Reply

Who is online

Users browsing this forum: No registered users and 33 guests