Error when doing a sytem restore

[Glen.R]

I have been testing VEB as a replacement for WHS client backup and am receiving an error when trying to do a system restore on my Win7 laptop. VEB has created the backup but when trying to restore from it the restore fails with the following error "Failed to decompress LZ4 block: Incorrect decompression result or length (result: '-935681', expected length: '1048576')."

The laptop will boot to what looks like a complete Win7 load (everything looks to be there) but after the restore VEB won't run. Forcing it to start in services just has it crash as soon as it starts. Obviously it runs fine before the restore as it does run the backup job.

I have lodged a support ticket (01697871) but am posting here as well in case someone else has experienced this or it may help someone else. The support ticket has the logs, but I can upload here if anyone wants them.

*Edit* I forgot to mention that after the restore the OS tried to do a CHDSK on the C: drive each reboot, but is unable to run chkdsk.

[Dima P.]

Hi Glen.R,

Can you elaborate was the original Recovery Media used for bare metal recovery? Does it have all the storage drivers for the hardware you are restoring to?

[Glen.R]

Yes, I used the original Recovery Media. I burnt a CD from that installation and used it to boot into recovery. It should have had all drivers for the laptop as it was produced on the laptop.

[Dima P.]

Thanks. From what I see this error may refer to possible target disk corruption. Did you try to start a check disk while booted in the Veeam Recovery Media?

[Glen.R]

Well, I ran chkdsk on the drive from the recovery media (one error fixed), then reran the backup job and attempted a restore. The restore failed with the same error.

This time it booted without trying to run chkdsk and the VEB service started correctly. It looks like everything restored OK, but an error message is worrying.

[Glen.R]

Update: rebooted again and this time it automatically ran chkdsk and fixed LOTS of errors the seemed to come up fine.

I rebooted again and it booted without issue.

The strange thing is, if I restore from my old WHS backup image, I don't get any errors. If it was a suspect HDD, I'd expect it to corrupt also.

[Glen.R]

To test if the laptop was causing an error I tried to backup and restore my desktop. This is a win7 machine with an SSD as the system disk and a 500GBHDD for data. I set VEB to back up the system disk (SSD) and the backup hung trying to create the VSS snapshot. The whole system hung at that point. I rebooted and tried again with the same result.

I am not having any luck with VEB.

[Dima P.]

Hello Glen.R,

Can you share the recovery log with me via google drive or dropbox (it should be placed in your backup repository)?

[Glen.R]

Restore log is in dropbox. I sent the link to you via PM.

The issue with the system hanging while trying to create a VSS snapshot; I tried backing up to a local drive and the back up ran fine ( or until lack of disk space caused it to end), but when I tried to back up again to a network drive everything hung while it was trying to create the VSS snapshot. The PC does not seem to exhibit any network issues in normal use, and this is my main workhorse PC that is always moving files across my network. I'm completely confused by these issues...

[Dima P.]

Thanks - we are checking your logs.

hung while it was trying to create the VSS snapshot.

Do you have enough free space on the source disk since that might be caused by lack of free space while creating the VSS snapshot?

[Glen.R]

It's a 120GB SSD with 46.5GB free.

[Dima P.]

Understood. Do you have the exact wording or screenshot of this VSS error? Thanks.

[Glen.R]

I think I have found the problem. I rolled back to the final backup I made with WHS (before installing Veeam). I then updated all my mainboard, graphics and network drivers as well as flashing the BIOS with a later version, then reinstalled Veeam. The backup ran without issues. I then formatted the system reserved partion and c: drive and restored from the Veeam backup and all came up without issue. Very happy with that. Thanks for your help on this.

[Glen.R]

I spoke too soon. Having run my test backup/restore, I thought I would now do a full system backup. Configured Veeam to back up the entire computer, except for a specific folder I want excluded, and when I ran the backup the system hung a creating the VSS snapshot again.

No error message. The details panel just listed
Preparing for backup - 28 sec
Creating VSS snapshot - 12min 57 sec
Task Failed Unexpectedly

The task failing is when I reset the system as the PC had become completely unresponsive. No mouse movement, no keyboard response, completely hung.

[ewkilian]

Hi,
Sorry I’m french and my English is very bad.
I such to restore a full volume but I have the error “failed to decompress LZ4 Block…” during the restoration.
I have change the target HD and I have do a CHDISK on the source HD.
Can you help me please ?
Thanks you very much
ewkilian

Ticket Number : 01724392

[Dima P.]

Glen.R,
I am sorry – I completely missed your last post. According to the logs you backup storage is corrupted (it might have a bad blocks or other issues) and if affects the stored backup file thus you are getting this error. I am afraid there is nothing we can do about it since the issue is related to hardware failure.

Ried,
Kindly, let the support team check you logs first. Thanks.

[ewkilian]

Hello,

(sorry i'm french my english is bad)
I'm user from veeam endpoint bakcup Free. This is the 2nd time, of 3, that I have a error on the Metal Restoration (same error, that the object, that last time, with another computer, not the same material)
That is really a problem for me.
I receive this e-mail with object "Success"
All is OK.
the HD crash for this computer. I think no problem I have a Veeam Backup FULL Computer
But when I try the restore, I have a error
I'm very sad, I use the software on many computer, and now I wonder if I can trust this software
I hope that you can help me too restore this computer.
I have test with different backup (Full and Incremental)
Can you help me ? Is very important for me to restore that Full metal
Is there a verification software ?
Thank you.

Reply from Veeam Support after I send the log

Definitely, Veeam fails on reading backup file:

..
[21.10.2016 20:14:50] <19> Error Failed to restore volume
[21.10.2016 20:14:50] <19> Error Invoke failed. Command: 'DataTransfer.SyncDisk', .
..
21.10.2016 20:14:50] <19> Error Failed to decompress LZ4 block: Incorrect decompression result or length (result: '-1002082', expected length: '1048576'). (Veeam.Backup.Common.CCppComponentException)
..

LZ4 error means that one of the backup file's blocks is corrupted (physically) and CRC sum is wrong. By default Veeam stops a restore procedure, because this failed block can contain system files and skipping them will cause a restored OS fail to boot.

Unfortunately from a Recovery Media right now it is not possible to skip bad blocks from a backup file.

However, as a workaround, it might be possible to open a backup file using VBR. VBR has specialized agents, which can skip faulty backup block and read only consistent ones. It's worth trying.

Can you help me to find a solution ?

for
To developper team, can you developp a solution to verify the CRC after the backup.


Thank you very much

[Shestakov]

Hello ewkilian,
I understand your frustration, but as was said it`s not software, but hardware issue.
Support engineers are correct, if data blocks are corrupted server restore is not possible, however you may try file-level restore. Do you also use Veeam Backup and Replication?
Since you have that issue on that repository I would suggest you to try backup on another repository.
Thanks!

[Dima P.]

To clarify, backup file is corrupted because it is stored on corrupted device (probably, a disk with bad blocks). In such case, entire computer recovery won’t work but you can try to restore the specific volumes (try to exclude volumes one by one, can’t promise – it depends on the allocation of the corrupted blocks). If volume level restore is not working too, last resort is file level recovery.

[ewkilian]

The restore in file level is ok. But always not Bare Metal restore.
Have you a solution to check the HD ? to check the backup ?

No I Don't use Veeam Backup and Replication

Thank you

[Shestakov]

There is no automatic backup recoverability check yet, the feature is going to be added in future versions.
You may install free or trial version of Veeam Backup and Replication to perform file and volume-level restores.
Thanks!

[Vitaliy S.]

And to check the HD health state, I would suggest to use any 3rd party tool (it doesn't really matter which one you select).

[ewkilian]

OK thanks you, Then I wait on the new feature for the next version.

[Dima P.]

Regarding the disk checking utilities, as far as I am concerned windows check disk is not the best solution (it may skip bad blocks at some point). Support folks from time to time use the tool called Victoria, so worth checking.

[kwinsor]

Hi everyone,
I use Veeam Endpoint Free for my Windows 10 PC. I run the latest version. I needed to restore my system volume yesterday to correct a change I made. The PC could boot with no issue so I figured easiest fix was to restore from my 4 AM backup. My backups are stored on external hard drive. All backups were successful for the past 10 days of my retention policy. No errors reported in any of my backups so I expected a successful restore. I booted the PC using Veeam Recovery USB drive and proceeded to restore the system volume. It failed around 30% due to CRC mismatch and failed to decompress LZ4 block. I tried 4 different incremental backups and got same error. I even went back 10 days to the full and received same error. Each time the same point failed with same CRC values.

Upon boot up from the failed restores, my PC was so messed up I could only login with a temporary profile. Luckily all my data is on a separate disk. I had master file table corruption all over the place which I managed to fix. 4 hours later I think I fixed all my errors. I had to recreate the Veeam Endpoint database as the service would not start and i had to repair a bunch of programs.

My question to this forum is why did this happen? I ran Seatools and chkdsk against my external drive and it has no issues. As far as I can tell, the external disk is fine.I use Veeam at work for our VMware environments so I know the products well. I've now lost confidence in Veeam EndPoint backup. I have 10 days worth of backups that all look fine in the GUI in terms of successful backups but how am I to know if I can restore at crunch time? Why doesn't the backup fail if it has issues storing the backup on disk due to file integrity issues?

I have no logs to attach. I have no Veeam configuration left with the database rebuild except the backup files that I can't restore. Anyone have any thoughts on this?

Thank you in advance.

[Mike Resseler]

Kent,

Obviously I don't like reading this one and I can not give you an answer why it happened. From what I can read I can only guess that there is a corruption in the full backup, and therefore the incrementals aren't working either. Do you still have the backups of the last 10 days left? I still would like you to create a support call through the VEB UI so our engineers can investigate this

Thanks
Mike

[Dima P.]

Hi Kent,

It failed around 30% due to CRC mismatch and failed to decompress LZ4 block.

This error usually reflects the disk storage corruption. Check disk might not be the best way to identify the bad storage. Please open a support case and our support folks should be able to help you identifying the root cause.

[kwinsor]

Hi thanks.....yes I still have backup files. It appears a bunch of zip files named appliance_log_<date> from the time the restores failed written to the backup directory of my external drive. I can open a case and maybe provide these logs.

Regarding chkdsk, the drive is a Seagate 4 TB and I also ran Seatools against it with a short generic test and it found no issue.

[Dima P.]

Kent,

Thanks! Restore logs will help to find out the root cause. I might be wrong, but it looks like seatools are leveraging the checkdisk functionality. Frankly speaking it may help to resolve the software block corruption but may miss or ignore the hardware issues. Check the drive with Victoria tool I’ve shared above in this thread. I trust this solution's output when it comes to the hardware corruption issues

[kwinsor]

Thanks i will try the tool. I opened a case - Veeam Support - Case # 02026596