Synthetic full with transform to rollback chains corruption

[andnym]

Hi I have ran into a problem where synthetic full with transform to rollback chains and NO active full backup corrupts the backup. This was discovered when we tried to start up several Windows Server 2003 from backup repository as a virtual lab. Several server didn't boot. We get an error with a failure saying boot.ini is missing or ntoskrnl.exe is corrupt. Restoring such a server gives you a non working server.

I wonder if there is anyone else having this problem? I have had an open case [ID#5173314] with support regarding this problem still solution.

If you are running backups with this configuration I would recommend to verify your backups with an Instant Recovery and see if the virtual machine is actually working.
Would be interesting to know if my customer are alone in this situation.

[zoltank]

Yours is actually a great argument for use SureBackup to verify you have viable backups. In fact, I just recently discovered one of our servers was having a issue and would not be recoverable due to it failing in our SureBackup job. It's a very powerful tool, and I highly recommend taking advantage of it.

[Andreas Neufert]

Hi Anders,

try to backup the same affected server with a normal job (just the first full is ok) and test if the same problems came up.
If so, SureBackup exactly doing his job.

CU Andy

[Gostev]

Because backup file updates are performed in the transactional manner, actual transformation process cannot cause data corruption or data loss. Thanks to certain level of redundancy in the backup file storage schema and how file updates are orchestrated, the data cannot be lost, as we always know where the "good" data is no matter at which moment of transformation process the failure has occurred.

Most typical caused of backup corruption are:

1. RAID controller failure (arguably the most common cause). Faulty RAID controller starts to write random rubbish instead of real data. If the controllers goes completely FUBAR, this is of course very easy to spot. What's more dangerous is rare, intermittent, random corruptions. I have had a chance to experience the latter one first hand, thanks to now-well-known bug in the built-in controller of Intel P67 chipset with revision prior to B3 (controller's silicon quickly degrades, and at some point starts randomly affecting SATA ports functionality... initially, it was completely impossible for me to notice).

2. Network protocol implementation bugs. Majority of corruptions we see through support seem to be happening in the environments with backup targets being a share on low end NAS device. It was almost a plague at some point, when many new small storage vendors were entering this market 3 years ago. Now, the situation is actually so much better - due to fixed up firmware I guess.

3. Very questionable performance optimizations of (again) low end NAS devices. While optimizing performance to come on top of synthetic performance tests in various review, in some cases vendors simply queue or do not process some performance-sensitive commands (such as flush) at all, while reporting back that the command was in fact processed. This results in our engine thinking that the data is committed to disk and causes it to finalize the transaction when in fact the data was not yet committed - and then the power outage or something like that comes. Of course, those reviews never test that part

4. "Bit rot" (considered less common with modern storage, but impossible to measure and say for sure)

Based on that, I always recommend:
- Local, DAS or proper SAN storage to store backups (helps you avoid bullets 2 and 3, and get better performance than with network share)
- Use SureBackup (those with Standard Edition can use manual process explained in SureBackup)

[zoltank]

After talking with support about a different issue, the possibility of corruption within a reverse incremental (RI) backup job, or within a standard incremental job synthetic full, being carried into the .vbk file. In other words, if one of your RI jobs has corrupt data then it will be built into the .vbk file, as the .vbk depends on every restore point before it. Any corruption is then carried forward into each subsequent restore point of the chain. The longer the RI chain becomes, the greater the possibility of corruption at some point in the chain.

The same with a standard incremental with synthetic fulls, as that synthetic full is built using the previous restore points, and any corruption will be carried into the synthetic full.

Now, I'm not sure exactly how often you'll have data corruption, but I've found at least one instance mentioned in this forum (caused by a corrupt CBT). Support told me that's why they recommend doing periodic active full backups, since it creates a new .vbk from the actual VMs and doesn't rely on any previous restore points. However, if the active full's restore point expires (per your retention policy) before another active full has been performed, your backups become completely dependent on the previous restore points again.

Please correct me if I'm wrong, but to insure I have at least one viable backup, it seems like I need restore points going back beyond my last active full backup - I need to be able to go back to an active full backup (and not a synthetic full) at any time.

In my thinking correct?

[Gostev]

Now, I'm not sure exactly how often you'll have data corruption, but I've found at least one instance mentioned in this forum (caused by a corrupt CBT).

Actually, no. Corrupt CBT data will not cause backup corruption. It will be simply discarded, and incremental run will fallback to "snap&scan" method of determining what has changed since last backup.

Please correct me if I'm wrong, but to insure I have at least one viable backup, it seems like I need restore points going back beyond my last active full backup - I need to be able to go back to an active full backup (and not a synthetic full) at any time.

The newly created active full backup is self-sufficient by definition. Then, it depends on the backup mode, and whether it involves transforming the full backup file.

With regular incremental backup, existing full backup files are never modified once created, so you can be sure it will remain the same as when it was created. However, in case of reversed incremental backup, or incremental backup w/transform, because the following incremental runs update it, you do need to keep rollback created since active full to be able to go back to original active full state.

In any case, the bigger issue with your thinking is that you seem to be thinking that you can rely on a single backup chain for your restores. And that is so wrong. Backup is not a backup until you have 3 copies of it, but at a bare minimum, you absolutely must have 2 (and one of them offsite). Typically, most economical way of meeting this requirement means copying monthly active full to an external drive, and taking it home with you. This way, you will always have that "good" active full backup available, so you don't really have to worry about setting up job retention to go all the way back to that same state. Remember that this won't save you from most common reason of backup file corruption (storage failure) anyway. THIS is the primary threat that you really need to focus on addressing (not some mysterious CBT bugs etc).

[zoltank]

We currently perform reverse incremental backups. Every other day the entire reverse incremental chain is backed up to tape and sent offsite. Additionally, every weekend the .vbk files from the reverse incremental jobs are pushed to an offsite storage server. However, all of those copies are coming from the same chain. Considering we're currently backing up 2.77tb, the amount of storage we have available and our available backup window become issues.

What's the best solution here? Maybe running two reverse incremental backups of the same VMs? Or making sure we always have at least two Active Full backups saved offsite?

[dellock6]

Two runs of reverse incremental could put a high I/O load on both production storage and backup storage, I will stay away from backing up the same VM twice, if not for backup+replication. Instead, you can use some replication at the storage level or via rsync-style software to move the VBK in an secondary location.

[zoltank]

So what Gostev was talking about wasn't keeping multiple backup chains, but rather keeping multiple backups (both on-site and off-site) of the same backup chain. Correct?

[dellock6]

Right. To be super-correct, it's enough to keep multiple copies of the same backup files.

[zoltank]

Ok, we're on the same page then. I'm already doing that, with approximately 7 copies of the data offsite at any time, on both tape and hard drive.

However, I'm just afraid of the possibility that the backup data itself could contain corruption, and I'm exploring how to minimize that risk.

[dellock6]

Oh, I see. First of all, Veeam has some internal mechanisms to check consistency of what it is writing on disk. But the best way to solve your problem is to use SureBackup to test the VMs you are saving can be really restored and booted

[Xelsior]

Ok, I have a quick question related to this topic.

If I am doing reverse incrementals for a job and I also schedule an active full on a saturday. At the end of that day do I end up with 2 vbk files, the full from the RI and the active full or does the active full replace the RI full in the chain leaving me with 1 vbk file with a number of vbr files that are still valid.
I dont have the space for 2 vbk files so can't just schedule it to see what happens. and I want to make sure if I do end up with 1 vbk file that the vbr files are still valid backups and I havent just broken the chain.

[zoltank]

Ok, I have a quick question related to this topic.

If I am doing reverse incrementals for a job and I also schedule an active full on a saturday. At the end of that day do I end up with 2 vbk files, the full from the RI and the active full or does the active full replace the RI full in the chain leaving me with 1 vbk file with a number of vbr files that are still valid.
I dont have the space for 2 vbk files so can't just schedule it to see what happens. and I want to make sure if I do end up with 1 vbk file that the vbr files are still valid backups and I havent just broken the chain.

You will end up with two .vbk files. The old .vbk file will eventually be deleted as it hits the end of the "backup points" policy set for the job.

[foggy]

Gavin, if you have space just for a single full backup, consider using the new incremental forever backup method introduced in v8.