Failed to repair VM '<VMNAME>'

[B.F.]

For reason's I'm not sure yet, we have a backup to disk that failed:
Full backup file merge failed Error: Insufficient system resources exist to complete the requested service. Failed to write data to the file [<FILEPATH .vbk>]

Not sure what resources were insufficient since there's still boat loads of space available on the disk.

Now when I try to run or retry the backup:
Error: Failed to repair VM '<VMName>' in backup '5/24/2016 1:05:43 PM'. Unknown algorithm 'TransformForeverIncremental'

Any suggestions on how to repair this?

Thanks

[PTide]

Hi,

Please contact our support team so they can investigate the error and post your case ID here.

Thanks

[B.F.]

Ticket #01825333 opened.

[electricd7]

We are seeing this exact same behavior. Are you by chance backing up to a scale-out repository? We didn't start seeing these until we started using the new scale-out repositories. I have had a case open (#01813976) since 5/31 and still no resolution. At first they pointed to my repository as we were using 2012 storage spaces to span disks into a large volume to use as the repository. They had me change to just a single large disk (we are using VMware so we just created a large VMDK) and started a new chain at that new repository (a member of a new scale-out repository as the only member.) We just crossed the retention period where a merge needed to happen and have the same error again on the new storage!

[B.F.]

Our repository is quite basic and had been used for a while now. The error only happened once and ended up needing to start a new chain of backups as well as added some more system memory to the backup server. So far so good.

[ccatlett1984]

I just upgraded last week (from v8) and am seeing the same thing on a few jobs, but not all.

I have opened case # 01880151

[prabhash.jena]

Our repository is quite basic and had been used for a while now. The error only happened once and ended up needing to start a new chain of backups as well as added some more system memory to the backup server. So far so good.

Hi,

Were Veeam able to fix this or you ended up creating new job?

Regards
Prabhash

[veremin]

A new backup chain was started. Most likely, a new active full backup was created. Besides that, system resources available for a backup server were increased. Thanks.

[prabhash.jena]

Hi,

I applied some fixes such as removing backup from configuration, rescan of scale out repository, run an active full, exclude the VM in question from backup job, perform a quick backup of the VM in question reboot of repository server but nothing worked. I created new job and am keeping the data from old job for restoration. I have plenty of resources on the servers so I am sure that's not the issue.

This is not a pleasant experience, 1 VM backup file corruption leading to corruption of whole job. I believe there can be a database fix for this.

Regards
Prabhash

[veremin]

I believe there can be a database fix for this.

And have you tried to reach our support team to check the availability of the said fix? Thanks.

[dpatel]

Hi,

We are facing same problem on our one of the backup job. We are using Veeam 9.0.0.1715 with scale-out repository.

On same repository other backups are working fine without any error.

Thanks,
Daxesh

[foggy]

Please contact technical support for troubleshooting. Thanks.

[Steen]

I have same problem on several job.
Tha backup-chain is destroyed and forced to make new backup-job.
Sadly this seeams to be an widely spread problem and no fix

[Gostev]

The fix is to add some RAM to a repository server (or whatever server runs the target data mover for the corresponding backup repository).

[Steen]

Hi.
How much memory?
We have 32GB on B&R and 32 GBV on all ours repositories so i dont think that is the problem.

[Gostev]

Depends primarily on the number of jobs you are running concurrently.

[Steen]

1 job with 1 vm. Still fails

[Gostev]

Once the issue has been introduced, it will fail regardless. I am talking about the time when the issue was introduced - in other words, how many jobs are running concurrently normally in your environment? Earlier you mentioned you have "multiple jobs".

[Steen]

We have 1 B&R and 4 repositories
Maximum amount of of concurent jobs is 24 (one concurrent snapshot/CSV)

[Gostev]

So assuming totally even spread, that is 6 jobs per repository... but if spread is not even, then even more. Depending on the job size, you may indeed be running out of memory on backup repository, because you need 6-8 GB RAM per concurrent job according to our System Requirements.

[Steen]

Yes. Its extrem high requiremnts.
altough if i look right now the usage of memory 8incl OS) is on repositories is about
1.6 GB
3.6 GB
4.6 GB
4.4 Gb

running 8 jobs

( i thought it was 4 GB / job )

[tsightler]

What OS version is your repository?

[Steen]

2012 R2