Synthetic full errors and retries.

[ashleyw]

Hi,

We are seeing a random issue with our jobs.
We have synthetic fulls set to start at 6pm on a Friday.
On these jobs the incremental backup goes through fine but the synthetic full rollup process gives an error of;
"Compile FIB failed Client error: Cannot allocate memory for an array".

The problem we have is that the run of the job returns an error status. Because the job itself is set for 3 retries, the retry sees that there are no VMs to be backed up (as the incremental went through fine) and then immediately finishes without attempting to retry the synthetic full creation again. This results in the retry giving a "success" message even though there is now no synthetic full.

We had increased the RAM on the backup proxies to 6GB ram each.

Can someone please confirm where the synthetic full is being generated from - I though that it would be the backup proxy that would be handling this process (and as such we had increased the RAM of each of our backup proxies to 6GB each). Because we are seeing memory related issues on the Console itself, it looks almost like it's the console performing the synthetic full creation - which means that our 4GB ram for the console is presumably not sufficient.

cheers
Ashley

[tsightler]

Synthetic full is performed by the repository. I would suggest opening a support case.

[ashleyw]

thanks. The problem in our case is that we are using CIFS presented backup target which does not use a repository proxy. So to me this looks like it is the main console that is performing the synthetic full process. This limits the sideways scalability of the product during the synthetic full generation process and it means our console has to have a relatively large amount of RAM (8GB ram) to handle the synthetic full process for multiple jobs.

Can we request a change so that the backup proxy in the case of CIFS is used to generate the synthetic full backup rather than the console?

[chrisdearden]

when you create CIFS share as a destination I think you have an option to specify which server would be used for that ?

[ashleyw]

thanks, the only thing we appear to be able to do by job is to specify the repository that is used, but there isn't any option to say where the synthetic full process occurs - how to we force the synthetic fulls to be processed by the proxy handling the job rather than the console in the case of CIFS?

One of jobs again failed on the synthetic full process over the weekend again with the same error (despite allocating 8GB ram to the console), and then the retry saw there were no VMs to be backed up so finished with a success about 2 minutes later when it should have made another attempt to retry the failed synthetic full - so this is definitely a bug around retries.

What makes things worse in this situation is that the synthetic jobs happen the same night (they need to), so as 4 jobs are allowed to run at once, there is increasing likelihood that the combined load will cause the console to run out of RAM.

[tsightler]

I have not tested this, however, I suspect that CIFS repositories are no different than any other repository in this respect, i.e. the server that "owns" the repository performs the synthetic full. If you add a CIFS repository to multiple proxies using the "Through the following proxy server" option then that proxy will "own" the repository. You could technically add the same CIFS share multiple times on different proxies with this option and then configure your jobs to use these "different" repositories even though they are actually all the same repository. You would need to change you jobs to select specific proxies and repositories, but I think this would spread the load of the synthetic fulls.

Once again, I haven't tested this, I'll try to spin it up in my lab later this week, but I think it should work for spreading the load of synthetic fulls.

[Gostev]

The setting Chris is talking about is located in your CIFS-based backup repository properties.

[ashleyw]

thanks guys, makes perfect sense now. We already have multiple entires for the same CIFS store so that we can push traffic over an active/active 10GbE backup target so thats great. I've just reconfigured the backup repository configuration for each of our CIFS shares to be "Write data to this share: Through the following proxying server" and then set alternate proxies for each repository.

I've re-configured each of our 2 backup proxies now to have 4xvCPUs and 8GB ram and I've dropped the console down to 6GB ram - I'll run some more tests during the day and report back.

Is there anything that can be done about the retries not taking into account "Compile FIB failed" errors in-case we should get the same sort of errors again?

There must be some correlation to the RAM required by the synthetic full generation and the size of the VIBs + previous VBK - do you know what this is? When we get those errors windows itself (previously the console) goes unresponsive and then causes further issues forcing us to reboot the VM.

thanks again.

[tsightler]

You should really open a support case regarding the "Compile FIB failed" and retry issues so that support can perform proper log investigation and get development involved if a fix is required (sounds likely). That's not really something that can be addressed via the forum.

[DSmith]

ashleyw, I have picked up your case. Please let me know if you have further questions.

tsightler,

Thanks for the clues.