Multi-threaded backup file validation for SureBackup

[cgsm]

Is there any setting to make backup file validation multi-threaded during a SureBackup? And by this I mean, can I run multiple validations at the same time?

I have a SureBackup job that runs daily. The SureBackup job "turns on" multiple VMs at the same time (per the job's settings), as expected, but the backup file validation occurs one VM at a time, sequentially. The backup file validation takes quite a long time.

However, if I run a backup file validation as part of a backup job for the same VMs, the validation only takes about a quarter of the time. I imagine this is because multiple VM's backup files are being validated at the same time, in parallel, although I am unsure.

In sum, my question is, how can I make SureBackup validations faster? Or is SureBackup simply checking more "stuff" then the validation performed with the backup job?

[HannesK]

Hello,
just to rule something out: could you maybe test 1 VM in SureBackup and 1 VM in a backup job and compare the times for health check?

To answer the question: there is no configuration setting to "make it faster"

Best regards,
Hannes

[cgsm]

Hannes,

I created a 1 VM backup job and SureBackup job. This VM is ~100GB in size. The backups files are stored in a locally attached SSD repo. When the backup job runs, the health check takes 0:56. When SureBackup runs, the validation takes 3:53. Quite a difference!

Let me know your thoughts.

[HannesK]

Hello,
my thought is that it is a bug that. Looks like asynchronous read is not used in SureBackup. We added asynchronous read support for health checks in 11a. The speed improvements are discussed here

The difference in speed you mention fit pretty good

Just do be sure: are you using 11a / latest update? QA confirmed that async read is used in SureBackup. So support should investigate why the difference exists. Please post the case number for reference.

Best regards,
Hannes

[cgsm]

Hannes,
Yes, I am on 11a.

I just opened a support case, #05287611.

Thanks

[cgsm]

First advice was to install 11a hotfix. Did this. No improvement in time to run validation between backup job and SureBackup job.

[HannesK]

ok, then support needs to research more in detail.

[cgsm]

As of now support has not found anything although they do recognize the issue and it has been elevated.

Maybe we are going at this the wrong way though. I submit this question:

Is there a difference between a Backup job "health check" (set via edit job > Storage > Advanced > Maintenance > Perform backup files health check) and a SureBackup job backup file "integrity scan" (set via edit job > Settings > Validate entire virtual disk content).

If there is a difference, what is is? Per my reading of the VBR docs, it seems like the same process occurs with either task: a CRC scan of the backup files. If this is true, it brings us back to the original topic of this thread: why does this take so much longer in a SureBackup job versus a Backup job.

[HannesK]

yes, health check and integrity scan are the same. I will talk to my colleagues whether we can change wording in future versions.

why does this take so much longer in a SureBackup job versus a Backup job.

yes, that's the question. I will check again with support what is taking so long to figure it out.

[cgsm]

Support has found that when per-VM backup files is enabled, a Backup job health check will process VMs in parallel. However, when a SureBackup job runs the verification is done sequentially. Obviously this causes the process to take significantly longer. Quite odd considering the "Process 'x' many VMs in parallel" option for a SureBackup job.

I, with my support representative, have filed a feature request for this so that SureBackup job verifications occur sequentially.

[HannesK]

thanks, I will check again. because I tested with one VM and had the same result. I will come back on this.

[HannesK]

sorry for the overall long delay... the issue is tracked as bug number 385854 and we will look into improvements after V12. It's more complex than my initial guess with "async read" vs "sync read".

[JPMS]

Did this get sorted in v12 or is "Perform backup files health check" in a backup job still much quicker than "Validate entire virtual disk contents" in a Surebackup job?

[HannesK]

it's still the same in V12, as we look into it only "after V12" (mentioned above).

[JPMS]

Sorry, you are quite correct. I saw the "V12" and didn't read the full sentence properly!