SAP HANA Copy Job between two StoreOnce very slow and binding resources

[mecki]

Hi,
I'm using the new SAP HANA Copy Job for the SAP HANA Plugin Backups. The HANA DB and Logs Backups are stored on a StoreOnce 3640 Catalyst Repository connected via FC SAN (32Gbs) to a Backup Proxy/gateway/repository (HW) server. The 2nd StoreOnce 3640 is connected the same way to a 2nd (HW) server. There is a 20Gbs LAN Infrastructure. There are 11 HANA Servers doing their backup in parallel. The Backup Proxies are configured to 16 tasks each, the repository/gateway is set to unlimited
When now the Veeam HANA Backup Copy job is running it claims up to 10 tasks/sessions for each HANA Backup done. Therefor all other backups (SAP HANA plugin and VMware) are waiting for backup resources.. The SAP HANA plugin backups will fail due to time out of HANA Backup. If the HANA Copy Job is disabled/stopped all is running fine.
The SAP HANA Copy Job(s) are very very slow to copy. IMHO its about the small log backups to copy and the storeonce catalyst deduplication.
Is there any parameter to limit the session/tasks on the SAP HANA Backup Copy Job(s)? Maybe a hidden registry value.
How many sessions/task will a storeonce 3640 be limited by VBR?
There is already a case 04393070 open.

Thanking for any help,

Martin

[mecki]

I'm running now only one HANA Backup copy Job with one HANA Server Backup configured. There are 10 Sessions copying 10 "vab" in parallel, but it took 25min for about 92MB data!!! It's impossible to copy 3804 Objects => 380x25min = over 6,5 days to copy...

[PetrM]

Hi Martin,

The best option is to continue working with our support team, I'll keep an eye on it to be sure that the investigation moves into the right direction. Please keep in mind that you always have an option to request an escalation of the support case.

You may try to add the following registry value and to decrease the number of threads leveraged by backup copy, for example set the value equal to 5:

Code: Select all

BackupCopyApplicationMaxThreadCount
HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication
Type: DWORD

The workaround will not accelerate backup copy but most likely will fix the timeout issue with the primary job as less resources will be required for backup copy. Anyway, it's better to ask our support to confirm that the workaround is applicable in your case.

One more idea is to use another repository for example Windows as a primary storage instead of StoreOnce but to use StoreOnce as a target for backup copy.

Thanks!

[mecki]

Hi Petr,
thank you for the quick answer, I'm just trying the parameter set to 2, it's working for the HANA Copy Job but also all SAP HANA Plugin job are now waiting for backup infrastructure resources availability. The HANA Backup Copy copied now two files, but is now also waiting for resources...for the next two files. So know the HANA Backup Jobs are running. Do you know, were are the used/available resources logged for the storeonce (RTS log?).

THX,

[PetrM]

Hi Martin,

Thanks for reply, let's try one more value: "BackupCopyApplicationMaxTaskCount". Please set both values (this new value and the first one BackupCopyApplicationMaxThreadCount) equal to 8.
If it does not help - please revert changes back to default: BackupCopyApplicationMaxThreadCount - 10, BackupCopyApplicationMaxTaskCount - 20.

Do you know, were are the used/available resources logged for the storeonce (RTS log?).

I don't think that you will find this information in RTS log but please let our support engineers to analyze logs. The case should be escalated to a higher level soon.

Thanks!

[mecki]

Hi,

thanks for sharing the parameters. BTW what does threads and tasks means in this case. The thread count here is the number of VAB files processed in parallel? But what does the tasks number influence?
We had fixed now the bad performance, it was a "StoreOnceLogLevel" parameter set to 1 in the registry of one of the gateway servers. We only set that parameter on backup server/gateway server (combo) to 3 (detail), never touched that on the 2nd gateway server (HANA copy target here). But as soon as we found out that there is heavy detailed logging of anything regarding Veeam/StoreOnce and set this parameter back to normal, the performance issues were gone.

Is there any detailed overview about the VBR registry parameters so far? Maybe not public but for partners?

Thanking in advance,

[PetrM]

Hi Martin,

The thread count here is the number of VAB files processed in parallel?

Yes, that's right.

Tasks number affects how many repository task slots are used by backup copy job, the value should normally exceed a number of threads so that a thread can lock a free slot immediately after file copy operation is done. This approach is efficient when when we deal with a huge amount of small files because the duration of slot retrieving might be comparable with the duration of copy of a single small file. However, I don't think that you should dig this logic deeper because the registry values above must be applied by support team only and only in case of real necessity.

As far as I know there is no detailed overview with all registry parameters, in most cases registry values must be used by support engineers.

Thanks!