AIX AGENT 3.0 RESTORE Slow, Times Out and Fails

[JCM]

Hello,
Case # 05736623

Our restore from VB&R to AIX fails after two hours. Connection and processing continue until it hangs at the 2 hour mark and then nothing.
The commands are at command line and not in the background.
The directory is .22GB
File count 30276

VB&R version 11a
AIX Agent 3.0
AIX OS 7100-05
IBM power 8
The restore hangs and does not continue.
The speed on the restore is 800Kb/s and the backup is 250M/s
loopback (lo0) MTU is at 16896

The error in the log once the process stops executing is:
There was no finalized server sessions during the last 60 minutes.

As seen in the logs, it repeats every hour.

I have changed OS tuneables but nothing seemed to have made a difference.
Made changes to:
tcp_keepidle, tcp_keepintvl, tcp_keepinit

In the logs, Reconnect options: [enabled: timeout 1800000, interval 5000], stay the same.

Network Admin does not see any issues.

Any suggestions of what to check would be appreciated.
Thanks
Jane

[HannesK]

Hello,
in the end it will be checking the logs on the agent and backup server side. As far as I see, support requested some more logs and hopefully they find something then.

TCP timeouts should be irrelevant for such an operations and I would not change too many things at the same time.

What I noticed is, that you still use AIX agent 3.0. I would update to 4.0 first and see whether that solves it.

Best regards,
Hannes

[JCM]

Good morning,
Thanks for responding.
Just to clarify, the TCP changes were done in accordance with a VEEAM support document I came across, they were returned to the original values when there were no improvements.

I am waiting to here from support before I do any upgrades but I assume that will be their suggestion also

Jane

[JCM]

Morning,
An update: We worked with VEEAM Support sending logs back and forth and the conclusion was that this is a known issue with AIX.
Large numbers of files on a restore process hangs using veeamconfig of the veeam agent.
We were not convinced of this at first because it was only one directory which had 37,000 files (Support mentioned their systems fail at a million or so records) but no other reason could be found.

That does not seem to be very many files for a restore but that is the final conclusion.

[HannesK]

Hello,
agree, I'm not convinced, that this should be a problem with only 37k files.

I asked again to "dig deeper".

Best regards,
Hannes

[JCM]

Hello,
An update:
After working with Support the cause of the slow restore remains as a result of 37,000+ small files in a directory.

They stated this is because of the use of NFS communication loop-back mounts with agent restores.

The final test that confirmed this for the Support team was the fact that when using the Appliance Helper in the VB&R we were able to restore (using rsync) in a few minutes as opposed to 3 hours with the AIX VeeamAgent restore and copy command.

Once a Linux server is available to me I will also be testing with the Helper Host in VB&R.
Hope this information helps.
thanks
Jane

[PTide]

This topic has been brought up on our internal meeting with the development team. Performance improvements are certainly planned for the future updates.

Thanks!