Standalone backup agents for Linux, Mac, AIX & Solaris workloads on-premises or in the public cloud
Post Reply
JCM
Novice
Posts: 9
Liked: never
Joined: Aug 30, 2022 6:02 pm
Full Name: Jane Caldwell-Myers
Contact:

AIX AGENT 3.0 RESTORE Slow, Times Out and Fails

Post by JCM »

Hello,
Case # 05736623

Our restore from VB&R to AIX fails after two hours. Connection and processing continue until it hangs at the 2 hour mark and then nothing.
The commands are at command line and not in the background.
The directory is .22GB
File count 30276

VB&R version 11a
AIX Agent 3.0
AIX OS 7100-05
IBM power 8
The restore hangs and does not continue.
The speed on the restore is 800Kb/s and the backup is 250M/s
loopback (lo0) MTU is at 16896

The error in the log once the process stops executing is:
There was no finalized server sessions during the last 60 minutes.

As seen in the logs, it repeats every hour.

I have changed OS tuneables but nothing seemed to have made a difference.
Made changes to:
tcp_keepidle, tcp_keepintvl, tcp_keepinit

In the logs, Reconnect options: [enabled: timeout 1800000, interval 5000], stay the same.

Network Admin does not see any issues.

Any suggestions of what to check would be appreciated.
Thanks
Jane
HannesK
Product Manager
Posts: 14824
Liked: 3075 times
Joined: Sep 01, 2014 11:46 am
Full Name: Hannes Kasparick
Location: Austria
Contact:

Re: AIX AGENT 3.0 RESTORE Slow, Times Out and Fails

Post by HannesK »

Hello,
in the end it will be checking the logs on the agent and backup server side. As far as I see, support requested some more logs and hopefully they find something then.

TCP timeouts should be irrelevant for such an operations and I would not change too many things at the same time.

What I noticed is, that you still use AIX agent 3.0. I would update to 4.0 first and see whether that solves it.

Best regards,
Hannes
JCM
Novice
Posts: 9
Liked: never
Joined: Aug 30, 2022 6:02 pm
Full Name: Jane Caldwell-Myers
Contact:

Re: AIX AGENT 3.0 RESTORE Slow, Times Out and Fails

Post by JCM »

Good morning,
Thanks for responding.
Just to clarify, the TCP changes were done in accordance with a VEEAM support document I came across, they were returned to the original values when there were no improvements.

I am waiting to here from support before I do any upgrades but I assume that will be their suggestion also

Jane
JCM
Novice
Posts: 9
Liked: never
Joined: Aug 30, 2022 6:02 pm
Full Name: Jane Caldwell-Myers
Contact:

Re: AIX AGENT 3.0 RESTORE Slow, Times Out and Fails

Post by JCM »

Morning,
An update: We worked with VEEAM Support sending logs back and forth and the conclusion was that this is a known issue with AIX.
Large numbers of files on a restore process hangs using veeamconfig of the veeam agent.
We were not convinced of this at first because it was only one directory which had 37,000 files (Support mentioned their systems fail at a million or so records) but no other reason could be found.

That does not seem to be very many files for a restore but that is the final conclusion.
HannesK
Product Manager
Posts: 14824
Liked: 3075 times
Joined: Sep 01, 2014 11:46 am
Full Name: Hannes Kasparick
Location: Austria
Contact:

Re: AIX AGENT 3.0 RESTORE Slow, Times Out and Fails

Post by HannesK »

Hello,
agree, I'm not convinced, that this should be a problem with only 37k files.

I asked again to "dig deeper".

Best regards,
Hannes
JCM
Novice
Posts: 9
Liked: never
Joined: Aug 30, 2022 6:02 pm
Full Name: Jane Caldwell-Myers
Contact:

Re: AIX AGENT 3.0 RESTORE Slow, Times Out and Fails

Post by JCM »

Hello,
An update:
After working with Support the cause of the slow restore remains as a result of 37,000+ small files in a directory.

They stated this is because of the use of NFS communication loop-back mounts with agent restores.

The final test that confirmed this for the Support team was the fact that when using the Appliance Helper in the VB&R we were able to restore (using rsync) in a few minutes as opposed to 3 hours with the AIX VeeamAgent restore and copy command.

Once a Linux server is available to me I will also be testing with the Helper Host in VB&R.
Hope this information helps.
thanks
Jane
PTide
Product Manager
Posts: 6550
Liked: 765 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: AIX AGENT 3.0 RESTORE Slow, Times Out and Fails

Post by PTide »

This topic has been brought up on our internal meeting with the development team. Performance improvements are certainly planned for the future updates.

Thanks!
Post Reply

Who is online

Users browsing this forum: No registered users and 5 guests