-
- Influencer
- Posts: 12
- Liked: 3 times
- Joined: Dec 04, 2022 12:03 am
- Full Name: Miguel Almeida
- Contact:
Nutanix AHV Plugin v5.1.0.7 - Low backup speed
Hi,
Since version 5.0 I have noticed that my backups take much longer to complete, currently going from practically 5h to 9h. Before I had speeds of 3GB/s in backups and now I can't go beyond 200/300MBs.
The infrastructure and storage are the same, backups are made in Layer 2 without any speed limitations...
Has anyone noticed this problem?
iperf tests between CVMs, backup server and backup storage show correct values of approximately 10GB/s. I configured a Veeam Worker on the AHV Proxy but even that didn't help...
The Nutanix cluster is configured in LACP and the Bottleneck is always the Source:
Before with Veeam v11 and Plugin v4:
I have an open case but no response yet: Case # 07257526
Thank you
Since version 5.0 I have noticed that my backups take much longer to complete, currently going from practically 5h to 9h. Before I had speeds of 3GB/s in backups and now I can't go beyond 200/300MBs.
The infrastructure and storage are the same, backups are made in Layer 2 without any speed limitations...
Has anyone noticed this problem?
iperf tests between CVMs, backup server and backup storage show correct values of approximately 10GB/s. I configured a Veeam Worker on the AHV Proxy but even that didn't help...
The Nutanix cluster is configured in LACP and the Bottleneck is always the Source:
Before with Veeam v11 and Plugin v4:
I have an open case but no response yet: Case # 07257526
Thank you
-
- Veeam Software
- Posts: 460
- Liked: 140 times
- Joined: Mar 07, 2016 3:55 pm
- Full Name: Ronn Martin
- Contact:
Re: Nutanix AHV Plugin v5.1.0.7 - Low backup speed
@asm19 just to confirm, when you say "I configured a Veeam Worker on the AHV Proxy" do you mean you set host affinity for a worker instance to the same cluster node as the backup appliance? Assuming "no" that you're simply running the appliance and backup processing on the backup appliance itself however if "yes" then you definitely want to disable host affinity so that the worker runs on a node other than the one hosting the backup appliance. It appears that support is responding in a timely manner. Please give them a bit to review the logs as they were just uploaded yesterday.
-
- Enthusiast
- Posts: 50
- Liked: 7 times
- Joined: Jan 24, 2023 11:14 pm
- Contact:
Re: Nutanix AHV Plugin v5.1.0.7 - Low backup speed
You might be onto something. We're on VBR12.1 + 5 of the plugin. We upgraded from 4 so long ago now I couldn't tell you what our old backup rates were.
We have some modest backup/recovery requirements so I don't know if it's comparable to your environment, but I can also say after checking the latest backup logs for all our jobs that indeed, all of them show source as the bottleneck and none of our jobs make it to the GBps for rate. One gets close, touching the 900MBps range, but not past.
With the general quality I've seen with the AHV side of VBR I wouldn't be shocked to be told this is a bug...
We have some modest backup/recovery requirements so I don't know if it's comparable to your environment, but I can also say after checking the latest backup logs for all our jobs that indeed, all of them show source as the bottleneck and none of our jobs make it to the GBps for rate. One gets close, touching the 900MBps range, but not past.
With the general quality I've seen with the AHV side of VBR I wouldn't be shocked to be told this is a bug...
-
- Veeam Software
- Posts: 60
- Liked: 20 times
- Joined: Sep 18, 2014 10:10 am
- Full Name: Nikolai Kochkin
- Contact:
Re: Nutanix AHV Plugin v5.1.0.7 - Low backup speed
The calculation has been changed between the versions to be in line with other VBR platforms.
Processing speed is smth like "Read size"/"processing time", you can try to compare this number between your versions to be sure we are comparing the same metrics
Processing speed is smth like "Read size"/"processing time", you can try to compare this number between your versions to be sure we are comparing the same metrics
-
- Enthusiast
- Posts: 50
- Liked: 7 times
- Joined: Jan 24, 2023 11:14 pm
- Contact:
Re: Nutanix AHV Plugin v5.1.0.7 - Low backup speed
I still think there could be something to the OP's report. I am doing a couple backup/restore tests today as it relates to some other Veeam work I am doing and when doing restores specifically, I haven't seen the restore speed go past the high 300MBps and around 300ish IOPS. And those restores are to all-flash clusters I know for a fact that can go faster than that.
-
- Veeam Software
- Posts: 460
- Liked: 140 times
- Joined: Mar 07, 2016 3:55 pm
- Full Name: Ronn Martin
- Contact:
Re: Nutanix AHV Plugin v5.1.0.7 - Low backup speed
@asm19 as @HYF_JE mentioned they are seeing >300MB/second rates so it's best to continue to work with support to identify the root cause of the apparent bottleneck. Of course we're always evolving performance and do have a number of enhancements coming in future versions for both backup and restore operations for AHV.
-
- Service Provider
- Posts: 3
- Liked: 1 time
- Joined: Jun 16, 2023 11:16 am
- Contact:
Re: Nutanix AHV Plugin v5.1.0.7 - Low backup speed
Same in our environment, before upgrade we saw restore speeds ~1GB/s, now we are at max. 300MB/s. Will open a support case later.HYF_JE wrote: ↑May 14, 2024 9:23 pm I still think there could be something to the OP's report. I am doing a couple backup/restore tests today as it relates to some other Veeam work I am doing and when doing restores specifically, I haven't seen the restore speed go past the high 300MBps and around 300ish IOPS. And those restores are to all-flash clusters I know for a fact that can go faster than that.
-
- Enthusiast
- Posts: 50
- Liked: 7 times
- Joined: Jan 24, 2023 11:14 pm
- Contact:
Re: Nutanix AHV Plugin v5.1.0.7 - Low backup speed
Chiming in to add that I think our speeds were around that 1GBps range too. I find it weird that this appears consistent. I would have assumed that hardware capabilities would influence backup/restore speeds and you'd see some variability. Almost feels like a rate limit thing.
-
- Enthusiast
- Posts: 50
- Liked: 7 times
- Joined: Jan 24, 2023 11:14 pm
- Contact:
Re: Nutanix AHV Plugin v5.1.0.7 - Low backup speed
Opened case 07269284 as a companion. If support wants logs let me know but I don't know what they would be able to show with respect to this issue.
-
- Influencer
- Posts: 12
- Liked: 3 times
- Joined: Dec 04, 2022 12:03 am
- Full Name: Miguel Almeida
- Contact:
Re: Nutanix AHV Plugin v5.1.0.7 - Low backup speed
Hello guys,
Sorry for the delay, but I've been in contact with Nutanix and Veeam support since opening the tickets.
I have some hybrid clusters and others All Flash, and the source bottleneck problem (98/99%) happens in all of them.
Finally my Veeam ticket was escalated, and I was asked yesterday to make some changes to the Proxy and do some tests because in v5 a new algorithm for data movers was introduced.
I will change this setting and test the backups again to compare, I will give feedback later.
Note: From all the testing and troubleshooting I've done with Nutanix, there are no issues with clusters/AOS.
Thank you
Sorry for the delay, but I've been in contact with Nutanix and Veeam support since opening the tickets.
I have some hybrid clusters and others All Flash, and the source bottleneck problem (98/99%) happens in all of them.
Finally my Veeam ticket was escalated, and I was asked yesterday to make some changes to the Proxy and do some tests because in v5 a new algorithm for data movers was introduced.
I will change this setting and test the backups again to compare, I will give feedback later.
Note: From all the testing and troubleshooting I've done with Nutanix, there are no issues with clusters/AOS.
Thank you
Who is online
Users browsing this forum: No registered users and 1 guest