Nutanix AHV Plugin v5.1.0.7 - Low backup speed

[asm19]

Hi,

Since version 5.0 I have noticed that my backups take much longer to complete, currently going from practically 5h to 9h. Before I had speeds of 3GB/s in backups and now I can't go beyond 200/300MBs.
The infrastructure and storage are the same, backups are made in Layer 2 without any speed limitations...
Has anyone noticed this problem?

iperf tests between CVMs, backup server and backup storage show correct values of approximately 10GB/s. I configured a Veeam Worker on the AHV Proxy but even that didn't help...

The Nutanix cluster is configured in LACP and the Bottleneck is always the Source:





Before with Veeam v11 and Plugin v4:





I have an open case but no response yet: Case # 07257526

Thank you

[ronnmartin61]

@asm19 just to confirm, when you say "I configured a Veeam Worker on the AHV Proxy" do you mean you set host affinity for a worker instance to the same cluster node as the backup appliance? Assuming "no" that you're simply running the appliance and backup processing on the backup appliance itself however if "yes" then you definitely want to disable host affinity so that the worker runs on a node other than the one hosting the backup appliance. It appears that support is responding in a timely manner. Please give them a bit to review the logs as they were just uploaded yesterday.

[HYF_JE]

You might be onto something. We're on VBR12.1 + 5 of the plugin. We upgraded from 4 so long ago now I couldn't tell you what our old backup rates were.

We have some modest backup/recovery requirements so I don't know if it's comparable to your environment, but I can also say after checking the latest backup logs for all our jobs that indeed, all of them show source as the bottleneck and none of our jobs make it to the GBps for rate. One gets close, touching the 900MBps range, but not past.

With the general quality I've seen with the AHV side of VBR I wouldn't be shocked to be told this is a bug...

[Kochkin]

The calculation has been changed between the versions to be in line with other VBR platforms.

Processing speed is smth like "Read size"/"processing time", you can try to compare this number between your versions to be sure we are comparing the same metrics

[HYF_JE]

I still think there could be something to the OP's report. I am doing a couple backup/restore tests today as it relates to some other Veeam work I am doing and when doing restores specifically, I haven't seen the restore speed go past the high 300MBps and around 300ish IOPS. And those restores are to all-flash clusters I know for a fact that can go faster than that.

[ronnmartin61]

@asm19 as @HYF_JE mentioned they are seeing >300MB/second rates so it's best to continue to work with support to identify the root cause of the apparent bottleneck. Of course we're always evolving performance and do have a number of enhancements coming in future versions for both backup and restore operations for AHV.

[JulienBubrecht]

I still think there could be something to the OP's report. I am doing a couple backup/restore tests today as it relates to some other Veeam work I am doing and when doing restores specifically, I haven't seen the restore speed go past the high 300MBps and around 300ish IOPS. And those restores are to all-flash clusters I know for a fact that can go faster than that.

Same in our environment, before upgrade we saw restore speeds ~1GB/s, now we are at max. 300MB/s. Will open a support case later.

[HYF_JE]

Chiming in to add that I think our speeds were around that 1GBps range too. I find it weird that this appears consistent. I would have assumed that hardware capabilities would influence backup/restore speeds and you'd see some variability. Almost feels like a rate limit thing.

[HYF_JE]

Opened case 07269284 as a companion. If support wants logs let me know but I don't know what they would be able to show with respect to this issue.