Host-based backup of VMware vSphere VMs.
Post Reply
m.costantino
Novice
Posts: 8
Liked: 2 times
Joined: May 14, 2024 8:59 am
Full Name: Michele Costantino
Contact:

How to investigate on performace dropdown

Post by m.costantino »

Hi,
I'm facing a very strange issue about performance.
Everything seems to work fine during backups and replicas but after sometime I get a drastical slowdown on throughput.
From a read speed of 100MB/sec down to 12MB/sec.
NO NEW overlap of jobs during the job.
This is happening on all my jobs.
Any idea on where to start to investigate?
I've no idea.
david.domask
Veeam Software
Posts: 1925
Liked: 463 times
Joined: Jun 28, 2016 12:12 pm
Contact:

Re: How to investigate on performace dropdown

Post by david.domask »

Hi Michele, welcome to the forums.

Sounds like a technical issue, so please open a Support case as is noted when you create a topic. Please share the case number here once you've created one. Be sure to include logs for the Support Team to review (select the 1st radio option, then select the affected job)

Performance issues like this typically require some review of the logs and an overview of the infrastructure components involved, checking their server load, checking for differences between backups that went fast vs slow. Starting with the Bottleneck from the job statistics window may be an indicator on where to focus, but it's really best to start with a Support Case to review the logs.
David Domask | Product Management: Principal Analyst
m.costantino
Novice
Posts: 8
Liked: 2 times
Joined: May 14, 2024 8:59 am
Full Name: Michele Costantino
Contact:

Re: How to investigate on performace dropdown

Post by m.costantino »

Opened a ticked, support case number #07260033
m.costantino
Novice
Posts: 8
Liked: 2 times
Joined: May 14, 2024 8:59 am
Full Name: Michele Costantino
Contact:

Re: How to investigate on performace dropdown

Post by m.costantino »

No more support for v11.
We cannot upgrade to v12 because the new infrastructure is not ready (we're waiting for some components).
And we can't upgrade to v12 because of esxi 5.5 (we're trying to move to v8)

Sigh!
david.domask
Veeam Software
Posts: 1925
Liked: 463 times
Joined: Jun 28, 2016 12:12 pm
Contact:

Re: How to investigate on performace dropdown

Post by david.domask » 1 person likes this post

Hi Michele,

I see, an unfortunate situation, and apologies, I was not aware you were on Veeam Backup and Replication v11, which indeed is out of Support.

I can see that support gave you a quick suggestion to check on the case regarding potential leftover snapshot delta files, and this is a reasonable potential cause in my opinion -- are you seeing that the "read" amount for this server during backup is always quite large compared to the transfer? If so, this is usually indicative of a problem with CBT like Support is suggesting, and leftover snapshots can introduce complications with CBT.
David Domask | Product Management: Principal Analyst
m.costantino
Novice
Posts: 8
Liked: 2 times
Joined: May 14, 2024 8:59 am
Full Name: Michele Costantino
Contact:

Re: How to investigate on performace dropdown

Post by m.costantino »

Yes David, just saw the the quick suggestion....but i'm really really scared manually deleting files from the datastore.
If you mean (in the graph) the read speed / transfer speed...yes read speed 94MB/sec transfer speed is quite near to 0

Really scared about manually deleting files (from the snapshot manager I don't see any snapshot, previous snapshot seems consolidated, I see only the latest snapshot taken by Veeam).
david.domask
Veeam Software
Posts: 1925
Liked: 463 times
Joined: Jun 28, 2016 12:12 pm
Contact:

Re: How to investigate on performace dropdown

Post by david.domask » 1 person likes this post

I can appreciate the caution here regarding the snapshot files -- if you follow the KB and see the leftover delta files like Support advised you likely will, then you can try:

1. Show the same orphaned delta files to VMware support and ask for assistance cleaning them
2. Try Storage vMotioning the VM to another datastore
3. Restore the VM from a known good/recent backup and continue using the restored machine (I realize this is quite drastic, but just trying to put options out there if you're not able to clear the oprhaned delta-files on your own)
David Domask | Product Management: Principal Analyst
m.costantino
Novice
Posts: 8
Liked: 2 times
Joined: May 14, 2024 8:59 am
Full Name: Michele Costantino
Contact:

Re: How to investigate on performace dropdown

Post by m.costantino » 1 person likes this post

Thank you David, I really appreciate your help.
I'll try to contact VMware support.

I'll let you know, maybe this post can be helpful for someone else.

Ah...now the support is from Broadcom I suppose....I'll see...
tyler.jurgens
Veeam Legend
Posts: 380
Liked: 215 times
Joined: Apr 11, 2023 1:18 pm
Full Name: Tyler Jurgens
Contact:

Re: How to investigate on performace dropdown

Post by tyler.jurgens » 1 person likes this post

If you have a VM where snapshots got stuck and won't remove/consolidate, you can also clone the VM. Cloning will merge all snapshot deltas during its process.
You could also restore from backup since you have Veeam already. It will recover the VM without any snapshots.
Tyler Jurgens
Veeam Legend x3 | vExpert ** | VMCE | VCP 2020 | Tanzu Vanguard | VUG Canada Leader | VMUG Calgary Leader
Blog: https://explosive.cloud
Twitter: @Tyler_Jurgens BlueSky: @tylerjurgens.bsky.social
m.costantino
Novice
Posts: 8
Liked: 2 times
Joined: May 14, 2024 8:59 am
Full Name: Michele Costantino
Contact:

Re: How to investigate on performace dropdown

Post by m.costantino » 1 person likes this post

Deleted the orphaned snapshots....started a replica...now it works....it is running very very very very slow calculating a digest for an hard drive (1TB,15 hours) but everythings seems to run without errors.
The digest calculation for that drive is almost finished (94%), so I hope this replica will finish without any error.

Now I'll try to expain (my english is not very good...hi hi hi hi) why the problem started....
I've investigated and found when the problem started....it was THE SAME day we got a network problem that was causing continous drop of packets on the entire DR site... easy answer...Veeam couldn't delete the old snapshots anymore (not link to te datastore when loosing packets).

That should be the problem (I'm quite sure!!!!).
david.domask
Veeam Software
Posts: 1925
Liked: 463 times
Joined: Jun 28, 2016 12:12 pm
Contact:

Re: How to investigate on performace dropdown

Post by david.domask »

Hi Michele,

I'm glad to hear you were able to remedy the orphaned snapshot files and that jobs are progressing.

For your replica, indeed, the digest calculation can take a bit of time, but this should be a "one time" thing, and after that you should see normal processing times again.

With the backup, I expect it should get back to normal processing on its own after a run or two, but you may need to reset CBT manually for this VM. Alternatively, if you enable the option to reset CBT during Active Fulls, Veeam can do this automatically for you (but naturally it will require the space for an Active Full Backup. Remember on v12, with upgraded backups, you can perform active fulls for individual workloads instead of the whole job)
David Domask | Product Management: Principal Analyst
Post Reply

Who is online

Users browsing this forum: No registered users and 29 guests