How to investigate on performace dropdown

[m.costantino]

Hi,
I'm facing a very strange issue about performance.
Everything seems to work fine during backups and replicas but after sometime I get a drastical slowdown on throughput.
From a read speed of 100MB/sec down to 12MB/sec.
NO NEW overlap of jobs during the job.
This is happening on all my jobs.
Any idea on where to start to investigate?
I've no idea.

[david.domask]

Hi Michele, welcome to the forums.

Sounds like a technical issue, so please open a Support case as is noted when you create a topic. Please share the case number here once you've created one. Be sure to include logs for the Support Team to review (select the 1st radio option, then select the affected job)

Performance issues like this typically require some review of the logs and an overview of the infrastructure components involved, checking their server load, checking for differences between backups that went fast vs slow. Starting with the Bottleneck from the job statistics window may be an indicator on where to focus, but it's really best to start with a Support Case to review the logs.

[m.costantino]

Opened a ticked, support case number #07260033

[m.costantino]

No more support for v11.
We cannot upgrade to v12 because the new infrastructure is not ready (we're waiting for some components).
And we can't upgrade to v12 because of esxi 5.5 (we're trying to move to v8)

Sigh!

[david.domask]

Hi Michele,

I see, an unfortunate situation, and apologies, I was not aware you were on Veeam Backup and Replication v11, which indeed is out of Support.

I can see that support gave you a quick suggestion to check on the case regarding potential leftover snapshot delta files, and this is a reasonable potential cause in my opinion -- are you seeing that the "read" amount for this server during backup is always quite large compared to the transfer? If so, this is usually indicative of a problem with CBT like Support is suggesting, and leftover snapshots can introduce complications with CBT.

[m.costantino]

Yes David, just saw the the quick suggestion....but i'm really really scared manually deleting files from the datastore.
If you mean (in the graph) the read speed / transfer speed...yes read speed 94MB/sec transfer speed is quite near to 0

Really scared about manually deleting files (from the snapshot manager I don't see any snapshot, previous snapshot seems consolidated, I see only the latest snapshot taken by Veeam).

[david.domask]

I can appreciate the caution here regarding the snapshot files -- if you follow the KB and see the leftover delta files like Support advised you likely will, then you can try:

1. Show the same orphaned delta files to VMware support and ask for assistance cleaning them
2. Try Storage vMotioning the VM to another datastore
3. Restore the VM from a known good/recent backup and continue using the restored machine (I realize this is quite drastic, but just trying to put options out there if you're not able to clear the oprhaned delta-files on your own)

[m.costantino]

Thank you David, I really appreciate your help.
I'll try to contact VMware support.

I'll let you know, maybe this post can be helpful for someone else.

Ah...now the support is from Broadcom I suppose....I'll see...

[tyler.jurgens]

If you have a VM where snapshots got stuck and won't remove/consolidate, you can also clone the VM. Cloning will merge all snapshot deltas during its process.
You could also restore from backup since you have Veeam already. It will recover the VM without any snapshots.

[m.costantino]

Deleted the orphaned snapshots....started a replica...now it works....it is running very very very very slow calculating a digest for an hard drive (1TB,15 hours) but everythings seems to run without errors.
The digest calculation for that drive is almost finished (94%), so I hope this replica will finish without any error.

Now I'll try to expain (my english is not very good...hi hi hi hi) why the problem started....
I've investigated and found when the problem started....it was THE SAME day we got a network problem that was causing continous drop of packets on the entire DR site... easy answer...Veeam couldn't delete the old snapshots anymore (not link to te datastore when loosing packets).

That should be the problem (I'm quite sure!!!!).

[david.domask]

Hi Michele,

I'm glad to hear you were able to remedy the orphaned snapshot files and that jobs are progressing.

For your replica, indeed, the digest calculation can take a bit of time, but this should be a "one time" thing, and after that you should see normal processing times again.

With the backup, I expect it should get back to normal processing on its own after a run or two, but you may need to reset CBT manually for this VM. Alternatively, if you enable the option to reset CBT during Active Fulls, Veeam can do this automatically for you (but naturally it will require the space for an Active Full Backup. Remember on v12, with upgraded backups, you can perform active fulls for individual workloads instead of the whole job)