V6 - Read-speeds have become extremely slow of some VM's

[stevenrodenburg1]

Hello,

Two nights ago, i started noticing that with two VM's (out of 19 being backed-up and 7 replicated), the read-speeds in their jobs, regardless if it's backup or replication, have become horrendously slow. They get read at 1MB/s if i'm lucky.

Those two VM's always ran fast and without a hitch. They still run without errors but it takes ages to finish a backup- or replica-job.

All the other VM's still run at their usual fast speeds. It's just those two who got slow. I cannot find the source of the problem. The storage-array is picking it's nose. CPU load is low, and again, everything else runs fast like they always have.

Example of one of the two VM's in question:
Processing finished at 02.02.2012 13:14:41
VM size: 170.0
Hard Disk 1 (40.0 GB) read at 1MB/s, CBT
Hard Disk 2 (30.0 GB) read at 1MB/s, CBT
Hard Disk 3 (100.0 GB) read at 1MB/s, CBT
Processing finished at 02.02.2012 13:14:41

That's 30 minutes for a machine, which is replicated every hour, normally takes under 3 minutes. Speeds are in the KB/s range for a while until it starts climbing up and settels at 1MB/s.
I'm not aware of any throttle mechanisms and again, all other VM's going through the same proxies are normal (fast).

I'm lost. Any clues where to look before i chime in with support ?

[Vitaliy S.]

Steven, your job debug logs should definitely pinpoint the reason for that and our support team will assists you with the review.

[lobo519]

I just started having the same problem out of the blue on my replication jobs. Did you ever find a solution?? I am waiting for support but a quick answer would be appreciated!

[Gostev]

Well, he never came back to post the resolution or the support case number, so I cannot lookup what was the reason. Perfect example why we typically delete topics about technical issues if they do not include support case ID. People are quick to post about the issue, but almost never come back to update on the resolution. And these kind of dead topics are absolutely useless to the community...

[lobo519]

here's to hoping he responds!

I as assuming your couldn't look it up by his name?... I long shot, i know..

[stevenrodenburg1]

I just started having the same problem out of the blue on my replication jobs. Did you ever find a solution?? I am waiting for support but a quick answer would be appreciated!

Never found a solution but it "fixed itself" after 4 or 5 days. Never could find out what caused it.

The obvious candidates of the slow-down were storage and hypervisor(s) but they checked out fine. I am quiet knowledgable (if i may say so myself at the risk of, well you know...) in the realms of VMware, Microsoft OS and Storage. After a couple of days, at the point where i started thinking "maybe the problem does not lie in the infrastructure, maybe it's in Veeam", the problem magically went away. I was happy, sure, but felt unsatisfied somehow. I'd like to get to the bottom of things and never got the chance.

[stevenrodenburg1]

Well, he never came back to post the resolution or the support case number, so I cannot lookup what was the reason. Perfect example why we typically delete topics about technical issues if they do not include support case ID. People are quick to post about the issue, but almost never come back to update on the resolution. And these kind of dead topics are absolutely useless to the community...

I never filed a support case. No need to.

I was just asking if anybody could hint me in a direction to search, before opening a support call. No one replied for a while (which is ok, no problem) so i left it at that. Especially after the "magic fix".

One last thing: you should have deleted this thread long ago then. After all, even now, after my response, it is still utterly useless.

[Gostev]

We should have, but I guess it slipped away somehow!

[stevenrodenburg1]

Sh*t happens

V6 with Hotfix 3 runs very stable by the way. Haven't seen any spooky behaviour since then. No need for the Ghost-Busters (Support).

[lobo519]

Everything is running fine again - It was our Target SAN (Veeam was correctly reporting the target as the bottleneck the whole time)

We are replicating to an MD3000i.

I eventually found in the log of the MD3000i (information, not error) that the array decided to disable its cache on the controllers while it did a battery learn cycle greatly reducing the performance of the array. It took almost a day for it to finish and when it did, all performance returned and the jobs are running normally.