Comprehensive data protection for all workloads
Post Reply
stevenrodenburg1
Expert
Posts: 135
Liked: 20 times
Joined: May 31, 2011 9:11 am
Full Name: Steven Rodenburg
Location: Switzerland
Contact:

V6 - Read-speeds have become extremely slow of some VM's

Post by stevenrodenburg1 »

Hello,

Two nights ago, i started noticing that with two VM's (out of 19 being backed-up and 7 replicated), the read-speeds in their jobs, regardless if it's backup or replication, have become horrendously slow. They get read at 1MB/s if i'm lucky.

Those two VM's always ran fast and without a hitch. They still run without errors but it takes ages to finish a backup- or replica-job.

All the other VM's still run at their usual fast speeds. It's just those two who got slow. I cannot find the source of the problem. The storage-array is picking it's nose. CPU load is low, and again, everything else runs fast like they always have.

Example of one of the two VM's in question:
Processing finished at 02.02.2012 13:14:41
VM size: 170.0
Hard Disk 1 (40.0 GB) read at 1MB/s, CBT
Hard Disk 2 (30.0 GB) read at 1MB/s, CBT
Hard Disk 3 (100.0 GB) read at 1MB/s, CBT
Processing finished at 02.02.2012 13:14:41

That's 30 minutes for a machine, which is replicated every hour, normally takes under 3 minutes. Speeds are in the KB/s range for a while until it starts climbing up and settels at 1MB/s.
I'm not aware of any throttle mechanisms and again, all other VM's going through the same proxies are normal (fast).

I'm lost. Any clues where to look before i chime in with support ?
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: V6 - Read-speeds have become extremely slow of some VM's

Post by Vitaliy S. »

Steven, your job debug logs should definitely pinpoint the reason for that and our support team will assists you with the review.
lobo519
Veteran
Posts: 315
Liked: 38 times
Joined: Sep 29, 2010 3:37 pm
Contact:

Re: V6 - Read-speeds have become extremely slow of some VM's

Post by lobo519 »

I just started having the same problem out of the blue on my replication jobs. Did you ever find a solution?? I am waiting for support but a quick answer would be appreciated!
Gostev
Chief Product Officer
Posts: 31459
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: V6 - Read-speeds have become extremely slow of some VM's

Post by Gostev »

Well, he never came back to post the resolution or the support case number, so I cannot lookup what was the reason. Perfect example why we typically delete topics about technical issues if they do not include support case ID. People are quick to post about the issue, but almost never come back to update on the resolution. And these kind of dead topics are absolutely useless to the community...
lobo519
Veteran
Posts: 315
Liked: 38 times
Joined: Sep 29, 2010 3:37 pm
Contact:

Re: V6 - Read-speeds have become extremely slow of some VM's

Post by lobo519 »

here's to hoping he responds!

I as assuming your couldn't look it up by his name?... I long shot, i know..
stevenrodenburg1
Expert
Posts: 135
Liked: 20 times
Joined: May 31, 2011 9:11 am
Full Name: Steven Rodenburg
Location: Switzerland
Contact:

Re: V6 - Read-speeds have become extremely slow of some VM's

Post by stevenrodenburg1 »

lobo519 wrote:I just started having the same problem out of the blue on my replication jobs. Did you ever find a solution?? I am waiting for support but a quick answer would be appreciated!
Never found a solution but it "fixed itself" after 4 or 5 days. Never could find out what caused it.

The obvious candidates of the slow-down were storage and hypervisor(s) but they checked out fine. I am quiet knowledgable (if i may say so myself at the risk of, well you know...) in the realms of VMware, Microsoft OS and Storage. After a couple of days, at the point where i started thinking "maybe the problem does not lie in the infrastructure, maybe it's in Veeam", the problem magically went away. I was happy, sure, but felt unsatisfied somehow. I'd like to get to the bottom of things and never got the chance.
stevenrodenburg1
Expert
Posts: 135
Liked: 20 times
Joined: May 31, 2011 9:11 am
Full Name: Steven Rodenburg
Location: Switzerland
Contact:

Re: V6 - Read-speeds have become extremely slow of some VM's

Post by stevenrodenburg1 »

Gostev wrote:Well, he never came back to post the resolution or the support case number, so I cannot lookup what was the reason. Perfect example why we typically delete topics about technical issues if they do not include support case ID. People are quick to post about the issue, but almost never come back to update on the resolution. And these kind of dead topics are absolutely useless to the community...
I never filed a support case. No need to.

I was just asking if anybody could hint me in a direction to search, before opening a support call. No one replied for a while (which is ok, no problem) so i left it at that. Especially after the "magic fix".

One last thing: you should have deleted this thread long ago then. After all, even now, after my response, it is still utterly useless.
Gostev
Chief Product Officer
Posts: 31459
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: V6 - Read-speeds have become extremely slow of some VM's

Post by Gostev »

We should have, but I guess it slipped away somehow! :D
stevenrodenburg1
Expert
Posts: 135
Liked: 20 times
Joined: May 31, 2011 9:11 am
Full Name: Steven Rodenburg
Location: Switzerland
Contact:

Re: V6 - Read-speeds have become extremely slow of some VM's

Post by stevenrodenburg1 »

Sh*t happens ;-)

V6 with Hotfix 3 runs very stable by the way. Haven't seen any spooky behaviour since then. No need for the Ghost-Busters (Support).
lobo519
Veteran
Posts: 315
Liked: 38 times
Joined: Sep 29, 2010 3:37 pm
Contact:

Re: V6 - Read-speeds have become extremely slow of some VM's

Post by lobo519 » 1 person likes this post

Everything is running fine again - It was our Target SAN (Veeam was correctly reporting the target as the bottleneck the whole time)

We are replicating to an MD3000i.

I eventually found in the log of the MD3000i (information, not error) that the array decided to disable its cache on the controllers while it did a battery learn cycle greatly reducing the performance of the array. It took almost a day for it to finish and when it did, all performance returned and the jobs are running normally.


:shock:
Post Reply

Who is online

Users browsing this forum: Bing [Bot], Google [Bot] and 217 guests