-
- Expert
- Posts: 135
- Liked: 20 times
- Joined: May 31, 2011 9:11 am
- Full Name: Steven Rodenburg
- Location: Switzerland
- Contact:
V6 - Read-speeds have become extremely slow of some VM's
Hello,
Two nights ago, i started noticing that with two VM's (out of 19 being backed-up and 7 replicated), the read-speeds in their jobs, regardless if it's backup or replication, have become horrendously slow. They get read at 1MB/s if i'm lucky.
Those two VM's always ran fast and without a hitch. They still run without errors but it takes ages to finish a backup- or replica-job.
All the other VM's still run at their usual fast speeds. It's just those two who got slow. I cannot find the source of the problem. The storage-array is picking it's nose. CPU load is low, and again, everything else runs fast like they always have.
Example of one of the two VM's in question:
Processing finished at 02.02.2012 13:14:41
VM size: 170.0
Hard Disk 1 (40.0 GB) read at 1MB/s, CBT
Hard Disk 2 (30.0 GB) read at 1MB/s, CBT
Hard Disk 3 (100.0 GB) read at 1MB/s, CBT
Processing finished at 02.02.2012 13:14:41
That's 30 minutes for a machine, which is replicated every hour, normally takes under 3 minutes. Speeds are in the KB/s range for a while until it starts climbing up and settels at 1MB/s.
I'm not aware of any throttle mechanisms and again, all other VM's going through the same proxies are normal (fast).
I'm lost. Any clues where to look before i chime in with support ?
Two nights ago, i started noticing that with two VM's (out of 19 being backed-up and 7 replicated), the read-speeds in their jobs, regardless if it's backup or replication, have become horrendously slow. They get read at 1MB/s if i'm lucky.
Those two VM's always ran fast and without a hitch. They still run without errors but it takes ages to finish a backup- or replica-job.
All the other VM's still run at their usual fast speeds. It's just those two who got slow. I cannot find the source of the problem. The storage-array is picking it's nose. CPU load is low, and again, everything else runs fast like they always have.
Example of one of the two VM's in question:
Processing finished at 02.02.2012 13:14:41
VM size: 170.0
Hard Disk 1 (40.0 GB) read at 1MB/s, CBT
Hard Disk 2 (30.0 GB) read at 1MB/s, CBT
Hard Disk 3 (100.0 GB) read at 1MB/s, CBT
Processing finished at 02.02.2012 13:14:41
That's 30 minutes for a machine, which is replicated every hour, normally takes under 3 minutes. Speeds are in the KB/s range for a while until it starts climbing up and settels at 1MB/s.
I'm not aware of any throttle mechanisms and again, all other VM's going through the same proxies are normal (fast).
I'm lost. Any clues where to look before i chime in with support ?
-
- VP, Product Management
- Posts: 27375
- Liked: 2799 times
- Joined: Mar 30, 2009 9:13 am
- Full Name: Vitaliy Safarov
- Contact:
Re: V6 - Read-speeds have become extremely slow of some VM's
Steven, your job debug logs should definitely pinpoint the reason for that and our support team will assists you with the review.
-
- Veteran
- Posts: 315
- Liked: 38 times
- Joined: Sep 29, 2010 3:37 pm
- Contact:
Re: V6 - Read-speeds have become extremely slow of some VM's
I just started having the same problem out of the blue on my replication jobs. Did you ever find a solution?? I am waiting for support but a quick answer would be appreciated!
-
- Chief Product Officer
- Posts: 31806
- Liked: 7299 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: V6 - Read-speeds have become extremely slow of some VM's
Well, he never came back to post the resolution or the support case number, so I cannot lookup what was the reason. Perfect example why we typically delete topics about technical issues if they do not include support case ID. People are quick to post about the issue, but almost never come back to update on the resolution. And these kind of dead topics are absolutely useless to the community...
-
- Veteran
- Posts: 315
- Liked: 38 times
- Joined: Sep 29, 2010 3:37 pm
- Contact:
Re: V6 - Read-speeds have become extremely slow of some VM's
here's to hoping he responds!
I as assuming your couldn't look it up by his name?... I long shot, i know..
I as assuming your couldn't look it up by his name?... I long shot, i know..
-
- Expert
- Posts: 135
- Liked: 20 times
- Joined: May 31, 2011 9:11 am
- Full Name: Steven Rodenburg
- Location: Switzerland
- Contact:
Re: V6 - Read-speeds have become extremely slow of some VM's
Never found a solution but it "fixed itself" after 4 or 5 days. Never could find out what caused it.lobo519 wrote:I just started having the same problem out of the blue on my replication jobs. Did you ever find a solution?? I am waiting for support but a quick answer would be appreciated!
The obvious candidates of the slow-down were storage and hypervisor(s) but they checked out fine. I am quiet knowledgable (if i may say so myself at the risk of, well you know...) in the realms of VMware, Microsoft OS and Storage. After a couple of days, at the point where i started thinking "maybe the problem does not lie in the infrastructure, maybe it's in Veeam", the problem magically went away. I was happy, sure, but felt unsatisfied somehow. I'd like to get to the bottom of things and never got the chance.
-
- Expert
- Posts: 135
- Liked: 20 times
- Joined: May 31, 2011 9:11 am
- Full Name: Steven Rodenburg
- Location: Switzerland
- Contact:
Re: V6 - Read-speeds have become extremely slow of some VM's
I never filed a support case. No need to.Gostev wrote:Well, he never came back to post the resolution or the support case number, so I cannot lookup what was the reason. Perfect example why we typically delete topics about technical issues if they do not include support case ID. People are quick to post about the issue, but almost never come back to update on the resolution. And these kind of dead topics are absolutely useless to the community...
I was just asking if anybody could hint me in a direction to search, before opening a support call. No one replied for a while (which is ok, no problem) so i left it at that. Especially after the "magic fix".
One last thing: you should have deleted this thread long ago then. After all, even now, after my response, it is still utterly useless.
-
- Chief Product Officer
- Posts: 31806
- Liked: 7299 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: V6 - Read-speeds have become extremely slow of some VM's
We should have, but I guess it slipped away somehow!
-
- Expert
- Posts: 135
- Liked: 20 times
- Joined: May 31, 2011 9:11 am
- Full Name: Steven Rodenburg
- Location: Switzerland
- Contact:
Re: V6 - Read-speeds have become extremely slow of some VM's
Sh*t happens
V6 with Hotfix 3 runs very stable by the way. Haven't seen any spooky behaviour since then. No need for the Ghost-Busters (Support).
V6 with Hotfix 3 runs very stable by the way. Haven't seen any spooky behaviour since then. No need for the Ghost-Busters (Support).
-
- Veteran
- Posts: 315
- Liked: 38 times
- Joined: Sep 29, 2010 3:37 pm
- Contact:
Re: V6 - Read-speeds have become extremely slow of some VM's
Everything is running fine again - It was our Target SAN (Veeam was correctly reporting the target as the bottleneck the whole time)
We are replicating to an MD3000i.
I eventually found in the log of the MD3000i (information, not error) that the array decided to disable its cache on the controllers while it did a battery learn cycle greatly reducing the performance of the array. It took almost a day for it to finish and when it did, all performance returned and the jobs are running normally.
We are replicating to an MD3000i.
I eventually found in the log of the MD3000i (information, not error) that the array decided to disable its cache on the controllers while it did a battery learn cycle greatly reducing the performance of the array. It took almost a day for it to finish and when it did, all performance returned and the jobs are running normally.
Who is online
Users browsing this forum: Google [Bot], jagdishsingh, mbrzezinski and 117 guests