-
- Enthusiast
- Posts: 36
- Liked: 9 times
- Joined: May 28, 2009 7:52 pm
- Full Name: Steve Philp
- Contact:
Re: Stumped by backup speed change after upgrading to vSphere
Gostev - Thank you for your work in testing things and confirming the speed problem we're seeing, it's highly appreciated.
As noted in your post, VCB doesn't suffer from the slowdown. We've switched our backups to use that mode and see speeds comparable to the 3.5 Service Console method we used previously.
At this point, I think we're happy again!
As noted in your post, VCB doesn't suffer from the slowdown. We've switched our backups to use that mode and see speeds comparable to the 3.5 Service Console method we used previously.
At this point, I think we're happy again!
-
- Chief Product Officer
- Posts: 31814
- Liked: 7302 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: Network backup slow on vSphere ESX 4.0
OK, thanks for the update Steve... I assume this issue will not be fixed by VMware until vSphere Update 1 (which is scheduled around Q4 time frame as far as I know). I've also updated this thread's name (search optimization).
-
- Chief Product Officer
- Posts: 31814
- Liked: 7302 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: Network backup slow on vSphere ESX 4.0
Steve, could you please clarify if you are now using VCB in SAN mode, or in NBD mode? I am just trying to understand if VCB NBD mode still suffers from the same issue (since in NBD mode, all data still goes through ESX I/O stack and thus may be subject to ESX4 service console throttling issue).
-
- Enthusiast
- Posts: 35
- Liked: never
- Joined: Jan 01, 2006 1:01 am
- Contact:
Re: Network backup slow on vSphere ESX 4.0
I have the same issue with network backup. my speed dropped 5-6 times and I do not have enough time to finish backups.
I tried to switch to VCB mode but got an error:
Initializing source
The object has already been deleted or has not been completely created
I installed latest VCB from esx4 CD
I tried to mount full VM using vcbMounter in ncb mode (simple vcbMounter -m nbd -u vmware-backup -t fullvm) it worked but was very slow as well ~21MB/s
Anything I can do?
I tried to switch to VCB mode but got an error:
Initializing source
The object has already been deleted or has not been completely created
I installed latest VCB from esx4 CD
I tried to mount full VM using vcbMounter in ncb mode (simple vcbMounter -m nbd -u vmware-backup -t fullvm) it worked but was very slow as well ~21MB/s
Anything I can do?
-
- Enthusiast
- Posts: 36
- Liked: 9 times
- Joined: May 28, 2009 7:52 pm
- Full Name: Steve Philp
- Contact:
Re: Network backup slow on vSphere ESX 4.0
Gostev - We're now using VCB in SAN mode.
-
- Chief Product Officer
- Posts: 31814
- Liked: 7302 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: Network backup slow on vSphere ESX 4.0
Steve, thanks for clarification. As I can see from what Alex is saying, NBD mode suffers from the same problem.
Alex, we have opened a support case with VMware on this issue to better track it. Unfortunately, at this time there is no workaround available except using VCB SAN mode, which requires shared storage.
Alex, we have opened a support case with VMware on this issue to better track it. Unfortunately, at this time there is no workaround available except using VCB SAN mode, which requires shared storage.
-
- Enthusiast
- Posts: 35
- Liked: never
- Joined: Jan 01, 2006 1:01 am
- Contact:
Re: Network backup slow on vSphere ESX 4.0
Any news from VMware?
-
- Chief Product Officer
- Posts: 31814
- Liked: 7302 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: Network backup slow on vSphere ESX 4.0
I have performed a test with requested logging enabled, and sent the logs today (this morning). I've also invited VMware support engineer to do a live webex for faster troubleshooting (actually, 2 times already). Waiting for response (have not heard back yet since Friday night).
I will post here as soon as I get any news.
I will post here as soon as I get any news.
-
- Chief Product Officer
- Posts: 31814
- Liked: 7302 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: Network backup slow on vSphere ESX 4.0
So just to provide some sort of update: the support case is keep going up the chain, so far we've been told we are not the only one reporting the degraded performance, and that it is currently being investigated by VMware engineering department.
-
- Influencer
- Posts: 14
- Liked: never
- Joined: Jul 13, 2009 5:58 pm
- Full Name: joemagjr
- Contact:
Re: Network backup slow on vSphere ESX 4.0
Curious to know, has their been any new developments on this? Will changing from using an agent (automatic) to agentless engine on the host under data transfer engine help with the speed of the backups as an attempt to band-aide the problem for now?
-
- Chief Product Officer
- Posts: 31814
- Liked: 7302 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: Network backup slow on vSphere ESX 4.0
Unfortunately, changing between agent and agentless will not help. Both technologies end up in the service console (either in Veeam agent, or in VMware service). And it is service console that is being "throttled" in ESX4, so anything running in service console is affected. This includes all 3rd party service console agents, as well as native VMware services (those serving VIC datastore browser, or VCB in NBD mode).
We are currently working together with VMware in testing this, and we will publish plans for a resolution as soon as they are available.
We are currently working together with VMware in testing this, and we will publish plans for a resolution as soon as they are available.
-
- Novice
- Posts: 8
- Liked: never
- Joined: Apr 22, 2009 7:23 pm
- Full Name: PSmith
- Location: Grand Rapids, Michigan
Re: Network backup slow on vSphere ESX 4.0
Got an email today that VMWare was able to diagnose the issue and is currently working on a fix for the problem. They will be contacting me to test it once they have a fix ready to test out, and I will post the results to those of you following this post.
In the meantime here is the Knowledge Base article they have created for it;
http://kb.vmware.com/selfservice/micros ... Id=1012159
In the meantime here is the Knowledge Base article they have created for it;
http://kb.vmware.com/selfservice/micros ... Id=1012159
-
- Chief Product Officer
- Posts: 31814
- Liked: 7302 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: Network backup slow on vSphere ESX 4.0
Finally! In fact, they diagnosed the issue long ago, it just took them a good month to create support KB article for some reason.
I am glad to hear they told you they are working on the fix. Initially, they were saying this issue cannot be fixed because it is inadvertant effect from another major vSphere change that they just cannot rollback.
We did test one potential workaround they provided couple of weeks ago (certain flag to use when opening file from service console). It does help with the speed, but (without going into much details) it did not work very well with hot backups of live VMs. But service console is just one side of the story, and this workaround obviously cannot help with either VCB/vStorageAPI NBD/NBDSSL modes, nor NFC API performance. So I hope they will fix this in the ESX code instead.
I am glad to hear they told you they are working on the fix. Initially, they were saying this issue cannot be fixed because it is inadvertant effect from another major vSphere change that they just cannot rollback.
We did test one potential workaround they provided couple of weeks ago (certain flag to use when opening file from service console). It does help with the speed, but (without going into much details) it did not work very well with hot backups of live VMs. But service console is just one side of the story, and this workaround obviously cannot help with either VCB/vStorageAPI NBD/NBDSSL modes, nor NFC API performance. So I hope they will fix this in the ESX code instead.
-
- Novice
- Posts: 3
- Liked: never
- Joined: Jul 29, 2009 6:08 pm
- Full Name: Rohit Paul
- Contact:
Re: Network backup slow on vSphere ESX 4.0
Last update I received from VMware support:
"Engineering is currently working on a potential fix for the buffer cache limitation."
I'm hoping this fixes the nbd performance issue. But if it doesn't, what are my options if using direct-attached storage in ESX 4 with the VCB proxy on a physical Windows box? Would I have to look into incorporating iSCSI into my setup in order to bypass the Service Console?
"Engineering is currently working on a potential fix for the buffer cache limitation."
I'm hoping this fixes the nbd performance issue. But if it doesn't, what are my options if using direct-attached storage in ESX 4 with the VCB proxy on a physical Windows box? Would I have to look into incorporating iSCSI into my setup in order to bypass the Service Console?
-
- Chief Product Officer
- Posts: 31814
- Liked: 7302 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: Network backup slow on vSphere ESX 4.0
Well, this is exactly the scenario that I've been pushing VMware with, because there is no good workaround when local DAS storage is used. There are no problems if storage is shared (VCB SAN mode rocks), but in scenarios without shared storage there is little that can be done until the issue is fixed
-
- Novice
- Posts: 3
- Liked: never
- Joined: Jul 29, 2009 6:08 pm
- Full Name: Rohit Paul
- Contact:
Re: Network backup slow on vSphere ESX 4.0
Gostev, first of all just want to say that we are currently evaluating Veeam Backup to replace our existing ESX backup solution and thus far we're loving your product. Flawless performance with ESX 3.5 but obviously a non-Veeam issue with vSphere when using DAS/nbd.
One of the workarounds suggested is:
One of the workarounds suggested is:
In the description for SCSI Hot-Add Mode, the VCB Admin Guide states the following:If you are using VCB on the Proxy, VMware recommends to use SAN or hot-add mode instead of nbd for data transport.
Does that "I/O stack" involve the Service Console in this scenario?Consolidated Backup creates a snapshot of the virtual disk to be protected and hot adds the snapshot to the VCB proxy, allowing it to access virtual machine disk data. The VCB proxy reads the data through the I/O stack of the ESX host.
-
- Chief Product Officer
- Posts: 31814
- Liked: 7302 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: Network backup slow on vSphere ESX 4.0
Unfortunately, hot-add mode requires that VCB proxy is virtual (installed in a virtual machine), and this way it can only access storage that is accessible to ESX host virtual VCB proxy is running on. So again, in case of shared storage everything is awesome - single virtual Veeam Backup install can access everything. But in case of local or DAS storage, it can only access own ESX host's storage. So you would have to install Veeam Backup VM on every single ESX host. This is hardly usable approach...
-
- Enthusiast
- Posts: 35
- Liked: never
- Joined: Jan 01, 2006 1:01 am
- Contact:
Re: Network backup slow on vSphere ESX 4.0
Interesting VMware only talks about backup degradation in this KB. Cloning VM from another VM or template or moving it to another host takes 10 times it took with 3.5 when local storage is used. If I am not mistaken deleting (consolidating) sizable snapshots is very slow as well.
-
- Chief Product Officer
- Posts: 31814
- Liked: 7302 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: Network backup slow on vSphere ESX 4.0
Finally some good new, I've just got confirmation from VMware that their engineering is indeed working on a code change. They will be doing in-house testing to validate the results first, and then will make the test build available for us.
-
- Novice
- Posts: 8
- Liked: never
- Joined: Apr 22, 2009 7:23 pm
- Full Name: PSmith
- Location: Grand Rapids, Michigan
Re: Network backup slow on vSphere ESX 4.0
Got a call from VMWare support that engineering does not have a hotfix available to users.
What I am told it that unfortunately the code is outside the mainline of the VMWare software so users who use the hotfix will never be able to get back into the mainline software. They will support this off shoot code for two years and then you will be forced to find a different backup method. It became quite clear that VMWare has no intentions of every allowing the backups through the service console again.
What I am told it that unfortunately the code is outside the mainline of the VMWare software so users who use the hotfix will never be able to get back into the mainline software. They will support this off shoot code for two years and then you will be forced to find a different backup method. It became quite clear that VMWare has no intentions of every allowing the backups through the service console again.
-
- Chief Product Officer
- Posts: 31814
- Liked: 7302 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: Network backup slow on vSphere ESX 4.0
If this issue only affected service console backups, then that would be acceptable. However, this issue also affects VCB and even the newest vStorage API running in modes other then direct SAN access (NBD and NBDSSL modes). So what are those customers with no shared storage are supposed to do down the road?
-
- Enthusiast
- Posts: 35
- Liked: never
- Joined: Jan 01, 2006 1:01 am
- Contact:
Re: Network backup slow on vSphere ESX 4.0
All standard operations such as moving, cloning, creating from template VMs residing on local storage are 5-7 times slower than on 3.5 for the same reason. Ate they going to just leave it as is?
-
- Influencer
- Posts: 16
- Liked: never
- Joined: Jul 15, 2009 11:04 am
- Full Name: RolandK
- Contact:
Re: Network backup slow on vSphere ESX 4.0
this issue is known for more than 3 months now.
all vendors of console based backup solutions should have a real problem.
i wonder what`s the status with this?
didn`t vmware still not provide a patch or undocumented param ?
ah, and btw - if you do a "dd if=/dev/localrawdiskdevice of=/dev/null bs=1M" (e.g. dd if=/dev/sda5 of=/dev/null bs=1M), then speed is at normal rate, so it´s the vmfs layer which is thwarting us, i.e. there is no "generic" read/write throttle in esx console.
all vendors of console based backup solutions should have a real problem.
i wonder what`s the status with this?
didn`t vmware still not provide a patch or undocumented param ?
ah, and btw - if you do a "dd if=/dev/localrawdiskdevice of=/dev/null bs=1M" (e.g. dd if=/dev/sda5 of=/dev/null bs=1M), then speed is at normal rate, so it´s the vmfs layer which is thwarting us, i.e. there is no "generic" read/write throttle in esx console.
-
- Chief Product Officer
- Posts: 31814
- Liked: 7302 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: Network backup slow on vSphere ESX 4.0
I will request the status update from VMware. This issue should become less of a problem with Backup 4.0 and changed block tracking enabled; however full backup will still be affected until this is fixed...
-
- Influencer
- Posts: 16
- Liked: never
- Joined: Jul 15, 2009 11:04 am
- Full Name: RolandK
- Contact:
Re: Network backup slow on vSphere ESX 4.0
did you get a status update?
-
- Chief Product Officer
- Posts: 31814
- Liked: 7302 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: Network backup slow on vSphere ESX 4.0
We did request it, but there was no response. I will ping.
-
- Chief Product Officer
- Posts: 31814
- Liked: 7302 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: Network backup slow on vSphere ESX 4.0
Got an update from VMware today: they are planning to provide us with a test version of patch for testing and validation purposes in the next couple of weeks.
-
- Lurker
- Posts: 1
- Liked: never
- Joined: Sep 30, 2009 5:56 pm
- Full Name: Maurice Baalman
- Contact:
Re: Network backup slow on vSphere ESX 4.0
As a user with direct attached storage on two hosts, I am very interested in this resolution. What is status from VMware on the patch???Gostev wrote:If this issue only affected service console backups, then that would be acceptable. However, this issue also affects VCB and even the newest vStorage API running in modes other then direct SAN access (NBD and NBDSSL modes). So what are those customers with no shared storage are supposed to do down the road?
Additionally, what is impact of moving to with Veeam 4.0 on hosts with direct attached storage?
-
- Chief Product Officer
- Posts: 31814
- Liked: 7302 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: Network backup slow on vSphere ESX 4.0
We have the alpha version of the patch (new ESX4 build), I will be testing it within next few days. I am not sure I understand your other question though.
-
- Veteran
- Posts: 357
- Liked: 17 times
- Joined: Feb 13, 2009 10:13 am
- Full Name: Trevor Bell
- Location: Worcester UK
- Contact:
Re: Network backup slow on vSphere ESX 4.0
Anton,
Any Furter feedback yet on this ? i think alot of people are awaiting some good news
Any Furter feedback yet on this ? i think alot of people are awaiting some good news
Who is online
Users browsing this forum: Bing [Bot], Semrush [Bot] and 75 guests