Network backup slow on vSphere ESX 4.0

[sphilp]

Gostev - Thank you for your work in testing things and confirming the speed problem we're seeing, it's highly appreciated.

As noted in your post, VCB doesn't suffer from the slowdown. We've switched our backups to use that mode and see speeds comparable to the 3.5 Service Console method we used previously.

At this point, I think we're happy again!

[Gostev]

OK, thanks for the update Steve... I assume this issue will not be fixed by VMware until vSphere Update 1 (which is scheduled around Q4 time frame as far as I know). I've also updated this thread's name (search optimization).

[Gostev]

Steve, could you please clarify if you are now using VCB in SAN mode, or in NBD mode? I am just trying to understand if VCB NBD mode still suffers from the same issue (since in NBD mode, all data still goes through ESX I/O stack and thus may be subject to ESX4 service console throttling issue).

[alexr]

I have the same issue with network backup. my speed dropped 5-6 times and I do not have enough time to finish backups.
I tried to switch to VCB mode but got an error:

Initializing source
The object has already been deleted or has not been completely created

I installed latest VCB from esx4 CD

I tried to mount full VM using vcbMounter in ncb mode (simple vcbMounter -m nbd -u vmware-backup -t fullvm) it worked but was very slow as well ~21MB/s

Anything I can do?

[sphilp]

Gostev - We're now using VCB in SAN mode.

[Gostev]

Steve, thanks for clarification. As I can see from what Alex is saying, NBD mode suffers from the same problem.

Alex, we have opened a support case with VMware on this issue to better track it. Unfortunately, at this time there is no workaround available except using VCB SAN mode, which requires shared storage.

[alexr]

Any news from VMware?

[Gostev]

I have performed a test with requested logging enabled, and sent the logs today (this morning). I've also invited VMware support engineer to do a live webex for faster troubleshooting (actually, 2 times already). Waiting for response (have not heard back yet since Friday night).

I will post here as soon as I get any news.

[Gostev]

So just to provide some sort of update: the support case is keep going up the chain, so far we've been told we are not the only one reporting the degraded performance, and that it is currently being investigated by VMware engineering department.

[joemagjr]

Curious to know, has their been any new developments on this? Will changing from using an agent (automatic) to agentless engine on the host under data transfer engine help with the speed of the backups as an attempt to band-aide the problem for now?

[Gostev]

Unfortunately, changing between agent and agentless will not help. Both technologies end up in the service console (either in Veeam agent, or in VMware service). And it is service console that is being "throttled" in ESX4, so anything running in service console is affected. This includes all 3rd party service console agents, as well as native VMware services (those serving VIC datastore browser, or VCB in NBD mode).

We are currently working together with VMware in testing this, and we will publish plans for a resolution as soon as they are available.

[psmith]

Got an email today that VMWare was able to diagnose the issue and is currently working on a fix for the problem. They will be contacting me to test it once they have a fix ready to test out, and I will post the results to those of you following this post.

In the meantime here is the Knowledge Base article they have created for it;
http://kb.vmware.com/selfservice/micros ... Id=1012159

[Gostev]

Finally! In fact, they diagnosed the issue long ago, it just took them a good month to create support KB article for some reason.

I am glad to hear they told you they are working on the fix. Initially, they were saying this issue cannot be fixed because it is inadvertant effect from another major vSphere change that they just cannot rollback.

We did test one potential workaround they provided couple of weeks ago (certain flag to use when opening file from service console). It does help with the speed, but (without going into much details) it did not work very well with hot backups of live VMs. But service console is just one side of the story, and this workaround obviously cannot help with either VCB/vStorageAPI NBD/NBDSSL modes, nor NFC API performance. So I hope they will fix this in the ESX code instead.

[rpaul]

Last update I received from VMware support:
"Engineering is currently working on a potential fix for the buffer cache limitation."

I'm hoping this fixes the nbd performance issue. But if it doesn't, what are my options if using direct-attached storage in ESX 4 with the VCB proxy on a physical Windows box? Would I have to look into incorporating iSCSI into my setup in order to bypass the Service Console?

[Gostev]

Well, this is exactly the scenario that I've been pushing VMware with, because there is no good workaround when local DAS storage is used. There are no problems if storage is shared (VCB SAN mode rocks), but in scenarios without shared storage there is little that can be done until the issue is fixed

[rpaul]

Gostev, first of all just want to say that we are currently evaluating Veeam Backup to replace our existing ESX backup solution and thus far we're loving your product. Flawless performance with ESX 3.5 but obviously a non-Veeam issue with vSphere when using DAS/nbd.

One of the workarounds suggested is:

If you are using VCB on the Proxy, VMware recommends to use SAN or hot-add mode instead of nbd for data transport.

In the description for SCSI Hot-Add Mode, the VCB Admin Guide states the following:

Consolidated Backup creates a snapshot of the virtual disk to be protected and hot adds the snapshot to the VCB proxy, allowing it to access virtual machine disk data. The VCB proxy reads the data through the I/O stack of the ESX host.

Does that "I/O stack" involve the Service Console in this scenario?

[Gostev]

Unfortunately, hot-add mode requires that VCB proxy is virtual (installed in a virtual machine), and this way it can only access storage that is accessible to ESX host virtual VCB proxy is running on. So again, in case of shared storage everything is awesome - single virtual Veeam Backup install can access everything. But in case of local or DAS storage, it can only access own ESX host's storage. So you would have to install Veeam Backup VM on every single ESX host. This is hardly usable approach...

[alexr]

Interesting VMware only talks about backup degradation in this KB. Cloning VM from another VM or template or moving it to another host takes 10 times it took with 3.5 when local storage is used. If I am not mistaken deleting (consolidating) sizable snapshots is very slow as well.

[Gostev]

Finally some good new, I've just got confirmation from VMware that their engineering is indeed working on a code change. They will be doing in-house testing to validate the results first, and then will make the test build available for us.

[psmith]

Got a call from VMWare support that engineering does not have a hotfix available to users.

What I am told it that unfortunately the code is outside the mainline of the VMWare software so users who use the hotfix will never be able to get back into the mainline software. They will support this off shoot code for two years and then you will be forced to find a different backup method. It became quite clear that VMWare has no intentions of every allowing the backups through the service console again.

[Gostev]

If this issue only affected service console backups, then that would be acceptable. However, this issue also affects VCB and even the newest vStorage API running in modes other then direct SAN access (NBD and NBDSSL modes). So what are those customers with no shared storage are supposed to do down the road?

[alexr]

All standard operations such as moving, cloning, creating from template VMs residing on local storage are 5-7 times slower than on 3.5 for the same reason. Ate they going to just leave it as is?

[rolandk]

this issue is known for more than 3 months now.
all vendors of console based backup solutions should have a real problem.

i wonder what`s the status with this?

didn`t vmware still not provide a patch or undocumented param ?

ah, and btw - if you do a "dd if=/dev/localrawdiskdevice of=/dev/null bs=1M" (e.g. dd if=/dev/sda5 of=/dev/null bs=1M), then speed is at normal rate, so it´s the vmfs layer which is thwarting us, i.e. there is no "generic" read/write throttle in esx console.

[Gostev]

I will request the status update from VMware. This issue should become less of a problem with Backup 4.0 and changed block tracking enabled; however full backup will still be affected until this is fixed...

[rolandk]

did you get a status update?

[Gostev]

We did request it, but there was no response. I will ping.

[Gostev]

Got an update from VMware today: they are planning to provide us with a test version of patch for testing and validation purposes in the next couple of weeks.

[MJB]

If this issue only affected service console backups, then that would be acceptable. However, this issue also affects VCB and even the newest vStorage API running in modes other then direct SAN access (NBD and NBDSSL modes). So what are those customers with no shared storage are supposed to do down the road?

As a user with direct attached storage on two hosts, I am very interested in this resolution. What is status from VMware on the patch???

Additionally, what is impact of moving to with Veeam 4.0 on hosts with direct attached storage?

[Gostev]

We have the alpha version of the patch (new ESX4 build), I will be testing it within next few days. I am not sure I understand your other question though.

[TrevorBell]

Anton,

Any Furter feedback yet on this ? i think alot of people are awaiting some good news