Network backup slow on vSphere ESX 4.0

[smileythf]

In response to rpauls note, does Veeam Backup and Replication support the SCSI Hotadd mode of VCB?

Thank,
Tim

[Gostev]

Veeam Backup and Replication v4 supports this mode.

[rpaul]

Any info on a finalized release date for version 4?

[Gostev]

We are planning to make this generally available this month, but I cannot name the exact date for a number of reasons.

[Gostev]

Anton, any Furter feedback yet on this ? i think alot of people are awaiting some good news

Trevor, I can see some improvement over ESX4 RTM. But let me please discuss results with VMware before posting them here.

[TrevorBell]

[MadDog2K]

ESXi4 with local storage (SAS) is also affected, is the 'supposed fix' from VMware also intended to correct that ?
Or only for ESX4 ?

[Gostev]

Yes, ESXi 4 is also affected by this issue.

[louch]

by Gostev » Wed Sep 30, 2009 2:27 pm

We have the alpha version of the patch (new ESX4 build), I will be testing it within next few days. I am not sure I understand your other question though.

-----------

Any update on the alpha code Gostev?

We are still getting very poor backup speeds using windows CIFS.

Can you please update us on the matter.

Has anyone else gotten a work around for the backups to work at better speeds??

[Gostev]

No new updates so far except the one I posted above on Oct 5th... first thing I will do when I have an update, I will post it here, trust me!

But to be honest, giving the speed we are seeing with Veeam Backup v4 on ESX4 (due to leveraging the vStorage API and changed block tracking), this issue becomes less important. Even though actual data copy is slow, change tracking still makes incremental backups extremely fast (VM processing speed is hundreds MB/s). So really only full backups are affected - which only happen once in Veeam anyway, thanks to synthetic backup.

[louch]

How can we get a copy of version 4.0 to test/try this?

or when will it be out for download?

even a beta would be great for us to test!

[Gostev]

Unfortunately we do not have public beta for this release. But we are planning to make version 4.0 generally available later this month, so the waiting is not going to be long.

[bbeavis]

I have read about quite a few people with this issue. Is there any way to remove the governor from the service console? Too bad there isn't a class action suit, since they are clearly using this to gently nudge people to buy a SAN (EMC wink wink).

We bought 10GbE connections only to find they are capped at 22MBps.

[Gostev]

10GbE and ESX4... ouch! Well, at least VMware is definitely working hard lately attempting to resolve this.

[louch]

after installing veeam 4.0 with vstorage api my backups are SUPER fast and I no longer have slow backup issues!

Even my first complete backup of the VM went pretty good! Now with synthetic backups my backups finish in no time at all!

Thank you VEEAM!!! I can finally stop worrying about my backups at night you kick vizioncores a$$

[bbeavis]

Has anyone tried replacing scp with a recompiled binary in order to remove the limits?

[tsightler]

The limit is not with SCP. It appears even when using other methods like agentless mode, and VMware's own VCB NBD mode and can even be seen simply copy files around with the service console. Basically, it appears that VMware didn't realize that there were actually good reasons you might need a decent performance out of the service console and, when they redesigned that environment to access the underlying hardware via the VMkernel they just didn't take performance into consideration. I've wondered if you could simply increase the resources to the service console to get some performance back but I haven't tried because I've decided vStorage API is the way to go.

[albertwt]

Yes, that's true, using the NBD with the current separate subnet of VB4.0 physical backup server and the SAN, I've got like:

22 of 22 VMs processed (0 warnings) Details
Total size: 792.0 GB Processed size: 712.0 GB
Processing rate: 15 MB/s Total time: 13 hours 54 minutes

so in this case you're all not alone in this matter.

Next time I'll try to use with vStorage API and make the backup server in the same subnet as the iSCSI SAN.

[vmbackupuser]

VMware's official stance on the the slower network speed via the service console in ESX 4 (Updated: Nov 19, 2009)
http://kb.vmware.com/selfservice/micros ... Id=1012159

[Eriontec]

Any info on this subject?

I have several small scale customers with directly attached disks who replicate their vm's between servers.

I've upgrade from 3.5 => 4 with performance downgrades on the replication (15 - 20 mb/s max)

Now i've upgrade to 4 U1 and still it doesn't seem to be fixed by Vmware the speeds my customers are getting are still the same.

[Gostev]

Erik, no news, but your customers should really be getting better incremental replication performance than on ESX 3.5 due to the ESX4 changed block tracking, despite service console disk read slowness. Have they switched their jobs to the vStorage API processing mode?

[Gostev]

Reportedly, the latest patch from VMware somewhat fixes the issue:
New ESX Patch ESX400-200912401-BG

[Gostev]

Confirmed the patch to improve the service console agent based backup speed on ESX4 in about 2 times for me compared to ESX4 RTM.

[deebsr]

Is this patch part of the vSphere Update1? what about Update2?

[Vitaliy S.]

Hi Shaun,

As far as I know this patch didn't make it to vSphere Update 1 release. Also there is no information about the patch if you look at the release notes for Update 2 either. So I believe you should still install it separately.

Thanks!