Comprehensive data protection for all workloads
sphilp
Enthusiast
Posts: 36
Liked: 9 times
Joined: May 28, 2009 7:52 pm
Full Name: Steve Philp
Contact:

Network backup slow on vSphere ESX 4.0

Post by sphilp » Jun 03, 2009 4:43 am

[Edited by Gostev] Issue summary: ESX 4.0 service console has disk read cap/throttling issue that results in network backup becoming 3-4 times slower after upgrading from ESX 3.5 to ESX 4.0. This issue is being investigated by VMware. For Veeam Backup, current recommended workaround is to use VCB SAN mode instead (if you are using shared storage). More information can be found in my blog at vNotion.com

Resolution:
ESX 4.0 patch is now available from VMware

Update:
ESX 4.1 fixes the issue completely


We have upgraded to vSphere (full) and we're seeing a drastic reduction in backup speeds. Where guest backups would previously report 50-70MB/s, we're now seeing things top out at 25MB/s, even for the followup incremental backups.

We struck it up as a fluke on the first datacenter in our main facility, but we're now seeing the exact same performance after installing in our second datacenter.

We're using Veeam Backup 3.1, installed on a fresh Windows 2008 64-bit guest with 2 CPU and 2GB memory. It was previously installed in a Windows 2008 32-bit guest.

The physical server has 8 2.83GHz CPUs, 32GB memory and is showing approximately 20% host utilization at the time backups are occurring.

The hosts are connected via 4GB/s FC to an 8-drive EVA 4400 through a Brocade 200E switch. The backups are writing to a 16-drive RAID50 Samba share. The Samba server isn't experiencing high CPU loads, we're seeing about 6% utilization there during the backups.

The vSphere Service Console is on vSwitch0. The vSwitch has a Service Console port and a VMKernel port. The vSwitch is connected to 2 physical adapters, both Broadcom BCM5708 gig NICs. These are the same NICs that were used under ESX 3.5 and Veeam Backup 3.0.1 with great backup performance.

Have checked Ethernet NICs and switch ports, no errors. Have checked FC ports and switches, nothing odd there either.

The vSphere installations on the two hosts were clean installs, not upgrades. The vCenter install was also on a fresh Windows 2008 64-bit guest. It was previously on a Windows 2003 32-bit guest.

The guests were all updated with the new VMware Tools, part of them were upgraded with the newer VM Hardware. Old or new VM Hardware doesn't seem to make a difference in the backup speed.

Testing the SAN and Samba server via IOmeter get us performance that we'd expect, much higher than we're seeing via Veeam backups.

We have tested backing up via a Samba share (\\server\share) and doing "direct" to the Linux Samba server. Both perform equally as bad.

Have checked the VMware forums for possible issues with the FC hardware, but came up empty. Have watched these forums for the past few days to see if others were seeing this, but didn't see anything there either.

Sort of at the end of my rope as to what is causing this drastic slowdown. A 330GB guest is now taking about 4.5 hours to backup and that's REALLY ridiculous... Sorry for the long post, but wanted to provide the hardware and install steps along with the troubleshooting we've attempted so far.

Would love to hear any suggestions for how to narrows this down and resolve it!

pjh
Novice
Posts: 7
Liked: never
Joined: Jun 03, 2009 4:00 am
Full Name: Peter Hosking

Re: Stumped by Backup Speed change

Post by pjh » Jun 03, 2009 5:39 am

Was your previous W2K8 32bit running with 2 vCPU's? How many other VM's are there running? How many vCPU's are they using?

Cheers!! PJH

sphilp
Enthusiast
Posts: 36
Liked: 9 times
Joined: May 28, 2009 7:52 pm
Full Name: Steve Philp
Contact:

Re: Stumped by Backup Speed change

Post by sphilp » Jun 03, 2009 5:46 am

Yes, the previous Veeam Backup guest was also running with 2VCPUs. We've tried both 1 and 2 VCPUs for the guest with no change in performance.

There are a total of 28 guests running on a host in our main facility, they're assigned a total of 36 VCPUs. The secondary facilities host runs 9 guests and they're assigned 19 VCPUs.

In day-to-day operations, only 1 server in each datacenter is needed, so the second host in the cluster is in standby mode. We did test bringing up the second server and putting ONLY the vCenter, Veeam and currently-backing-up guest on the machine. No change in backup speed....

pjh
Novice
Posts: 7
Liked: never
Joined: Jun 03, 2009 4:00 am
Full Name: Peter Hosking

Re: Stumped by Backup Speed change

Post by pjh » Jun 03, 2009 5:54 am

Sounds like the only difference apart from going to vSphere is your VM changing from 32bit to 64bit?

From what I've seen so far Vsphere improves performance.

sphilp
Enthusiast
Posts: 36
Liked: 9 times
Joined: May 28, 2009 7:52 pm
Full Name: Steve Philp
Contact:

Re: Stumped by Backup Speed change

Post by sphilp » Jun 03, 2009 6:15 am

Yep, the only changes would be the vCenter and Veeam Backup guests going from 2003 32-bit to 2008 64-bit.

We did test running backups in the original Veeam Backup guest, but had the same dismal speeds there. Since it wasn't that, we went ahead and created the new 2008 64-bit guest and that's what the guest is using now.

We're happy with the stuff we're seeing in vSphere -- memory usage has declined a bunch. Just need to figure out what's happening with the backups...

lohelle
Service Provider
Posts: 77
Liked: 15 times
Joined: Jun 03, 2009 7:45 am
Full Name: Lars O Helle
Contact:

Re: Stumped by Backup Speed change

Post by lohelle » Jun 03, 2009 7:53 am

[Edited by Gostev]Issue summary: faulty backup storage disk controller (see below for more details and troubleshooting path).

I have tested Veeam Backup 3.0 before against ESX 3.5. Backup speed was 50-80 MB/s (in SAN mode). VCB from command line = 70 - 80 MB/s.

Now in Veeam Backup 3.1 + ESX 4.0, the same SAN and the same backup server/storage backup speed is 15-25 MB/s (optimal compression). VCB is still 70-80 MB/s from command line, but not in Veeam Backup.
The speed is so bad that er are having problems completing backups before the next should start, and that with only 2 TB of data.
The backup server is running Windows Server 2003 SP2 (fully patched) with the newest MS Iscsi Initiator. (32-bit Windows)

Gostev
SVP, Product Management
Posts: 24638
Liked: 3467 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Stumped by Backup Speed change

Post by Gostev » Jun 03, 2009 12:03 pm

All, have you upgraded VCB to version 1.5 Update 1 that is required for vSphere?

Steve,
What backup mode do you use in Veeam Backup?

Lars,
What command line do you use to test VCB speed?
Do I understand it correctly that you have Veeam Backup installed on physical server, but using software iSCSI initiator?

sphilp
Enthusiast
Posts: 36
Liked: 9 times
Joined: May 28, 2009 7:52 pm
Full Name: Steve Philp
Contact:

Re: Stumped by Backup Speed change

Post by sphilp » Jun 03, 2009 12:31 pm

Gostev wrote: All, have you upgraded VCB to version 1.5 Update 1 that is required for vSphere?

Steve,
What backup mode do you use in Veeam Backup?
We do not use a VCB server.

The backup jobs are setup to use "Network backup" (VCB Consolidated backup is greyed out) on the Job properties.

lohelle
Service Provider
Posts: 77
Liked: 15 times
Joined: Jun 03, 2009 7:45 am
Full Name: Lars O Helle
Contact:

Re: Stumped by Backup Speed change

Post by lohelle » Jun 03, 2009 12:54 pm

Using this command when using VCB only (not Veeam)
vcbmounter -h vcenter.domain.local -u administrator -p secretpassword -a ipaddr:212.184.222.222 -r j:\vcb\dc01 -t fullvm -m san

The newest MS SW initiator. Reinstalled the backup server yesterday to check if that helped (did not help)
Single path to Infortrend ISCSI SAN. As mentioned, the manual vcbmounter command gives us ca 80 MB/s backup speed.

Gostev
SVP, Product Management
Posts: 24638
Liked: 3467 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Stumped by Backup Speed change

Post by Gostev » Jun 03, 2009 12:55 pm

Steve, can you try to copy/paste some large VMDK file of some VM that is not running from your ESX server to Veeam Backup server using Veeam Backup servers tree, and let me know what speed are you getting? Then select "Force service console agent" mode for this server, and do the same thing (just to make sure that you are using service console agent mode).

Gostev
SVP, Product Management
Posts: 24638
Liked: 3467 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Stumped by Backup Speed change

Post by Gostev » Jun 03, 2009 1:01 pm

lohelle wrote:Using this command when using VCB only (not Veeam)
vcbmounter -h vcenter.domain.local -u administrator -p secretpassword -a ipaddr:212.184.222.222 -r j:\vcb\dc01 -t fullvm -m san

The newest MS SW initiator. Reinstalled the backup server yesterday to check if that helped (did not help)
Single path to Infortrend ISCSI SAN. As mentioned, the manual vcbmounter command gives us ca 80 MB/s backup speed.
Lars, you are not performing thу VCB speed test correctly, since Veeam Backup configures vcbmounter.exe with different options. You command does not create bit-identical output of VM image as it misses some options. Instead, it creates sparse output, this is what also makes it work faster.

Please, lookup the actual vcbmounter command line in Veeam Backup logs, and try this test again.

Also, I think you missed my question about VCB 1.5 Update 1.

sphilp
Enthusiast
Posts: 36
Liked: 9 times
Joined: May 28, 2009 7:52 pm
Full Name: Steve Philp
Contact:

Re: Stumped by Backup Speed change

Post by sphilp » Jun 03, 2009 1:16 pm

Gostev wrote:Steve, can you try to copy/paste some large VMDK file of some VM that is not running from your ESX server to Veeam Backup server using Veeam Backup servers tree, and let me know what speed are you getting? Then select "Force service console agent" mode for this server, and do the same thing (just to make sure that you are using service console agent mode).
Copy / paste: Used a 1.75GB ISO file stored on the same SAN LUN as the VMDK files. The transfer was reported as 18MB/s by the time the transfer finished.

Service Console agent: Turning on the Force flag and running a backup gets us the same 20-25MB/s as we were seeing before. In watching the backups, we're consistently seeing "Backup mode: service console agent", so I don't think anything changed there.

Gostev
SVP, Product Management
Posts: 24638
Liked: 3467 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Stumped by Backup Speed change

Post by Gostev » Jun 03, 2009 1:35 pm

Steve, ISO is not really good for testing since it does not contain white space. But for instance, for VM with 60% of disk space being free, you should be getting 3 times faster transfer speed than with ISO because white space will be excluded from transfer. This theoretically should provide 50+ MB/s throughtput as you've seen before (and even faster for incremental passes), and we could move on to looking for other bottlenecks.

Could you confirm this by re-testing with some VMDK file?

sphilp
Enthusiast
Posts: 36
Liked: 9 times
Joined: May 28, 2009 7:52 pm
Full Name: Steve Philp
Contact:

Re: Stumped by Backup Speed change

Post by sphilp » Jun 03, 2009 2:11 pm

Gostev wrote:Steve, ISO is not really good for testing since it does not contain white space. But for instance, for VM with 60% of disk space being free, you should be getting 3 times faster transfer speed than with ISO because white space will be excluded from transfer. This theoretically should provide 50+ MB/s throughtput as you've seen before (and even faster for incremental passes), and we could move on to looking for other bottlenecks.

Could you confirm this by re-testing with some VMDK file?
Tested with a shut down guest (32GB VMDK, 9GB used, changed to thin provisioned from thick after vSphere installation using svMotion), and I'm seeing 34MB/s at the end of the transfer. We do see the expected increase in backup speed once the "end" of the used space is reached. Prior to reaching that, the transfer was running at around 22MB/s.

lohelle
Service Provider
Posts: 77
Liked: 15 times
Joined: Jun 03, 2009 7:45 am
Full Name: Lars O Helle
Contact:

Re: Stumped by Backup Speed change

Post by lohelle » Jun 03, 2009 2:12 pm

I just tested with the VCB command from Veeam logs. The main difference is the -F 1 -M 1 export flags.
I see that when using the -F 1 parameter the VMDK is preallocated on the backup server (almost no network traffic), and after som time (when passing 50% on the progress bar) network traffic starts (70-80MB/s)
When removing the -F 1 parameter the backup start to "download" at once, and backup is completed very fast.

The backup speed of Veeam Backup 3.0 + ESX 3.5 was about the same as the VCB backup without the -F 1 parameter.

This does not explain Steve's problem I think, but it narrows down mine.

btw: the VM I'm running backup of:
8GB VMDK (thick)
7,5GB data inside

Gostev
SVP, Product Management
Posts: 24638
Liked: 3467 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Stumped by Backup Speed change

Post by Gostev » Jun 03, 2009 3:07 pm

Steve, can you please clarify? You are saying that you don't see expected speed increase in the end of transfer, yet you also mention it is 34MB/s at the end of transfer vs. 22MB/s prior reaching the end. I don't understand?

Lars, Veeam Backup has been using the same command line for vcbmounter.exe since 1.0, so we did not have any changes here. I think you missed my question about VCB 1.5 Update 1?

May be it would be best if you both open support cases and work directly with our support.

If anyone else is seeing or not seeing speed reduction after upgrading to vSphere, please post your experiences and stats in this thread.

sphilp
Enthusiast
Posts: 36
Liked: 9 times
Joined: May 28, 2009 7:52 pm
Full Name: Steve Philp
Contact:

Re: Stumped by Backup Speed change

Post by sphilp » Jun 03, 2009 3:20 pm

Gostev wrote:Steve, can you please clarify? You are saying that you don't see expected speed increase in the end of transfer, yet you also mention it is 34MB/s at the end of transfer vs. 22MB/s prior reaching the end. I don't understand?
Sorry, may have jumbled up my terminology there...

The VMDK that we're testing with is 32GB, thin-provisioned. The used portion is 9GB. The free space would be 23GB.

During the initial 9GB of the file transfer, we're seeing consistent transfer speeds of 22MB/s. Once we get past the 9GB point, transfer speeds increase and by the time the file transfer finishes, it is reporting 34MB/s.

We're NOT seeing those transfer speeds when doing backups. A backup job for that same guest will report only 20-25MB/s.

Gostev
SVP, Product Management
Posts: 24638
Liked: 3467 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Stumped by backup speed change after upgrading to vSphere

Post by Gostev » Jun 03, 2009 3:28 pm

Steve, this would indicate problems with your backup storage speed, because backup job uses the same engine as copy. Can you create new backup job for the same guest, configure it to save backup locally to Veeam Backup server, and run it? It should be just a little slower than regular copy, because snapshot operations and small VM files processing (VMX etc.) all take time affecting avg. backup performance rate counter. But not significantly slower.

sphilp
Enthusiast
Posts: 36
Liked: 9 times
Joined: May 28, 2009 7:52 pm
Full Name: Steve Philp
Contact:

Re: Stumped by backup speed change after upgrading to vSphere

Post by sphilp » Jun 03, 2009 3:36 pm

But it doesn't explain why the exact same backup storage worked at a significantly higher speed with ESX 3.5/Backup 3.0.1.

Testing local backup now...

lohelle
Service Provider
Posts: 77
Liked: 15 times
Joined: Jun 03, 2009 7:45 am
Full Name: Lars O Helle
Contact:

Re: Stumped by backup speed change after upgrading to vSphere

Post by lohelle » Jun 03, 2009 3:49 pm

The setup-file is VMware-vcb-150805.exe
A quick search on Google seems to indicate that this is VCB 1.5 Update 1
Maybe this new version have some changes in the way it preallocate the vmdk's on the target?

Gostev
SVP, Product Management
Posts: 24638
Liked: 3467 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Stumped by backup speed change after upgrading to vSphere

Post by Gostev » Jun 03, 2009 3:50 pm

Steve, it could be an issue with ESX4, or the storage you are using. We have not mentioned any differences during our own testing, but we have different storage hardware (mostly testing on ESX with local storage, and iSCSI SAN).

It would be nice if you could perform the same test on ESX 3.5 but I am guessing you don't have any 3.5 hosts left.

sphilp
Enthusiast
Posts: 36
Liked: 9 times
Joined: May 28, 2009 7:52 pm
Full Name: Steve Philp
Contact:

Re: Stumped by backup speed change after upgrading to vSphere

Post by sphilp » Jun 03, 2009 3:57 pm

Results from backing up to a local drive located on the SAN:

Code: Select all

Total VM size: 32.00 GB
Processed size: 32.00 GB
Avg. performance rate: 34 MB/s
Backup mode: service console agent
Looks like we start pointing fingers at the Samba server?

Gostev
SVP, Product Management
Posts: 24638
Liked: 3467 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Stumped by backup speed change after upgrading to vSphere

Post by Gostev » Jun 03, 2009 4:10 pm

...or some virtual networking "optimization" in ESX4? Since you write to Samba server over the network from Veeam Backup installed in a VM.

sphilp
Enthusiast
Posts: 36
Liked: 9 times
Joined: May 28, 2009 7:52 pm
Full Name: Steve Philp
Contact:

Re: Stumped by backup speed change after upgrading to vSphere

Post by sphilp » Jun 03, 2009 4:42 pm

Thanks, we'll continue digging!

Gostev
SVP, Product Management
Posts: 24638
Liked: 3467 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Stumped by Backup Speed change

Post by Gostev » Jun 03, 2009 7:21 pm

lohelle wrote:I see that when using the -F 1 parameter the VMDK is preallocated on the backup server (almost no network traffic), and after som time (when passing 50% on the progress bar) network traffic starts (70-80MB/s)
Lars, for this experiment, I was mostly interested to see the VCB throughtput for actual data copy, and it looks decent. Can you please confirm that during this VCB testing, you pointed vcbmounter.exe to the same location used by your backup jobs, or was it different location (local drive on VCB proxy)?

Secondly, what network traffic do you see when creating Veeam Backup job for the same VM, and poining the job to the same backup storage location as during VCB testing?

lohelle
Service Provider
Posts: 77
Liked: 15 times
Joined: Jun 03, 2009 7:45 am
Full Name: Lars O Helle
Contact:

Re: Stumped by backup speed change after upgrading to vSphere

Post by lohelle » Jun 03, 2009 10:02 pm

Pointing to the same backup target.
When running vcbmounter with the same parameters as Veeam (-F 1 -M 1) backup is slow. When running the "normal" parameters (that is without -F 1 and -M 1) the backup speed is very good.
I have tested with many VM's. Most VM's have almost full disks, so the speed is actual VM data transfered, not "white space"

VCB backup of the main cluster (vcbmounter) takes approx 8 hours. Veeam backup (vcb) takes almost 24 hours. Non-VCB backup is not any better (only tried fulls)

As mentioned: "Normal" VCB backup to the same backup store (local disks) from the same SAN and same "cluster" is 3x faster. Very strange.
I do not have any esx 3.5 hosts right now, so I can not test against an older ESX version.

Gostev
SVP, Product Management
Posts: 24638
Liked: 3467 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Stumped by backup speed change after upgrading to vSphere

Post by Gostev » Jun 03, 2009 10:29 pm

Lars, can you please provide some specific metrics for that 8GB VM you mentioned earlier backed up to the same target?

Time to complete operation for:
- vcbmounter.exe without special parameters (or what you call "normal")
- vcbmounter.exe with Veeam command line parameters
- Veeam Backup job in "VCB SAN" mode (first run after job is created, and second run)
- Veeam Backup job in "Network" mode (first run after job is created, and second run)

I hope this data will give me more clues. By the way, where are you backing up to?

sphilp
Enthusiast
Posts: 36
Liked: 9 times
Joined: May 28, 2009 7:52 pm
Full Name: Steve Philp
Contact:

Re: Stumped by backup speed change after upgrading to vSphere

Post by sphilp » Jun 04, 2009 2:41 am

Did some additional testing tonight to try to narrow down where the problems might be...

For this test, I copied an ISO file from the vSphere host (stored on the same SAN LUN as the VMDKs being backed up) to a local drive on the Veeam guest. The 1.75GB ISO file reported a transfer speed of 18MB/s.

I then copied that ISO file from the Veeam guest to the same share used as a backup target for Veeam Backup using Windows Explorer. The transfer reported its speed as 75MB/s.

I'm stumped about why the SAN-to-local transfer is so slow. It's connected via 4GB FC which should give us approximately 400MB/s (if the winds are blowing in the right direction, etc). The ethernet connection to the backup target is via a single 1GB copper connection and should be 4x slower, but instead it's 3x faster?!

I think I've pretty much cleared the reputation of the backup target and its network, along with the VM network connection of the guest. Those transfer speeds are what I'd expect to see.

I don't understand the inner machinations of Veeam's backup, but I'm leaning toward the belief that the problem is somewhere in Veeam's transfer from the Service Console to the Veeam guest... It certainly doesn't look to be downstream of that!

TrevorBell
Expert
Posts: 353
Liked: 7 times
Joined: Feb 13, 2009 10:13 am
Full Name: Trevor Bell
Location: Worcester UK
Contact:

Re: Stumped by backup speed change after upgrading to vSphere

Post by TrevorBell » Jun 04, 2009 10:12 am

Hi All,

I too have slower speeds with v4.. i normally backup my ISCSI Equallogic box via Network backup achieving around 30-50mb/s which i was happy with.. now i get 15mb/s max apart from a test Dc wich is 60Mb/s! I will post stats later... im just converting all Vm`s to Thin provision to find i cant back them up with VCB so doing some VCB with NBD test...will post more later i will also check out my VCB convertor and upgrade if a newer version is out.

Trev

Gostev
SVP, Product Management
Posts: 24638
Liked: 3467 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Stumped by backup speed change after upgrading to vSphere

Post by Gostev » Jun 04, 2009 10:31 am

Steve, Trevor - could you please perform the following quick test too, to check SAN-to-local transfer channel:
1. Open VMware Infrastructure Client on your Veeam Backup server.
2. Right-click your datastore, open the datastore browser.
3. Download a large test file from you storage to the local drive.
4. Note the time the operation took to calculate average transfer speed.

Hint: don't copy the same file more than once, or Windows file system cache will affect the results.

Locked

Who is online

Users browsing this forum: Baidu [Spider] and 26 guests