Discussions specific to the VMware vSphere hypervisor
Post Reply
beistrich
Influencer
Posts: 16
Liked: never
Joined: Feb 04, 2014 8:06 am
Full Name: Christoph
Contact:

Backup Incredible Slow - Processing at about 200kb/s

Post by beistrich » Feb 04, 2014 8:15 am

Hallo!

I set up Veeam Backup&Replication as usual.
- Veeam is installed on a virtual server (and is used as Proxy)
- Backup repository is a NAS
- VSS, indexing and cbt enabled

Never had any issues before. But now i have enormous performance problems - running backups with about 200kb/s. It is a new environment with fresh Veeam Installation. (Its a single vSphere-host deployment)

I already tried disabling different Options (vss, indexing cbt...) - nothing helped.

Has anybody an idea what i can try?

Image

Vitaliy S.
Product Manager
Posts: 23384
Liked: 1658 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Backup Incredible Slow - Processing at about 200kb/s

Post by Vitaliy S. » Feb 04, 2014 8:18 am

Hello Christoph,

According to the screenshot above, source might be the problem, but I would still ask for full bottleneck stats for this job. Also what kind of VM is it? Can you please try to backup another VM and confirm that you have the same performance issues?

Thank you!

veremin
Product Manager
Posts: 17229
Liked: 1506 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Backup Incredible Slow - Processing at about 200kb/s

Post by veremin » Feb 04, 2014 8:36 am

As to CBT, disabling it won't increase the processing speed. Instead, it will increase the job time, as without CBT the whole VM image needs to be read by VB&R in order to find changes.

Thanks.

foggy
Veeam Software
Posts: 18637
Liked: 1627 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Backup Incredible Slow - Processing at about 200kb/s

Post by foggy » Feb 04, 2014 9:48 am

In addition to full bottleneck stats,I also wonder what transport mode is being utilized to retrieve VM data from the source? (You can select the particular VM in the list to the left and see the transport mode tag next to the proxy server name.)

beistrich
Influencer
Posts: 16
Liked: never
Joined: Feb 04, 2014 8:06 am
Full Name: Christoph
Contact:

Re: Backup Incredible Slow - Processing at about 200kb/s

Post by beistrich » Feb 04, 2014 9:51 am

Yes, i see that the bottleneck is the source - But i don't know why? I have not recognized any performance issues of other applications...

Vitaliy S.
Product Manager
Posts: 23384
Liked: 1658 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Backup Incredible Slow - Processing at about 200kb/s

Post by Vitaliy S. » Feb 04, 2014 9:56 am

Can you please clarify other information, so we could assist you further?

beistrich
Influencer
Posts: 16
Liked: never
Joined: Feb 04, 2014 8:06 am
Full Name: Christoph
Contact:

Re: Backup Incredible Slow - Processing at about 200kb/s

Post by beistrich » Feb 04, 2014 10:47 am

THe following shows up for backup Proxy:

Using backup proxy VMware Backup Proxy for disk Hard disk 1 [hotadd]

VMware Backup Proxy is "this Server" with Transport mode set to "Automatic"

foggy
Veeam Software
Posts: 18637
Liked: 1627 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Backup Incredible Slow - Processing at about 200kb/s

Post by foggy » Feb 04, 2014 11:00 am

What if you force the proxy to use network mode?

veremin
Product Manager
Posts: 17229
Liked: 1506 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Backup Incredible Slow - Processing at about 200kb/s

Post by veremin » Feb 04, 2014 11:10 am

What about CBT information? Do you see the corresponding mark ([CBT]) nearby virtual disk name? Also, as mentioned above, can you try to backup different VM and see whether the speed rates are the same? This should give us a better understanding of the scope of the issue.

Thanks.

beistrich
Influencer
Posts: 16
Liked: never
Joined: Feb 04, 2014 8:06 am
Full Name: Christoph
Contact:

Re: Backup Incredible Slow - Processing at about 200kb/s

Post by beistrich » Feb 04, 2014 2:34 pm

Hi!

I have now force "Network" on Transport mode

It doubled the "speed" to 400 kb/s

Image

beistrich
Influencer
Posts: 16
Liked: never
Joined: Feb 04, 2014 8:06 am
Full Name: Christoph
Contact:

Re: Backup Incredible Slow - Processing at about 200kb/s

Post by beistrich » Feb 04, 2014 2:44 pm

I also want to mention that it is VM Independent. Also other vms backup at same slow speed.
Could it be the disk array? I doubt that, becaus overall server performance is very good...

foggy
Veeam Software
Posts: 18637
Liked: 1627 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Backup Incredible Slow - Processing at about 200kb/s

Post by foggy » Feb 04, 2014 2:47 pm

What kind of storage is it? Bottleneck source means that your production VM storage cannot provide data any faster.

beistrich
Influencer
Posts: 16
Liked: never
Joined: Feb 04, 2014 8:06 am
Full Name: Christoph
Contact:

Re: Backup Incredible Slow - Processing at about 200kb/s

Post by beistrich » Feb 04, 2014 3:02 pm

Its a ESXi 5.5 host with local storage (6 disks with RAID 5)
Write Cache enabled on RAID Adapter

Vitaliy S.
Product Manager
Posts: 23384
Liked: 1658 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Backup Incredible Slow - Processing at about 200kb/s

Post by Vitaliy S. » Feb 04, 2014 4:23 pm

What download speed do you have if you try to download any big file from the datastore via vSphere Client -> Datastore Browser?

beistrich
Influencer
Posts: 16
Liked: never
Joined: Feb 04, 2014 8:06 am
Full Name: Christoph
Contact:

Re: Backup Incredible Slow - Processing at about 200kb/s

Post by beistrich » Feb 04, 2014 5:03 pm

WOW, ok, i think i have found the problem

Copying TO the Datastore ist quick, but also not very fast.
Copying FROM the Datastore ist incredible slow! (20 minutes for a 230 MB file)

BUT copying via SMB share at full Gigabit speed!

Anyone knows about issues about using local storage with ESXi 5.5?

This is very strange, because i have already deployed various servers with local storage, not having such issues!

sbbots
Enthusiast
Posts: 96
Liked: 25 times
Joined: Aug 16, 2013 5:44 pm
Full Name: Matt B

Re: Backup Incredible Slow - Processing at about 200kb/s

Post by sbbots » Feb 04, 2014 5:07 pm

I am using a similar setup at two remote sites; Single ESXi host, Veeam installed on a VM, backup repository on a Synology NAS (CIFS share).

I noticed slow speeds below 1Mb when going through a Cisco POE Gigabit switch. I attached the NAS directly to an extra NIC on the host server, then passed that NIC to the backup VM using Directpath. All speed issues went away immediately and backup transfer speed now completely saturates the GbE connection.

beistrich
Influencer
Posts: 16
Liked: never
Joined: Feb 04, 2014 8:06 am
Full Name: Christoph
Contact:

Re: Backup Incredible Slow - Processing at about 200kb/s

Post by beistrich » Feb 04, 2014 5:23 pm

Yes, would be an option (which i will try)
But i don't think thats really the problem, because the copying from the datastore i run from a VM ON the ESXi host. Does this traffic passes the physical switch?

sbbots
Enthusiast
Posts: 96
Liked: 25 times
Joined: Aug 16, 2013 5:44 pm
Full Name: Matt B

Re: Backup Incredible Slow - Processing at about 200kb/s

Post by sbbots » Feb 04, 2014 6:23 pm

I also run Veeam on a VM on the host, so your setup is similar. Once you connect the NAS directly to the host the traffic bypasses the switch.

Vitaliy S.
Product Manager
Posts: 23384
Liked: 1658 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Backup Incredible Slow - Processing at about 200kb/s

Post by Vitaliy S. » Feb 04, 2014 8:21 pm

Christoph, just to isolate the issue, can you please try to perform a backup of any test VM to the local storage (virtual disk of your backup proxy). What performance do you have in this case?

beistrich
Influencer
Posts: 16
Liked: never
Joined: Feb 04, 2014 8:06 am
Full Name: Christoph
Contact:

Re: Backup Incredible Slow - Processing at about 200kb/s

Post by beistrich » Feb 05, 2014 10:55 am

Hi!
Now i tried a backup to the backup-Proxy. Also slow at 300kbit/s... :(

I found an VMware article adressing such Problem:
http://kb.vmware.com/selfservice/micros ... Id=1006602
I have already configured this...

beistrich
Influencer
Posts: 16
Liked: never
Joined: Feb 04, 2014 8:06 am
Full Name: Christoph
Contact:

Re: Backup Incredible Slow - Processing at about 200kb/s

Post by beistrich » Feb 05, 2014 11:25 am

I now did a test with "esxtop"

I see that the value of CMDS/s and READ/WRITES is significantly different!

This is the result:
Write to Datastore
DEVICE PATH/WORLD/PARTITION DQLEN WQLEN ACTV QUED %USD LOAD CMDS/s READS/s WRITES/s MBREAD/s MBWRTN/s DAVG/cmd KAVG/
naa.600605b006e979901a583d9b1ad8fb71 - 128 - 0 0 0 0.00 108.02 1.00 107.01 0.03 16.27 0.16 0


READ from Datastore
DEVICE PATH/WORLD/PARTITION DQLEN WQLEN ACTV QUED %USD LOAD CMDS/s READS/s WRITES/s MBREAD/s MBWRTN/s DAVG/cmd KAVG/
naa.600605b006e979901a583d9b1ad8fb71 - 128 - 0 0 0 0.00 11.83 1.91 9.92 0.24 0.25 0.21 0

Vitaliy S.
Product Manager
Posts: 23384
Liked: 1658 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Backup Incredible Slow - Processing at about 200kb/s

Post by Vitaliy S. » Feb 06, 2014 8:05 am

beistrich wrote:Now i tried a backup to the backup-Proxy. Also slow at 300kbit/s... :(
Was it a hotadd or a network processing mode? Do you have any other ESXi host in the same environment to compare job performance rates?

beistrich
Influencer
Posts: 16
Liked: never
Joined: Feb 04, 2014 8:06 am
Full Name: Christoph
Contact:

Re: Backup Incredible Slow - Processing at about 200kb/s

Post by beistrich » Feb 09, 2014 7:26 pm

Hi!
I want to give an update to this...
On the weekend i tried different settings of the raid controller (write back, write back with bbu, write throuch, no read ahead...)
Nothing worked...
I really don't know what to try next, anybody has further ideas?

veremin
Product Manager
Posts: 17229
Liked: 1506 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Backup Incredible Slow - Processing at about 200kb/s

Post by veremin » Feb 10, 2014 9:13 am

Then, you can open a ticket with our support team and let them confirm your environment, as some of the environmental changes that decrease performance might have slipped out of your attention. Thanks.

Vitaliy S.
Product Manager
Posts: 23384
Liked: 1658 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Backup Incredible Slow - Processing at about 200kb/s

Post by Vitaliy S. » Feb 10, 2014 11:20 am

If it is a fresh environment, then using a different RAID level (for example, 10) might also help.

beistrich
Influencer
Posts: 16
Liked: never
Joined: Feb 04, 2014 8:06 am
Full Name: Christoph
Contact:

Re: Backup Incredible Slow - Processing at about 200kb/s

Post by beistrich » May 18, 2014 10:02 pm

I know its a long time.

but the problem was the Switch/ ESXi Nic configuration! It was set to autodetect, but had 100Mbit non-duplex! - This means VERY slow...
We manually set the link speed and everything worked fine!

foggy
Veeam Software
Posts: 18637
Liked: 1627 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Backup Incredible Slow - Processing at about 200kb/s

Post by foggy » May 19, 2014 9:17 am

Christoph, thanks for coming back after a long time and finally updating the thread with the resolution, much appreciated!

Post Reply

Who is online

Users browsing this forum: ABCN, Google [Bot], Majestic-12 [Bot] and 38 guests