Host-based backup of VMware vSphere VMs.
Post Reply
beistrich
Influencer
Posts: 16
Liked: never
Joined: Feb 04, 2014 8:06 am
Full Name: Christoph
Contact:

Backup Incredible Slow - Processing at about 200kb/s

Post by beistrich »

Hallo!

I set up Veeam Backup&Replication as usual.
- Veeam is installed on a virtual server (and is used as Proxy)
- Backup repository is a NAS
- VSS, indexing and cbt enabled

Never had any issues before. But now i have enormous performance problems - running backups with about 200kb/s. It is a new environment with fresh Veeam Installation. (Its a single vSphere-host deployment)

I already tried disabling different Options (vss, indexing cbt...) - nothing helped.

Has anybody an idea what i can try?

Image
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Backup Incredible Slow - Processing at about 200kb/s

Post by Vitaliy S. »

Hello Christoph,

According to the screenshot above, source might be the problem, but I would still ask for full bottleneck stats for this job. Also what kind of VM is it? Can you please try to backup another VM and confirm that you have the same performance issues?

Thank you!
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Backup Incredible Slow - Processing at about 200kb/s

Post by veremin »

As to CBT, disabling it won't increase the processing speed. Instead, it will increase the job time, as without CBT the whole VM image needs to be read by VB&R in order to find changes.

Thanks.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Backup Incredible Slow - Processing at about 200kb/s

Post by foggy »

In addition to full bottleneck stats,I also wonder what transport mode is being utilized to retrieve VM data from the source? (You can select the particular VM in the list to the left and see the transport mode tag next to the proxy server name.)
beistrich
Influencer
Posts: 16
Liked: never
Joined: Feb 04, 2014 8:06 am
Full Name: Christoph
Contact:

Re: Backup Incredible Slow - Processing at about 200kb/s

Post by beistrich »

Yes, i see that the bottleneck is the source - But i don't know why? I have not recognized any performance issues of other applications...
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Backup Incredible Slow - Processing at about 200kb/s

Post by Vitaliy S. »

Can you please clarify other information, so we could assist you further?
beistrich
Influencer
Posts: 16
Liked: never
Joined: Feb 04, 2014 8:06 am
Full Name: Christoph
Contact:

Re: Backup Incredible Slow - Processing at about 200kb/s

Post by beistrich »

THe following shows up for backup Proxy:

Using backup proxy VMware Backup Proxy for disk Hard disk 1 [hotadd]

VMware Backup Proxy is "this Server" with Transport mode set to "Automatic"
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Backup Incredible Slow - Processing at about 200kb/s

Post by foggy »

What if you force the proxy to use network mode?
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Backup Incredible Slow - Processing at about 200kb/s

Post by veremin »

What about CBT information? Do you see the corresponding mark ([CBT]) nearby virtual disk name? Also, as mentioned above, can you try to backup different VM and see whether the speed rates are the same? This should give us a better understanding of the scope of the issue.

Thanks.
beistrich
Influencer
Posts: 16
Liked: never
Joined: Feb 04, 2014 8:06 am
Full Name: Christoph
Contact:

Re: Backup Incredible Slow - Processing at about 200kb/s

Post by beistrich »

Hi!

I have now force "Network" on Transport mode

It doubled the "speed" to 400 kb/s

Image
beistrich
Influencer
Posts: 16
Liked: never
Joined: Feb 04, 2014 8:06 am
Full Name: Christoph
Contact:

Re: Backup Incredible Slow - Processing at about 200kb/s

Post by beistrich »

I also want to mention that it is VM Independent. Also other vms backup at same slow speed.
Could it be the disk array? I doubt that, becaus overall server performance is very good...
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Backup Incredible Slow - Processing at about 200kb/s

Post by foggy »

What kind of storage is it? Bottleneck source means that your production VM storage cannot provide data any faster.
beistrich
Influencer
Posts: 16
Liked: never
Joined: Feb 04, 2014 8:06 am
Full Name: Christoph
Contact:

Re: Backup Incredible Slow - Processing at about 200kb/s

Post by beistrich »

Its a ESXi 5.5 host with local storage (6 disks with RAID 5)
Write Cache enabled on RAID Adapter
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Backup Incredible Slow - Processing at about 200kb/s

Post by Vitaliy S. »

What download speed do you have if you try to download any big file from the datastore via vSphere Client -> Datastore Browser?
beistrich
Influencer
Posts: 16
Liked: never
Joined: Feb 04, 2014 8:06 am
Full Name: Christoph
Contact:

Re: Backup Incredible Slow - Processing at about 200kb/s

Post by beistrich »

WOW, ok, i think i have found the problem

Copying TO the Datastore ist quick, but also not very fast.
Copying FROM the Datastore ist incredible slow! (20 minutes for a 230 MB file)

BUT copying via SMB share at full Gigabit speed!

Anyone knows about issues about using local storage with ESXi 5.5?

This is very strange, because i have already deployed various servers with local storage, not having such issues!
sbbots
Enthusiast
Posts: 96
Liked: 25 times
Joined: Aug 16, 2013 5:44 pm
Full Name: Matt B

Re: Backup Incredible Slow - Processing at about 200kb/s

Post by sbbots »

I am using a similar setup at two remote sites; Single ESXi host, Veeam installed on a VM, backup repository on a Synology NAS (CIFS share).

I noticed slow speeds below 1Mb when going through a Cisco POE Gigabit switch. I attached the NAS directly to an extra NIC on the host server, then passed that NIC to the backup VM using Directpath. All speed issues went away immediately and backup transfer speed now completely saturates the GbE connection.
beistrich
Influencer
Posts: 16
Liked: never
Joined: Feb 04, 2014 8:06 am
Full Name: Christoph
Contact:

Re: Backup Incredible Slow - Processing at about 200kb/s

Post by beistrich »

Yes, would be an option (which i will try)
But i don't think thats really the problem, because the copying from the datastore i run from a VM ON the ESXi host. Does this traffic passes the physical switch?
sbbots
Enthusiast
Posts: 96
Liked: 25 times
Joined: Aug 16, 2013 5:44 pm
Full Name: Matt B

Re: Backup Incredible Slow - Processing at about 200kb/s

Post by sbbots »

I also run Veeam on a VM on the host, so your setup is similar. Once you connect the NAS directly to the host the traffic bypasses the switch.
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Backup Incredible Slow - Processing at about 200kb/s

Post by Vitaliy S. »

Christoph, just to isolate the issue, can you please try to perform a backup of any test VM to the local storage (virtual disk of your backup proxy). What performance do you have in this case?
beistrich
Influencer
Posts: 16
Liked: never
Joined: Feb 04, 2014 8:06 am
Full Name: Christoph
Contact:

Re: Backup Incredible Slow - Processing at about 200kb/s

Post by beistrich »

Hi!
Now i tried a backup to the backup-Proxy. Also slow at 300kbit/s... :(

I found an VMware article adressing such Problem:
http://kb.vmware.com/selfservice/micros ... Id=1006602
I have already configured this...
beistrich
Influencer
Posts: 16
Liked: never
Joined: Feb 04, 2014 8:06 am
Full Name: Christoph
Contact:

Re: Backup Incredible Slow - Processing at about 200kb/s

Post by beistrich »

I now did a test with "esxtop"

I see that the value of CMDS/s and READ/WRITES is significantly different!

This is the result:
Write to Datastore
DEVICE PATH/WORLD/PARTITION DQLEN WQLEN ACTV QUED %USD LOAD CMDS/s READS/s WRITES/s MBREAD/s MBWRTN/s DAVG/cmd KAVG/
naa.600605b006e979901a583d9b1ad8fb71 - 128 - 0 0 0 0.00 108.02 1.00 107.01 0.03 16.27 0.16 0


READ from Datastore
DEVICE PATH/WORLD/PARTITION DQLEN WQLEN ACTV QUED %USD LOAD CMDS/s READS/s WRITES/s MBREAD/s MBWRTN/s DAVG/cmd KAVG/
naa.600605b006e979901a583d9b1ad8fb71 - 128 - 0 0 0 0.00 11.83 1.91 9.92 0.24 0.25 0.21 0
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Backup Incredible Slow - Processing at about 200kb/s

Post by Vitaliy S. »

beistrich wrote:Now i tried a backup to the backup-Proxy. Also slow at 300kbit/s... :(
Was it a hotadd or a network processing mode? Do you have any other ESXi host in the same environment to compare job performance rates?
beistrich
Influencer
Posts: 16
Liked: never
Joined: Feb 04, 2014 8:06 am
Full Name: Christoph
Contact:

Re: Backup Incredible Slow - Processing at about 200kb/s

Post by beistrich »

Hi!
I want to give an update to this...
On the weekend i tried different settings of the raid controller (write back, write back with bbu, write throuch, no read ahead...)
Nothing worked...
I really don't know what to try next, anybody has further ideas?
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Backup Incredible Slow - Processing at about 200kb/s

Post by veremin »

Then, you can open a ticket with our support team and let them confirm your environment, as some of the environmental changes that decrease performance might have slipped out of your attention. Thanks.
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Backup Incredible Slow - Processing at about 200kb/s

Post by Vitaliy S. »

If it is a fresh environment, then using a different RAID level (for example, 10) might also help.
beistrich
Influencer
Posts: 16
Liked: never
Joined: Feb 04, 2014 8:06 am
Full Name: Christoph
Contact:

Re: Backup Incredible Slow - Processing at about 200kb/s

Post by beistrich »

I know its a long time.

but the problem was the Switch/ ESXi Nic configuration! It was set to autodetect, but had 100Mbit non-duplex! - This means VERY slow...
We manually set the link speed and everything worked fine!
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Backup Incredible Slow - Processing at about 200kb/s

Post by foggy »

Christoph, thanks for coming back after a long time and finally updating the thread with the resolution, much appreciated!
Post Reply

Who is online

Users browsing this forum: No registered users and 69 guests