-
- Influencer
- Posts: 16
- Liked: never
- Joined: Feb 04, 2014 8:06 am
- Full Name: Christoph
- Contact:
Backup Incredible Slow - Processing at about 200kb/s
Hallo!
I set up Veeam Backup&Replication as usual.
- Veeam is installed on a virtual server (and is used as Proxy)
- Backup repository is a NAS
- VSS, indexing and cbt enabled
Never had any issues before. But now i have enormous performance problems - running backups with about 200kb/s. It is a new environment with fresh Veeam Installation. (Its a single vSphere-host deployment)
I already tried disabling different Options (vss, indexing cbt...) - nothing helped.
Has anybody an idea what i can try?
I set up Veeam Backup&Replication as usual.
- Veeam is installed on a virtual server (and is used as Proxy)
- Backup repository is a NAS
- VSS, indexing and cbt enabled
Never had any issues before. But now i have enormous performance problems - running backups with about 200kb/s. It is a new environment with fresh Veeam Installation. (Its a single vSphere-host deployment)
I already tried disabling different Options (vss, indexing cbt...) - nothing helped.
Has anybody an idea what i can try?
-
- VP, Product Management
- Posts: 27377
- Liked: 2800 times
- Joined: Mar 30, 2009 9:13 am
- Full Name: Vitaliy Safarov
- Contact:
Re: Backup Incredible Slow - Processing at about 200kb/s
Hello Christoph,
According to the screenshot above, source might be the problem, but I would still ask for full bottleneck stats for this job. Also what kind of VM is it? Can you please try to backup another VM and confirm that you have the same performance issues?
Thank you!
According to the screenshot above, source might be the problem, but I would still ask for full bottleneck stats for this job. Also what kind of VM is it? Can you please try to backup another VM and confirm that you have the same performance issues?
Thank you!
-
- Product Manager
- Posts: 20415
- Liked: 2302 times
- Joined: Oct 26, 2012 3:28 pm
- Full Name: Vladimir Eremin
- Contact:
Re: Backup Incredible Slow - Processing at about 200kb/s
As to CBT, disabling it won't increase the processing speed. Instead, it will increase the job time, as without CBT the whole VM image needs to be read by VB&R in order to find changes.
Thanks.
Thanks.
-
- Veeam Software
- Posts: 21139
- Liked: 2141 times
- Joined: Jul 11, 2011 10:22 am
- Full Name: Alexander Fogelson
- Contact:
Re: Backup Incredible Slow - Processing at about 200kb/s
In addition to full bottleneck stats,I also wonder what transport mode is being utilized to retrieve VM data from the source? (You can select the particular VM in the list to the left and see the transport mode tag next to the proxy server name.)
-
- Influencer
- Posts: 16
- Liked: never
- Joined: Feb 04, 2014 8:06 am
- Full Name: Christoph
- Contact:
Re: Backup Incredible Slow - Processing at about 200kb/s
Yes, i see that the bottleneck is the source - But i don't know why? I have not recognized any performance issues of other applications...
-
- VP, Product Management
- Posts: 27377
- Liked: 2800 times
- Joined: Mar 30, 2009 9:13 am
- Full Name: Vitaliy Safarov
- Contact:
Re: Backup Incredible Slow - Processing at about 200kb/s
Can you please clarify other information, so we could assist you further?
-
- Influencer
- Posts: 16
- Liked: never
- Joined: Feb 04, 2014 8:06 am
- Full Name: Christoph
- Contact:
Re: Backup Incredible Slow - Processing at about 200kb/s
THe following shows up for backup Proxy:
Using backup proxy VMware Backup Proxy for disk Hard disk 1 [hotadd]
VMware Backup Proxy is "this Server" with Transport mode set to "Automatic"
Using backup proxy VMware Backup Proxy for disk Hard disk 1 [hotadd]
VMware Backup Proxy is "this Server" with Transport mode set to "Automatic"
-
- Veeam Software
- Posts: 21139
- Liked: 2141 times
- Joined: Jul 11, 2011 10:22 am
- Full Name: Alexander Fogelson
- Contact:
Re: Backup Incredible Slow - Processing at about 200kb/s
What if you force the proxy to use network mode?
-
- Product Manager
- Posts: 20415
- Liked: 2302 times
- Joined: Oct 26, 2012 3:28 pm
- Full Name: Vladimir Eremin
- Contact:
Re: Backup Incredible Slow - Processing at about 200kb/s
What about CBT information? Do you see the corresponding mark ([CBT]) nearby virtual disk name? Also, as mentioned above, can you try to backup different VM and see whether the speed rates are the same? This should give us a better understanding of the scope of the issue.
Thanks.
Thanks.
-
- Influencer
- Posts: 16
- Liked: never
- Joined: Feb 04, 2014 8:06 am
- Full Name: Christoph
- Contact:
Re: Backup Incredible Slow - Processing at about 200kb/s
Hi!
I have now force "Network" on Transport mode
It doubled the "speed" to 400 kb/s
I have now force "Network" on Transport mode
It doubled the "speed" to 400 kb/s
-
- Influencer
- Posts: 16
- Liked: never
- Joined: Feb 04, 2014 8:06 am
- Full Name: Christoph
- Contact:
Re: Backup Incredible Slow - Processing at about 200kb/s
I also want to mention that it is VM Independent. Also other vms backup at same slow speed.
Could it be the disk array? I doubt that, becaus overall server performance is very good...
Could it be the disk array? I doubt that, becaus overall server performance is very good...
-
- Veeam Software
- Posts: 21139
- Liked: 2141 times
- Joined: Jul 11, 2011 10:22 am
- Full Name: Alexander Fogelson
- Contact:
Re: Backup Incredible Slow - Processing at about 200kb/s
What kind of storage is it? Bottleneck source means that your production VM storage cannot provide data any faster.
-
- Influencer
- Posts: 16
- Liked: never
- Joined: Feb 04, 2014 8:06 am
- Full Name: Christoph
- Contact:
Re: Backup Incredible Slow - Processing at about 200kb/s
Its a ESXi 5.5 host with local storage (6 disks with RAID 5)
Write Cache enabled on RAID Adapter
Write Cache enabled on RAID Adapter
-
- VP, Product Management
- Posts: 27377
- Liked: 2800 times
- Joined: Mar 30, 2009 9:13 am
- Full Name: Vitaliy Safarov
- Contact:
Re: Backup Incredible Slow - Processing at about 200kb/s
What download speed do you have if you try to download any big file from the datastore via vSphere Client -> Datastore Browser?
-
- Influencer
- Posts: 16
- Liked: never
- Joined: Feb 04, 2014 8:06 am
- Full Name: Christoph
- Contact:
Re: Backup Incredible Slow - Processing at about 200kb/s
WOW, ok, i think i have found the problem
Copying TO the Datastore ist quick, but also not very fast.
Copying FROM the Datastore ist incredible slow! (20 minutes for a 230 MB file)
BUT copying via SMB share at full Gigabit speed!
Anyone knows about issues about using local storage with ESXi 5.5?
This is very strange, because i have already deployed various servers with local storage, not having such issues!
Copying TO the Datastore ist quick, but also not very fast.
Copying FROM the Datastore ist incredible slow! (20 minutes for a 230 MB file)
BUT copying via SMB share at full Gigabit speed!
Anyone knows about issues about using local storage with ESXi 5.5?
This is very strange, because i have already deployed various servers with local storage, not having such issues!
-
- Enthusiast
- Posts: 96
- Liked: 25 times
- Joined: Aug 16, 2013 5:44 pm
- Full Name: Matt B
Re: Backup Incredible Slow - Processing at about 200kb/s
I am using a similar setup at two remote sites; Single ESXi host, Veeam installed on a VM, backup repository on a Synology NAS (CIFS share).
I noticed slow speeds below 1Mb when going through a Cisco POE Gigabit switch. I attached the NAS directly to an extra NIC on the host server, then passed that NIC to the backup VM using Directpath. All speed issues went away immediately and backup transfer speed now completely saturates the GbE connection.
I noticed slow speeds below 1Mb when going through a Cisco POE Gigabit switch. I attached the NAS directly to an extra NIC on the host server, then passed that NIC to the backup VM using Directpath. All speed issues went away immediately and backup transfer speed now completely saturates the GbE connection.
-
- Influencer
- Posts: 16
- Liked: never
- Joined: Feb 04, 2014 8:06 am
- Full Name: Christoph
- Contact:
Re: Backup Incredible Slow - Processing at about 200kb/s
Yes, would be an option (which i will try)
But i don't think thats really the problem, because the copying from the datastore i run from a VM ON the ESXi host. Does this traffic passes the physical switch?
But i don't think thats really the problem, because the copying from the datastore i run from a VM ON the ESXi host. Does this traffic passes the physical switch?
-
- Enthusiast
- Posts: 96
- Liked: 25 times
- Joined: Aug 16, 2013 5:44 pm
- Full Name: Matt B
Re: Backup Incredible Slow - Processing at about 200kb/s
I also run Veeam on a VM on the host, so your setup is similar. Once you connect the NAS directly to the host the traffic bypasses the switch.
-
- VP, Product Management
- Posts: 27377
- Liked: 2800 times
- Joined: Mar 30, 2009 9:13 am
- Full Name: Vitaliy Safarov
- Contact:
Re: Backup Incredible Slow - Processing at about 200kb/s
Christoph, just to isolate the issue, can you please try to perform a backup of any test VM to the local storage (virtual disk of your backup proxy). What performance do you have in this case?
-
- Influencer
- Posts: 16
- Liked: never
- Joined: Feb 04, 2014 8:06 am
- Full Name: Christoph
- Contact:
Re: Backup Incredible Slow - Processing at about 200kb/s
Hi!
Now i tried a backup to the backup-Proxy. Also slow at 300kbit/s...
I found an VMware article adressing such Problem:
http://kb.vmware.com/selfservice/micros ... Id=1006602
I have already configured this...
Now i tried a backup to the backup-Proxy. Also slow at 300kbit/s...
I found an VMware article adressing such Problem:
http://kb.vmware.com/selfservice/micros ... Id=1006602
I have already configured this...
-
- Influencer
- Posts: 16
- Liked: never
- Joined: Feb 04, 2014 8:06 am
- Full Name: Christoph
- Contact:
Re: Backup Incredible Slow - Processing at about 200kb/s
I now did a test with "esxtop"
I see that the value of CMDS/s and READ/WRITES is significantly different!
This is the result:
Write to Datastore
DEVICE PATH/WORLD/PARTITION DQLEN WQLEN ACTV QUED %USD LOAD CMDS/s READS/s WRITES/s MBREAD/s MBWRTN/s DAVG/cmd KAVG/
naa.600605b006e979901a583d9b1ad8fb71 - 128 - 0 0 0 0.00 108.02 1.00 107.01 0.03 16.27 0.16 0
READ from Datastore
DEVICE PATH/WORLD/PARTITION DQLEN WQLEN ACTV QUED %USD LOAD CMDS/s READS/s WRITES/s MBREAD/s MBWRTN/s DAVG/cmd KAVG/
naa.600605b006e979901a583d9b1ad8fb71 - 128 - 0 0 0 0.00 11.83 1.91 9.92 0.24 0.25 0.21 0
I see that the value of CMDS/s and READ/WRITES is significantly different!
This is the result:
Write to Datastore
DEVICE PATH/WORLD/PARTITION DQLEN WQLEN ACTV QUED %USD LOAD CMDS/s READS/s WRITES/s MBREAD/s MBWRTN/s DAVG/cmd KAVG/
naa.600605b006e979901a583d9b1ad8fb71 - 128 - 0 0 0 0.00 108.02 1.00 107.01 0.03 16.27 0.16 0
READ from Datastore
DEVICE PATH/WORLD/PARTITION DQLEN WQLEN ACTV QUED %USD LOAD CMDS/s READS/s WRITES/s MBREAD/s MBWRTN/s DAVG/cmd KAVG/
naa.600605b006e979901a583d9b1ad8fb71 - 128 - 0 0 0 0.00 11.83 1.91 9.92 0.24 0.25 0.21 0
-
- VP, Product Management
- Posts: 27377
- Liked: 2800 times
- Joined: Mar 30, 2009 9:13 am
- Full Name: Vitaliy Safarov
- Contact:
Re: Backup Incredible Slow - Processing at about 200kb/s
Was it a hotadd or a network processing mode? Do you have any other ESXi host in the same environment to compare job performance rates?beistrich wrote:Now i tried a backup to the backup-Proxy. Also slow at 300kbit/s...
-
- Influencer
- Posts: 16
- Liked: never
- Joined: Feb 04, 2014 8:06 am
- Full Name: Christoph
- Contact:
Re: Backup Incredible Slow - Processing at about 200kb/s
Hi!
I want to give an update to this...
On the weekend i tried different settings of the raid controller (write back, write back with bbu, write throuch, no read ahead...)
Nothing worked...
I really don't know what to try next, anybody has further ideas?
I want to give an update to this...
On the weekend i tried different settings of the raid controller (write back, write back with bbu, write throuch, no read ahead...)
Nothing worked...
I really don't know what to try next, anybody has further ideas?
-
- Product Manager
- Posts: 20415
- Liked: 2302 times
- Joined: Oct 26, 2012 3:28 pm
- Full Name: Vladimir Eremin
- Contact:
Re: Backup Incredible Slow - Processing at about 200kb/s
Then, you can open a ticket with our support team and let them confirm your environment, as some of the environmental changes that decrease performance might have slipped out of your attention. Thanks.
-
- VP, Product Management
- Posts: 27377
- Liked: 2800 times
- Joined: Mar 30, 2009 9:13 am
- Full Name: Vitaliy Safarov
- Contact:
Re: Backup Incredible Slow - Processing at about 200kb/s
If it is a fresh environment, then using a different RAID level (for example, 10) might also help.
-
- Influencer
- Posts: 16
- Liked: never
- Joined: Feb 04, 2014 8:06 am
- Full Name: Christoph
- Contact:
Re: Backup Incredible Slow - Processing at about 200kb/s
I know its a long time.
but the problem was the Switch/ ESXi Nic configuration! It was set to autodetect, but had 100Mbit non-duplex! - This means VERY slow...
We manually set the link speed and everything worked fine!
but the problem was the Switch/ ESXi Nic configuration! It was set to autodetect, but had 100Mbit non-duplex! - This means VERY slow...
We manually set the link speed and everything worked fine!
-
- Veeam Software
- Posts: 21139
- Liked: 2141 times
- Joined: Jul 11, 2011 10:22 am
- Full Name: Alexander Fogelson
- Contact:
Re: Backup Incredible Slow - Processing at about 200kb/s
Christoph, thanks for coming back after a long time and finally updating the thread with the resolution, much appreciated!
Who is online
Users browsing this forum: Bing [Bot], Semrush [Bot] and 13 guests