Discussions specific to the VMware vSphere hypervisor
Post Reply
dgatta
Influencer
Posts: 16
Liked: never
Joined: Oct 24, 2011 1:33 pm
Full Name: Davide Gatta
Contact:

performance (very very slow speed) and 99%

Post by dgatta » Oct 12, 2013 9:07 am

Hello Sirs.
one customer was experiencing very very slow performance: global processing rate is at 6Mb/sec.
backup proxy transport mode is virtual appliance,
veeam essentlias standard is intaslled on a virtual Windows 2003 X64 machine and repository is a Buffalo therastation, linked via iSCSI to the virtual backup server (backup repository is drive E:\)
jumboframe is enabled in network infrastructure.


Backup job are scheduled daily with and are reverse incremental .
these backup has very slow performance (read at less than10 MB/sec as speed)
Since a couple of day, it does not end correctly because on a machine it remains at 99%.
investigating in statistic for the job one can see in the problematic virtual machine (with two hard disk) that one disk stay for hours at "59,9 GB read at 5 MB/sec" (Hard disk is 60GB)
Job stay in this status for hours and cannot be terminated without restarting the whole backup server

any idea?
Performance are very slow but can be ok because backup end in not-workin time.
but what about the job at 99%?

thanks for your time

Vitaliy S.
Product Manager
Posts: 22445
Liked: 1443 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: performance (very very slow speed) and 99%

Post by Vitaliy S. » Oct 12, 2013 2:36 pm

Hello Davide,
dgatta wrote:these backup has very slow performance (read at less than10 MB/sec as speed)
What does bottleneck stats show for this backup job?
dgatta wrote:but what about the job at 99%?
It's hard to say why the job stuck at 99% without seeing full debug log files, so please contact our technical team for assistance with job log review.

Thank you!

dgatta
Influencer
Posts: 16
Liked: never
Joined: Oct 24, 2011 1:33 pm
Full Name: Davide Gatta
Contact:

Re: performance (very very slow speed) and 99%

Post by dgatta » Oct 12, 2013 6:23 pm

Vitaliy S. wrote:Hello Davide,
What does bottleneck stats show for this backup job?


It's hard to say why the job stuck at 99% without seeing full debug log files, so please contact our technical team for assistance with job log review.



Thank you!
Bottleneck = target
I will try to open a ticket for this, but with another account (the customer one)

Thanks

v.eremin
Product Manager
Posts: 16232
Liked: 1325 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: performance (very very slow speed) and 99%

Post by v.eremin » Oct 14, 2013 8:34 am

“Target” bottleneck refers to disk writer component that, in your case, seems to spent most of its time performing I/O to backup files. In order to increase backup performance you might want to switch to forward incremental mode as it’s more storage-friendly than reversed incremental one.

Thanks.

dgatta
Influencer
Posts: 16
Liked: never
Joined: Oct 24, 2011 1:33 pm
Full Name: Davide Gatta
Contact:

Re: performance (very very slow speed) and 99%

Post by dgatta » Oct 14, 2013 1:42 pm

Vladimir

I have another job that use "forward Incremental" and performance are similar (always lower than 10Mb/sec).
I use this job as nackup on secondary Thecus NAS in another site for redundancy, but when I run full backup it takes too much time (more than a day)

v.eremin
Product Manager
Posts: 16232
Liked: 1325 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: performance (very very slow speed) and 99%

Post by v.eremin » Oct 14, 2013 1:53 pm

Then, it seems that a given device can’t handle backup load any better. However, the issue with backup jobs being stuck on 99 percent still needs further investigation. So, opening a ticket is, indeed, the best way to go.

Thanks.

dellock6
Veeam Software
Posts: 5650
Liked: 1586 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: performance (very very slow speed) and 99%

Post by dellock6 » Oct 14, 2013 2:18 pm

Have you checked vCenter console? It happens many times a job is "stuck" at 99% because there is a snapshot removal that takes quite some time.

Luca.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2019
Veeam VMCE #1

J1mbo
Expert
Posts: 261
Liked: 29 times
Joined: May 03, 2011 12:51 pm
Full Name: James Pearce
Contact:

Re: performance (very very slow speed) and 99%

Post by J1mbo » Oct 14, 2013 3:41 pm

Check target has battery-backed write cache, and that write-back caching is enabled...

dgatta
Influencer
Posts: 16
Liked: never
Joined: Oct 24, 2011 1:33 pm
Full Name: Davide Gatta
Contact:

Re: performance (very very slow speed) and 99%

Post by dgatta » Oct 15, 2013 7:20 am

Luca, in vCenter no snapshot removal.
after forcing the job to stop according KB article (http://www.veeam.com/kb1727), if I restart job the VM stay in pending status, so I decided to reset backup and recreate it.

I do not think is a NAS issue because the second NAS is in a network where Switch are 100Mb/s and performances are the same (iSCSI NAS is on 1 GB network with jumboframe enabled, SMB NAS is in 100 Mb network and both perfrom at 6 or 7 MB/sec)

v.eremin
Product Manager
Posts: 16232
Liked: 1325 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: performance (very very slow speed) and 99%

Post by v.eremin » Oct 15, 2013 9:03 am

As mentioned, it seems that given devices can't cope any better with the load put on them by the backup activities, since both appliances show approximately the same speed.

As to the stuck jobs issue, then, only logs will certainly tell what is going on wrong in your environment.

Thanks.

J1mbo
Expert
Posts: 261
Liked: 29 times
Joined: May 03, 2011 12:51 pm
Full Name: James Pearce
Contact:

Re: performance (very very slow speed) and 99%

Post by J1mbo » Oct 15, 2013 9:43 am

If I'm following correctly, the storage analysis serves only to confirm that physical disk IO (rather than network) is absolutely the issue here.

dellock6
Veeam Software
Posts: 5650
Liked: 1586 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: performance (very very slow speed) and 99%

Post by dellock6 » Oct 16, 2013 11:16 am

Also, to add to others, I had "horrible" performance issues with Buffalo storage models: is not about the disks you put into, is the internal controller and the software that often cannot cope with the I/O Veeam is able to put onto it, almost regardless the type of backup (reversed or forward). On those kind of storage sadly there are really almost no viable optimizations, being them write-back cache or the like...

And all of this, regardless spot problems like the stuck at 99% or other that could appear.

Luca.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2019
Veeam VMCE #1

Post Reply

Who is online

Users browsing this forum: No registered users and 7 guests