performance (very very slow speed) and 99%

[dgatta]

Hello Sirs.
one customer was experiencing very very slow performance: global processing rate is at 6Mb/sec.
backup proxy transport mode is virtual appliance,
veeam essentlias standard is intaslled on a virtual Windows 2003 X64 machine and repository is a Buffalo therastation, linked via iSCSI to the virtual backup server (backup repository is drive E:\)
jumboframe is enabled in network infrastructure.


Backup job are scheduled daily with and are reverse incremental .
these backup has very slow performance (read at less than10 MB/sec as speed)
Since a couple of day, it does not end correctly because on a machine it remains at 99%.
investigating in statistic for the job one can see in the problematic virtual machine (with two hard disk) that one disk stay for hours at "59,9 GB read at 5 MB/sec" (Hard disk is 60GB)
Job stay in this status for hours and cannot be terminated without restarting the whole backup server

any idea?
Performance are very slow but can be ok because backup end in not-workin time.
but what about the job at 99%?

thanks for your time

[Vitaliy S.]

Hello Davide,

these backup has very slow performance (read at less than10 MB/sec as speed)

What does bottleneck stats show for this backup job?

but what about the job at 99%?

It's hard to say why the job stuck at 99% without seeing full debug log files, so please contact our technical team for assistance with job log review.

Thank you!

[dgatta]

Hello Davide,
What does bottleneck stats show for this backup job?


It's hard to say why the job stuck at 99% without seeing full debug log files, so please contact our technical team for assistance with job log review.



Thank you!

Bottleneck = target
I will try to open a ticket for this, but with another account (the customer one)

Thanks

[veremin]

“Target” bottleneck refers to disk writer component that, in your case, seems to spent most of its time performing I/O to backup files. In order to increase backup performance you might want to switch to forward incremental mode as it’s more storage-friendly than reversed incremental one.

Thanks.

[dgatta]

Vladimir

I have another job that use "forward Incremental" and performance are similar (always lower than 10Mb/sec).
I use this job as nackup on secondary Thecus NAS in another site for redundancy, but when I run full backup it takes too much time (more than a day)

[veremin]

Then, it seems that a given device can’t handle backup load any better. However, the issue with backup jobs being stuck on 99 percent still needs further investigation. So, opening a ticket is, indeed, the best way to go.

Thanks.

[dellock6]

Have you checked vCenter console? It happens many times a job is "stuck" at 99% because there is a snapshot removal that takes quite some time.

Luca.

[J1mbo]

Check target has battery-backed write cache, and that write-back caching is enabled...

[dgatta]

Luca, in vCenter no snapshot removal.
after forcing the job to stop according KB article (http://www.veeam.com/kb1727), if I restart job the VM stay in pending status, so I decided to reset backup and recreate it.

I do not think is a NAS issue because the second NAS is in a network where Switch are 100Mb/s and performances are the same (iSCSI NAS is on 1 GB network with jumboframe enabled, SMB NAS is in 100 Mb network and both perfrom at 6 or 7 MB/sec)

[veremin]

As mentioned, it seems that given devices can't cope any better with the load put on them by the backup activities, since both appliances show approximately the same speed.

As to the stuck jobs issue, then, only logs will certainly tell what is going on wrong in your environment.

Thanks.

[J1mbo]

If I'm following correctly, the storage analysis serves only to confirm that physical disk IO (rather than network) is absolutely the issue here.

[dellock6]

Also, to add to others, I had "horrible" performance issues with Buffalo storage models: is not about the disks you put into, is the internal controller and the software that often cannot cope with the I/O Veeam is able to put onto it, almost regardless the type of backup (reversed or forward). On those kind of storage sadly there are really almost no viable optimizations, being them write-back cache or the like...

And all of this, regardless spot problems like the stuck at 99% or other that could appear.

Luca.