Strange performance issues

[m.kisselyov]

Hi all!

Today i find some problems with my backup job.
I have job, whitch backing up a lot of VMs and two VMs with size around 10TB. Usually this job takes 10-15h with NBD method over 10GB links, but now job still running aroud 30h, and progress near 50%.
Usually Veeam says, that i have bottleneck at Source, but now it says, that my bottleneck - Target. I doesn't change something at my configuration. Proxy and Repository located at one VM. What can i check? Network and storage administrators says, that everything good with hardware and there is no problem at Network, SAN, etc.
My backup speed various from 0 to 30MBS. And this is very slow, because usually it around 100-200 (different on backed up VMs)

I can see

Code: Select all

[27.03.2016 11:56:41] <  2548> stg| Banks cache flush is needed and should be performed soon.
[27.03.2016 11:56:52] <  2548> stg|   Metadata snapshot was saved. [214] banks were written to disk.
[27.03.2016 11:56:59] <  2548> stg|   Metadata snapshot was saved. [214] banks were written to disk.
[27.03.2016 11:56:59] <  2548> stg| Committing metadata. ok.

In my target.log, and at this time my throughput goes to 0. But mb there is no problem.

My proxy and repo configured for 8 parallel tasks, have 8 vCPU and 20GB Memory.

Thanks in advice.

[foggy]

Mikhail, what backup method do you use? What kind of backup repository do you have and were there probably any changes made to the target storage before this behavior started to show up? Bottleneck "Target" means that the target disk writer component spends most of the time writing data to the storage and target storage speed is presenting a bottleneck for the whole data processing conveyor. So I recommend paying attention at the target storage itself and contact technical support for a closer investigation. Thanks.

[m.kisselyov]

Thx for reply.
As i say, i use NBD method. Target repository is Windows Server with attached FC LUN.
I started another task to this target from same source via same proxy, and have 150MBPs speed, and 0% target loading. But concurrent task still running and says that target overloaded.

[foggy]

I was asking about backup method (forward/reversed incremental), not transport mode. Thanks!

[m.kisselyov]

Oh, sorry. Active full backup.

[foggy]

If you see that performance decrease on the active full backup, then something is wrong with the target storage, indeed.

[m.kisselyov]

So why another full backup to the same windows machine and same storage running with full speed idk

[foggy]

Not sure, logs need to be investigated. My initial thought was that some sort of synthetic activity started to take place for this job (as in case with reversed incremental mode), however you're saying it is active full.

[m.kisselyov]

I think, you're right about synthetic activity. I don't notice this before, but this job was re runned after fail of some VMs. Is there reason of low performance?

[foggy]

Logs will tell what operations exactly are being performed and why.

[m.kisselyov]

It's really strange. This job was successful with very low speed. I run it again and now my speed 200+. Magic

[m.kisselyov]

What kind of logs i can examine? If i find something useful, i'll post it here.

[foggy]

This is not a support forum, if you'd like to investigate the behavior, feel free to open a case with technical support and provide full logs for analysis.

[m.kisselyov]

I haven't problems with veeam and haven't reasons to open case.
Thanks in advice.
Veeam the best.

[rhannevig]

Did this happen after installing Veeam B&R V9 UPDATE 1 ?

[m.kisselyov]

Did this happen after installing Veeam B&R V9 UPDATE 1 ?

No, Veeam 9, without Update1