Comprehensive data protection for all workloads
Post Reply
JosueM
Expert
Posts: 187
Liked: 12 times
Joined: Sep 01, 2012 2:53 pm
Full Name: Josue Maldonado
Contact:

Job Performance.

Post by JosueM »

Good day everyone,

I'm getting around 30MB/s processing rate for local replication.

Backup and Proxy is a VM (6 processos and 12 GB RAM) with access to both disk cabins, production is a dell MD3200i with 12 disks SAS 15K Rpm and LUN is raid 5, destination SAN is a DELL MD3220i in the same LAN with 12 disks SAS 10K Rpms Lun in raid 5.

2 Esxi Servers in a cluster , DELL R710 with 2 Intel Xeon X5680 @ 3.33Ghz and 144 Gb RAM.

Both ISCSI SAN at 1Gbps each accesed by esxi host with multipath/roundrobing (4 dedicated nics on each server for this traffic)

Veeam version is 7.0.0.690. Wonder if that performance is "normal" for such config?

Thanks in advance.
Vitaliy S.
VP, Product Management
Posts: 27320
Liked: 2778 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Job Performance.

Post by Vitaliy S. »

Hello Josue,

What does bottleneck statistics show for this job?

Thank you!
JosueM
Expert
Posts: 187
Liked: 12 times
Joined: Sep 01, 2012 2:53 pm
Full Name: Josue Maldonado
Contact:

Re: Job Performance.

Post by JosueM »

Hello Vitaliy,

Some jobs bottleneck is target and in others is source.

The veeam backup and proxy vm seems to have low usage of CPU and Memory.
Vitaliy S.
VP, Product Management
Posts: 27320
Liked: 2778 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Job Performance.

Post by Vitaliy S. »

Bottleneck stats should be the same for all VMs assuming that you are replicating them from the same source hosts with the same backup mode selected. Can you please post full bottleneck stats for the replicated VMs? Also is 30 MB/s reported for the full run or incremental job pass?
JosueM
Expert
Posts: 187
Liked: 12 times
Joined: Sep 01, 2012 2:53 pm
Full Name: Josue Maldonado
Contact:

Re: Job Performance.

Post by JosueM »

I noticed that first replication the bottleneck was target on subsequents it is source. Also noticed that for some reason not using hotadd and is failing over to network.

Image

Image
Vitaliy S.
VP, Product Management
Posts: 27320
Liked: 2778 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Job Performance.

Post by Vitaliy S. »

In this case the performance you see is somewhat expected. Could the failover be caused by one of these hotadd limitations? Does your proxy VM have access to all datastores you're backing up VMs from?
JosueM
Expert
Posts: 187
Liked: 12 times
Joined: Sep 01, 2012 2:53 pm
Full Name: Josue Maldonado
Contact:

Re: Job Performance.

Post by JosueM »

The VM has added small disks for each SAN at hardware level, but noticed those disk are not initialized and appears as offline at guest (windows) level. Will initialize them and try again.
Vitaliy S.
VP, Product Management
Posts: 27320
Liked: 2778 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Job Performance.

Post by Vitaliy S. »

You do not need to initialize them, disks should just be visible in the VM (proxy) configuration. Once VMs disks are mounted, our proxy server will start backing them up.
JosueM
Expert
Posts: 187
Liked: 12 times
Joined: Sep 01, 2012 2:53 pm
Full Name: Josue Maldonado
Contact:

Re: Job Performance.

Post by JosueM »

Did that and see that some disk are mount on the proxy vm at windows level storage management but the job still using network instead of hotadd. Will search thru the link above and hope that performance is related to the hotadd feature.
JosueM
Expert
Posts: 187
Liked: 12 times
Joined: Sep 01, 2012 2:53 pm
Full Name: Josue Maldonado
Contact:

Re: Job Performance.

Post by JosueM »

Vitaliy S. wrote:In this case the performance you see is somewhat expected. Could the failover be caused by one of these hotadd limitations? Does your proxy VM have access to all datastores you're backing up VMs from?
Vitaliy from the link above only item I could think a possible cause is the VM has LSI Logic SAS adapter, wich is the default adapter when creating a new VM on vmware for this operating system.

Do you think it should be changed on the veeam VM ?
Vitaliy S.
VP, Product Management
Posts: 27320
Liked: 2778 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Job Performance.

Post by Vitaliy S. »

Hmm...job log would have more information on the reason of the failover. Can you please contact our support team to review your backup job log? BTW, do all other VMs failover to the network mode as well?
JosueM
Expert
Posts: 187
Liked: 12 times
Joined: Sep 01, 2012 2:53 pm
Full Name: Josue Maldonado
Contact:

Re: Job Performance.

Post by JosueM »

Hello Vitaliy,

Already opened a case on support.

Other replications jobs are offsite and they use hotadd properly, some VMs of these jobs are stored on the same SAN as the one that is failing over to network. For these other jobs main bottleneck is target.
Vitaliy S.
VP, Product Management
Posts: 27320
Liked: 2778 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Job Performance.

Post by Vitaliy S. »

JosueM wrote:I'm getting around 30MB/s processing rate for local replication.
JosueM wrote:Other replications jobs are offsite and they use hotadd properly
Now I'm a bit confused...if you're replicating VMs offsite, then it is recommended to have a second Veeam proxy server in the offsite location. Do you have it configured?
JosueM
Expert
Posts: 187
Liked: 12 times
Joined: Sep 01, 2012 2:53 pm
Full Name: Josue Maldonado
Contact:

Re: Job Performance.

Post by JosueM »

I just commented about others job since you asked if there were other jobs also failing over to network. For this case the only job that failover to network is the local replication , the offsite are using hotadd and yes there is a remote proxy for them.

That's why I suspect the local replication low processing rate could be related to the job failing over network instead of hotadd. Weird is I did open a console to the proxy during the job is running and see the source VM disks are mounted on the proxy but for some reason does not use hotadd.
Vitaliy S.
VP, Product Management
Posts: 27320
Liked: 2778 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Job Performance.

Post by Vitaliy S. »

Ok, let's wait for our support team response after reviewing your job log files.
JosueM
Expert
Posts: 187
Liked: 12 times
Joined: Sep 01, 2012 2:53 pm
Full Name: Josue Maldonado
Contact:

Re: Job Performance.

Post by JosueM »

This is weird after 5 days of posting the ticket, the support guy still trying to understand the issue!

I hope they can redirect me with another person, this is not the perception I had about veeam support.
Vitaliy S.
VP, Product Management
Posts: 27320
Liked: 2778 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Job Performance.

Post by Vitaliy S. »

Josue, if you're not satisfied with the level of support provided, you can always ask to escalate your ticket to another engineer.
Gostev
Chief Product Officer
Posts: 31690
Liked: 7201 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Job Performance.

Post by Gostev »

Or you could click "Talk to a Manager" button on the support portal, and this will go straight to the support management DL.

However, please also appreciate that most likely, they are researching an infrastructure issue here for you (and not the issue with the code), as we know for the fact that the actual backup engine can dish out hundreds of MB per second if infrastructure allows. From my experience, most other vendors would simply ask you to hire a consultant to troubleshoot this. However, we always try to find and fix an issue for you, since it involves using our product, and we want you to be successful with your implementation.

Additionally, performance issues are not exactly the highest severity ones, and the SLAs for your case is set as for a low severity case. During "hot" times such as after a new major release, this may directly translate into slower responses from support.

Finally, I'd like to remind that you must include support case ID when posting about technical issues on these forums, as requested when you click New Topic. I see you still did not post it, after 7 days. Without the support case ID, we cannot even check on the current status of your case, or facilitate resolution anyhow. More importantly, you want to include support case ID because moderators will periodically prune all topics about technical issues without case ID included (this is also explained when you click New Topic).
JosueM
Expert
Posts: 187
Liked: 12 times
Joined: Sep 01, 2012 2:53 pm
Full Name: Josue Maldonado
Contact:

Re: Job Performance. (Case # 00446441)

Post by JosueM »

I forgot to post it before. Case # 00446441
JosueM
Expert
Posts: 187
Liked: 12 times
Joined: Sep 01, 2012 2:53 pm
Full Name: Josue Maldonado
Contact:

Re: Job Performance.

Post by JosueM »

Finally support tech did a remote session to verify. The solution was to force proxy to work in network mode since for an unknown reason the hotadd is not working properly.
Post Reply

Who is online

Users browsing this forum: Bing [Bot], Google [Bot] and 31 guests