If backup goes into production time it slows everything down

[ceez]

So I am wondering how I can address the issue of my network bottling up when the jobs run during production hours.

If my backup jobs start too early then those that are still here after 6pm get affected, and if for any reason the jobs run past 7am then everyone in the morning gets affected. I've tried throttling the first job of the evening but that just pushes everything later, if I throttle my other jobs then who knows when the jobs will finish.

What can I look into in order to improve the performance? Currently backing up to a QNAP, our production is a 3-host hyper-v cluster environment in which all VM's are on a PowerVault MD3200i on one big raid 10. We have 1gig switches.

Thanks!

[foggy]

So basically you do not meet your backup window, right? What are the bottleneck stats for your jobs?

[nielsengelen]

Are you using an onhost or offhost proxy? How is the QNAP configured in terms of amount of disks & size as well as the RAID configuration?

[ceez]

Sorry for late reply but here it goes...

So basically you do not meet your backup window, right? What are the bottleneck stats for your jobs?

I typically meet that window, but there are times when the backups are disabled one night because of some after-hours work and the next nightly backup might run long because of the extra data that it backs up.

I have 8 jobs that run one after the next starting at 830pm. The bottlenecks change between Source (MD3200i) and Target (QNAP)

Are you using an onhost or offhost proxy? How is the QNAP configured in terms of amount of disks & size as well as the RAID configuration?

I am not to keen on the on/offhost proxy setup, but I have a plain vanilla setup of Veeam so I believe it's just the onhost proxy of the server where Veeam is installed which is actually a physical server running Win2012.

The QNAP is a 12.6TB storage pool on a RAID5, it's not used for anything else other than backups and nothing else is writing to it other than Veeam. Again, the connection is a 1GB switch and on the same subnet as our prod network.


Hope this answers your questions.

[ceez]

Just brushed up on my proxy, and by looking at most of my jobs it's using one of my three cluster nodes as a backup proxy.

It shows "Using backup proxy SERVER1 for C:\ClusterStorage\Volume3\Virtual Machines\myserver\myserver.vhdx.

Then there's another line that does change which shows:

Using source proxy SERVER3 (onhost).

By what I can deduce is that the source proxy server is in which cluster node the vm resides in. So in the example above the vm is running off server3 at the moment.

[foggy]

What is the reason of chaining the jobs in your case, instead of starting them all at once?

[ceez]

I have 8 jobs which in total backup 18 vm's. The thought of running all at once would make me believe of a giant bottleneck and I have it setup to run 2 concurrent backup jobs. So even if they all started at the same time it would only do 2 vm's at a time. Is there another school of thought or maybe much deeper configuration that I have not yet discovered?

[foggy]

Running them all at once would eliminate standby occurred when the job has already completed actual data processing and performs finalizing tasks - processing resources are already available for the next tasks, but need to wait for the next job to start. Provided you properly configure the max number of concurrent tasks, running them all at once will not cause the bottleneck you're talking about, but will allow to use resources in a more effective manner.

[ceez]

sorry didnt quite understand the first part regarding the processing and tasks, but what I gather is that I can start all my jobs at the same time and it would just do 2 vm's at the same time since that's my limit. How do i know what is the proper number of concurrent tasks?

And how about the proxies? How can I properly configure them? Went to the help center https://helpcenter.veeam.com/backup/hyp ... proxy.html and found this:

By default, source Microsoft Hyper-V hosts perform the role of default backup proxies and let you perform backup and replication in the on-host mode. Resources of such local backup proxy may be sufficient if you use Veeam Backup & Replication to protect VMs in a small virtual environment or for testing purposes. For larger virtual environments, you need to deploy a number of off-host backup proxies to offload VM data processing and transport tasks from source Microsoft Hyper-V hosts.


So at the moment I have two proxies listed under Backup Proxies and both show the host as a physical server (not part of the hyper-v cluster) that I have which runs Veeam:

Hyper-V Off-Host Backup Proxy
VMWare Backup Proxy (we are NOT VMWare so can I delete this one?).

Per the support article the on-host is already being used by default because it's a Hyper-V environment which would be each of my 3 servers in the cluster. If I wanted Off-Host i would need to find either another physical server or another VM to become an Off-Host proxy? Would selecting a VM that is part of the already scheduled backup work as an Off-Host proxy?

Thanks again.

[foggy]

sorry didnt quite understand the first part regarding the processing and tasks, but what I gather is that I can start all my jobs at the same time and it would just do 2 vm's at the same time since that's my limit.

Yes, automatic resource scheduling will make sure there're no gaps in data processing (you currently have due to job chaining) and available resources are utilized effectively.

How do i know what is the proper number of concurrent tasks?

You can figure that out taking into account system requirements and keeping in mind that in case of on-host backup all data processing tasks are performed by the host itself.

Per the support article the on-host is already being used by default because it's a Hyper-V environment which would be each of my 3 servers in the cluster. If I wanted Off-Host i would need to find either another physical server or another VM to become an Off-Host proxy? Would selecting a VM that is part of the already scheduled backup work as an Off-Host proxy?

It should be a physical server with the Hyper-V role enabled that is not a member of the cluster (seems you already have it listed under Backup Proxies, so check other requirements to see why it is not used during backup).

[ceez]

thanks for the info foggy.

In the last link you sent, I see that the off-host for hyper-v has to be the same version of windows. So if my 3 node cluster is running on Hyper-V Server 2012 R2, then does my off-host (non-cluster member) server need to be running the same or can it be running Server 2012 R2?

[veremin]

No need to stick to exact edition (Hyper-V Server) - you can run Windows Server 2012 R2 without any issues. Thanks.

[ceez]

but it at least has to be R2 level? I can't have an off-host as Windows Server 2012 (non-R2)

[foggy]

Yes, it should have exactly the same platform, while OS edition doesn't matter.

[ChrisJ0907]

Apologies for reviving such an old thread, but I'm having the same issue with a similar setup. We only have 2 hosts and more VMs, a QNAP with about 14.5TB available storage setup in RAID 5 with 4 1GB links setup in LACP. The VMs are all on NetApp storage, which is beginning to show its age now I think, yet still very reliable.

I have 6 jobs all chained, so should I just start them all at 7pm for instance? Also, could I use the backup server as an off-host proxy? It's a physical server with 96GB RAM, running the same OS version as the cluster. The QNAP only has 3GB memory, so I guess 2 concurrent tasks is about the limit? I think it's currently set to 4.

[foggy]

I have 6 jobs all chained, so should I just start them all at 7pm for instance?

Yes, this is the recommended approach.

Also, could I use the backup server as an off-host proxy? It's a physical server with 96GB RAM, running the same OS version as the cluster.

Right, it can be used as an off-host proxy.

The QNAP only has 3GB memory, so I guess 2 concurrent tasks is about the limit? I think it's currently set to 4.

The memory requirement listed in the user guide for repository is not for the storage itself but the server running the data mover agent - the gateway server in your case.