Host-based backup of Microsoft Hyper-V VMs.
Post Reply
ceez
Enthusiast
Posts: 59
Liked: 3 times
Joined: Feb 18, 2014 2:10 pm
Full Name: Cesar
Contact:

If backup goes into production time it slows everything down

Post by ceez »

So I am wondering how I can address the issue of my network bottling up when the jobs run during production hours.

If my backup jobs start too early then those that are still here after 6pm get affected, and if for any reason the jobs run past 7am then everyone in the morning gets affected. I've tried throttling the first job of the evening but that just pushes everything later, if I throttle my other jobs then who knows when the jobs will finish.

What can I look into in order to improve the performance? Currently backing up to a QNAP, our production is a 3-host hyper-v cluster environment in which all VM's are on a PowerVault MD3200i on one big raid 10. We have 1gig switches.

Thanks!
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: If backup goes into production time it slows everything

Post by foggy »

So basically you do not meet your backup window, right? What are the bottleneck stats for your jobs?
nielsengelen
Product Manager
Posts: 5619
Liked: 1177 times
Joined: Jul 15, 2013 11:09 am
Full Name: Niels Engelen
Contact:

Re: If backup goes into production time it slows everything

Post by nielsengelen »

Are you using an onhost or offhost proxy? How is the QNAP configured in terms of amount of disks & size as well as the RAID configuration?
Personal blog: https://foonet.be
GitHub: https://github.com/nielsengelen
ceez
Enthusiast
Posts: 59
Liked: 3 times
Joined: Feb 18, 2014 2:10 pm
Full Name: Cesar
Contact:

Re: If backup goes into production time it slows everything

Post by ceez »

Sorry for late reply but here it goes...
foggy wrote:So basically you do not meet your backup window, right? What are the bottleneck stats for your jobs?
I typically meet that window, but there are times when the backups are disabled one night because of some after-hours work and the next nightly backup might run long because of the extra data that it backs up.

I have 8 jobs that run one after the next starting at 830pm. The bottlenecks change between Source (MD3200i) and Target (QNAP)
vmniels wrote:Are you using an onhost or offhost proxy? How is the QNAP configured in terms of amount of disks & size as well as the RAID configuration?
I am not to keen on the on/offhost proxy setup, but I have a plain vanilla setup of Veeam so I believe it's just the onhost proxy of the server where Veeam is installed which is actually a physical server running Win2012.

The QNAP is a 12.6TB storage pool on a RAID5, it's not used for anything else other than backups and nothing else is writing to it other than Veeam. Again, the connection is a 1GB switch and on the same subnet as our prod network.


Hope this answers your questions.
ceez
Enthusiast
Posts: 59
Liked: 3 times
Joined: Feb 18, 2014 2:10 pm
Full Name: Cesar
Contact:

Re: If backup goes into production time it slows everything

Post by ceez »

Just brushed up on my proxy, and by looking at most of my jobs it's using one of my three cluster nodes as a backup proxy.

It shows "Using backup proxy SERVER1 for C:\ClusterStorage\Volume3\Virtual Machines\myserver\myserver.vhdx.

Then there's another line that does change which shows:

Using source proxy SERVER3 (onhost).

By what I can deduce is that the source proxy server is in which cluster node the vm resides in. So in the example above the vm is running off server3 at the moment.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: If backup goes into production time it slows everything

Post by foggy »

What is the reason of chaining the jobs in your case, instead of starting them all at once?
ceez
Enthusiast
Posts: 59
Liked: 3 times
Joined: Feb 18, 2014 2:10 pm
Full Name: Cesar
Contact:

Re: If backup goes into production time it slows everything

Post by ceez »

I have 8 jobs which in total backup 18 vm's. The thought of running all at once would make me believe of a giant bottleneck and I have it setup to run 2 concurrent backup jobs. So even if they all started at the same time it would only do 2 vm's at a time. Is there another school of thought or maybe much deeper configuration that I have not yet discovered?
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: If backup goes into production time it slows everything

Post by foggy »

Running them all at once would eliminate standby occurred when the job has already completed actual data processing and performs finalizing tasks - processing resources are already available for the next tasks, but need to wait for the next job to start. Provided you properly configure the max number of concurrent tasks, running them all at once will not cause the bottleneck you're talking about, but will allow to use resources in a more effective manner.
ceez
Enthusiast
Posts: 59
Liked: 3 times
Joined: Feb 18, 2014 2:10 pm
Full Name: Cesar
Contact:

Re: If backup goes into production time it slows everything

Post by ceez »

sorry didnt quite understand the first part regarding the processing and tasks, but what I gather is that I can start all my jobs at the same time and it would just do 2 vm's at the same time since that's my limit. How do i know what is the proper number of concurrent tasks?

And how about the proxies? How can I properly configure them? Went to the help center https://helpcenter.veeam.com/backup/hyp ... proxy.html and found this:

By default, source Microsoft Hyper-V hosts perform the role of default backup proxies and let you perform backup and replication in the on-host mode. Resources of such local backup proxy may be sufficient if you use Veeam Backup & Replication to protect VMs in a small virtual environment or for testing purposes. For larger virtual environments, you need to deploy a number of off-host backup proxies to offload VM data processing and transport tasks from source Microsoft Hyper-V hosts.


So at the moment I have two proxies listed under Backup Proxies and both show the host as a physical server (not part of the hyper-v cluster) that I have which runs Veeam:

Hyper-V Off-Host Backup Proxy
VMWare Backup Proxy (we are NOT VMWare so can I delete this one?).

Per the support article the on-host is already being used by default because it's a Hyper-V environment which would be each of my 3 servers in the cluster. If I wanted Off-Host i would need to find either another physical server or another VM to become an Off-Host proxy? Would selecting a VM that is part of the already scheduled backup work as an Off-Host proxy?

Thanks again.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: If backup goes into production time it slows everything

Post by foggy »

ceez wrote:sorry didnt quite understand the first part regarding the processing and tasks, but what I gather is that I can start all my jobs at the same time and it would just do 2 vm's at the same time since that's my limit.
Yes, automatic resource scheduling will make sure there're no gaps in data processing (you currently have due to job chaining) and available resources are utilized effectively.
ceez wrote:How do i know what is the proper number of concurrent tasks?
You can figure that out taking into account system requirements and keeping in mind that in case of on-host backup all data processing tasks are performed by the host itself.
ceez wrote:Per the support article the on-host is already being used by default because it's a Hyper-V environment which would be each of my 3 servers in the cluster. If I wanted Off-Host i would need to find either another physical server or another VM to become an Off-Host proxy? Would selecting a VM that is part of the already scheduled backup work as an Off-Host proxy?
It should be a physical server with the Hyper-V role enabled that is not a member of the cluster (seems you already have it listed under Backup Proxies, so check other requirements to see why it is not used during backup).
ceez
Enthusiast
Posts: 59
Liked: 3 times
Joined: Feb 18, 2014 2:10 pm
Full Name: Cesar
Contact:

Re: If backup goes into production time it slows everything

Post by ceez »

thanks for the info foggy.

In the last link you sent, I see that the off-host for hyper-v has to be the same version of windows. So if my 3 node cluster is running on Hyper-V Server 2012 R2, then does my off-host (non-cluster member) server need to be running the same or can it be running Server 2012 R2?
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: If backup goes into production time it slows everything

Post by veremin »

No need to stick to exact edition (Hyper-V Server) - you can run Windows Server 2012 R2 without any issues. Thanks.
ceez
Enthusiast
Posts: 59
Liked: 3 times
Joined: Feb 18, 2014 2:10 pm
Full Name: Cesar
Contact:

Re: If backup goes into production time it slows everything

Post by ceez »

but it at least has to be R2 level? I can't have an off-host as Windows Server 2012 (non-R2)
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: If backup goes into production time it slows everything

Post by foggy »

Yes, it should have exactly the same platform, while OS edition doesn't matter.
ChrisJ0907
Influencer
Posts: 14
Liked: 1 time
Joined: Jun 16, 2016 10:55 pm
Full Name: Chris James
Contact:

Re: If backup goes into production time it slows everything

Post by ChrisJ0907 »

Apologies for reviving such an old thread, but I'm having the same issue with a similar setup. We only have 2 hosts and more VMs, a QNAP with about 14.5TB available storage setup in RAID 5 with 4 1GB links setup in LACP. The VMs are all on NetApp storage, which is beginning to show its age now I think, yet still very reliable.

I have 6 jobs all chained, so should I just start them all at 7pm for instance? Also, could I use the backup server as an off-host proxy? It's a physical server with 96GB RAM, running the same OS version as the cluster. The QNAP only has 3GB memory, so I guess 2 concurrent tasks is about the limit? I think it's currently set to 4.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: If backup goes into production time it slows everything

Post by foggy »

ChrisJ0907 wrote:I have 6 jobs all chained, so should I just start them all at 7pm for instance?
Yes, this is the recommended approach.
ChrisJ0907 wrote:Also, could I use the backup server as an off-host proxy? It's a physical server with 96GB RAM, running the same OS version as the cluster.
Right, it can be used as an off-host proxy.
ChrisJ0907 wrote:The QNAP only has 3GB memory, so I guess 2 concurrent tasks is about the limit? I think it's currently set to 4.
The memory requirement listed in the user guide for repository is not for the storage itself but the server running the data mover agent - the gateway server in your case.
Post Reply

Who is online

Users browsing this forum: No registered users and 30 guests