If backup goes into production time it slows everything down

Hyper-V specific discussions

If backup goes into production time it slows everything down

Veeam Logoby ceez » Tue May 24, 2016 12:55 pm

So I am wondering how I can address the issue of my network bottling up when the jobs run during production hours.

If my backup jobs start too early then those that are still here after 6pm get affected, and if for any reason the jobs run past 7am then everyone in the morning gets affected. I've tried throttling the first job of the evening but that just pushes everything later, if I throttle my other jobs then who knows when the jobs will finish.

What can I look into in order to improve the performance? Currently backing up to a QNAP, our production is a 3-host hyper-v cluster environment in which all VM's are on a PowerVault MD3200i on one big raid 10. We have 1gig switches.

Thanks!
ceez
Enthusiast
 
Posts: 58
Liked: 3 times
Joined: Tue Feb 18, 2014 2:10 pm
Full Name: Cesar

Re: If backup goes into production time it slows everything

Veeam Logoby foggy » Tue May 24, 2016 5:03 pm

So basically you do not meet your backup window, right? What are the bottleneck stats for your jobs?
foggy
Veeam Software
 
Posts: 14742
Liked: 1079 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: If backup goes into production time it slows everything

Veeam Logoby vmniels » Tue May 24, 2016 5:33 pm

Are you using an onhost or offhost proxy? How is the QNAP configured in terms of amount of disks & size as well as the RAID configuration?
VCP-DCV
Veeam Certified Engineer
http://foonet.be
vmniels
Veeam Software
 
Posts: 1505
Liked: 325 times
Joined: Mon Jul 15, 2013 11:09 am
Full Name: Niels Engelen

Re: If backup goes into production time it slows everything

Veeam Logoby ceez » Mon Jul 18, 2016 9:09 pm

Sorry for late reply but here it goes...

foggy wrote:So basically you do not meet your backup window, right? What are the bottleneck stats for your jobs?


I typically meet that window, but there are times when the backups are disabled one night because of some after-hours work and the next nightly backup might run long because of the extra data that it backs up.

I have 8 jobs that run one after the next starting at 830pm. The bottlenecks change between Source (MD3200i) and Target (QNAP)

vmniels wrote:Are you using an onhost or offhost proxy? How is the QNAP configured in terms of amount of disks & size as well as the RAID configuration?


I am not to keen on the on/offhost proxy setup, but I have a plain vanilla setup of Veeam so I believe it's just the onhost proxy of the server where Veeam is installed which is actually a physical server running Win2012.

The QNAP is a 12.6TB storage pool on a RAID5, it's not used for anything else other than backups and nothing else is writing to it other than Veeam. Again, the connection is a 1GB switch and on the same subnet as our prod network.


Hope this answers your questions.
ceez
Enthusiast
 
Posts: 58
Liked: 3 times
Joined: Tue Feb 18, 2014 2:10 pm
Full Name: Cesar

Re: If backup goes into production time it slows everything

Veeam Logoby ceez » Mon Jul 18, 2016 9:48 pm

Just brushed up on my proxy, and by looking at most of my jobs it's using one of my three cluster nodes as a backup proxy.

It shows "Using backup proxy SERVER1 for C:\ClusterStorage\Volume3\Virtual Machines\myserver\myserver.vhdx.

Then there's another line that does change which shows:

Using source proxy SERVER3 (onhost).

By what I can deduce is that the source proxy server is in which cluster node the vm resides in. So in the example above the vm is running off server3 at the moment.
ceez
Enthusiast
 
Posts: 58
Liked: 3 times
Joined: Tue Feb 18, 2014 2:10 pm
Full Name: Cesar

Re: If backup goes into production time it slows everything

Veeam Logoby foggy » Tue Jul 19, 2016 2:06 pm

What is the reason of chaining the jobs in your case, instead of starting them all at once?
foggy
Veeam Software
 
Posts: 14742
Liked: 1079 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: If backup goes into production time it slows everything

Veeam Logoby ceez » Tue Jul 19, 2016 2:16 pm

I have 8 jobs which in total backup 18 vm's. The thought of running all at once would make me believe of a giant bottleneck and I have it setup to run 2 concurrent backup jobs. So even if they all started at the same time it would only do 2 vm's at a time. Is there another school of thought or maybe much deeper configuration that I have not yet discovered?
ceez
Enthusiast
 
Posts: 58
Liked: 3 times
Joined: Tue Feb 18, 2014 2:10 pm
Full Name: Cesar

Re: If backup goes into production time it slows everything

Veeam Logoby foggy » Tue Jul 19, 2016 2:33 pm

Running them all at once would eliminate standby occurred when the job has already completed actual data processing and performs finalizing tasks - processing resources are already available for the next tasks, but need to wait for the next job to start. Provided you properly configure the max number of concurrent tasks, running them all at once will not cause the bottleneck you're talking about, but will allow to use resources in a more effective manner.
foggy
Veeam Software
 
Posts: 14742
Liked: 1079 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: If backup goes into production time it slows everything

Veeam Logoby ceez » Tue Jul 19, 2016 7:17 pm

sorry didnt quite understand the first part regarding the processing and tasks, but what I gather is that I can start all my jobs at the same time and it would just do 2 vm's at the same time since that's my limit. How do i know what is the proper number of concurrent tasks?

And how about the proxies? How can I properly configure them? Went to the help center https://helpcenter.veeam.com/backup/hyp ... proxy.html and found this:

By default, source Microsoft Hyper-V hosts perform the role of default backup proxies and let you perform backup and replication in the on-host mode. Resources of such local backup proxy may be sufficient if you use Veeam Backup & Replication to protect VMs in a small virtual environment or for testing purposes. For larger virtual environments, you need to deploy a number of off-host backup proxies to offload VM data processing and transport tasks from source Microsoft Hyper-V hosts.


So at the moment I have two proxies listed under Backup Proxies and both show the host as a physical server (not part of the hyper-v cluster) that I have which runs Veeam:

Hyper-V Off-Host Backup Proxy
VMWare Backup Proxy (we are NOT VMWare so can I delete this one?).

Per the support article the on-host is already being used by default because it's a Hyper-V environment which would be each of my 3 servers in the cluster. If I wanted Off-Host i would need to find either another physical server or another VM to become an Off-Host proxy? Would selecting a VM that is part of the already scheduled backup work as an Off-Host proxy?

Thanks again.
ceez
Enthusiast
 
Posts: 58
Liked: 3 times
Joined: Tue Feb 18, 2014 2:10 pm
Full Name: Cesar

Re: If backup goes into production time it slows everything

Veeam Logoby foggy » Wed Jul 20, 2016 11:06 am

ceez wrote:sorry didnt quite understand the first part regarding the processing and tasks, but what I gather is that I can start all my jobs at the same time and it would just do 2 vm's at the same time since that's my limit.

Yes, automatic resource scheduling will make sure there're no gaps in data processing (you currently have due to job chaining) and available resources are utilized effectively.

ceez wrote:How do i know what is the proper number of concurrent tasks?

You can figure that out taking into account system requirements and keeping in mind that in case of on-host backup all data processing tasks are performed by the host itself.

ceez wrote:Per the support article the on-host is already being used by default because it's a Hyper-V environment which would be each of my 3 servers in the cluster. If I wanted Off-Host i would need to find either another physical server or another VM to become an Off-Host proxy? Would selecting a VM that is part of the already scheduled backup work as an Off-Host proxy?

It should be a physical server with the Hyper-V role enabled that is not a member of the cluster (seems you already have it listed under Backup Proxies, so check other requirements to see why it is not used during backup).
foggy
Veeam Software
 
Posts: 14742
Liked: 1079 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: If backup goes into production time it slows everything

Veeam Logoby ceez » Thu Jul 21, 2016 1:15 pm

thanks for the info foggy.

In the last link you sent, I see that the off-host for hyper-v has to be the same version of windows. So if my 3 node cluster is running on Hyper-V Server 2012 R2, then does my off-host (non-cluster member) server need to be running the same or can it be running Server 2012 R2?
ceez
Enthusiast
 
Posts: 58
Liked: 3 times
Joined: Tue Feb 18, 2014 2:10 pm
Full Name: Cesar

Re: If backup goes into production time it slows everything

Veeam Logoby v.Eremin » Thu Jul 21, 2016 1:26 pm

No need to stick to exact edition (Hyper-V Server) - you can run Windows Server 2012 R2 without any issues. Thanks.
v.Eremin
Veeam Software
 
Posts: 13266
Liked: 968 times
Joined: Fri Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin

Re: If backup goes into production time it slows everything

Veeam Logoby ceez » Thu Jul 21, 2016 4:24 pm

but it at least has to be R2 level? I can't have an off-host as Windows Server 2012 (non-R2)
ceez
Enthusiast
 
Posts: 58
Liked: 3 times
Joined: Tue Feb 18, 2014 2:10 pm
Full Name: Cesar

Re: If backup goes into production time it slows everything

Veeam Logoby foggy » Thu Jul 21, 2016 4:27 pm

Yes, it should have exactly the same platform, while OS edition doesn't matter.
foggy
Veeam Software
 
Posts: 14742
Liked: 1079 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson


Return to Microsoft Hyper-V



Who is online

Users browsing this forum: Google [Bot] and 4 guests