Comprehensive data protection for all workloads
Post Reply
andre.lang
Novice
Posts: 9
Liked: never
Joined: May 13, 2011 12:25 am
Full Name: Andre Lang

Slow backups

Post by andre.lang » May 13, 2011 12:32 am

Hi guys.

I have the following setup

1x Server DL380g6 2xQuads 24Gig Ram which runs 5x VM's
1x Server which is used to replicate all VM's onto nightly

3x External hard drives

Now what we do is have 3x different jobs which backup onto each external hard drive, EG Job 1 destination is E, Job 2 is F and Job 3 is G.
Now if the device is not connected, it is told not to retry.

The issue is, I feel all of our backups are going slow. Veeam is installed on our main Vm, which is 110gb, and runs exchange and our CRM. a incremental backup usually takes 2-3 hours for this machine per job. The process rate is around the 9 mb for both backup and replication jobs.

I feel this is a little slow. Is it because veeam is installed on the main large server, and if so, where should it be installed on!!

Let me know.
Thanks
Andre

Alexey D.

Re: Slow backups

Post by Alexey D. » May 13, 2011 7:41 am

Hello Andre,

Please provide more details on your setup. Which backup mode are you using? What is the configuration of that "main VM" (number of vCPUs, memory)? What is the CPU utilization during backup cycle?

Vitaliy S.
Product Manager
Posts: 23001
Liked: 1557 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Slow backups

Post by Vitaliy S. » May 13, 2011 8:18 am

andre.lang wrote:Veeam is installed on our main Vm, which is 110gb, and runs exchange and our CRM.
Yes, more details would definitely help here. But first of all I would recommend moving backup server from Exchange/CRM servers, as it seems like you backup servers might be starving on CPU resources.

By the way, how do you backup your Exchange now?

andre.lang
Novice
Posts: 9
Liked: never
Joined: May 13, 2011 12:25 am
Full Name: Andre Lang

Re: Slow backups

Post by andre.lang » May 13, 2011 10:55 pm

Hi Guys, Its got access to 2x Quads and 16gb memory.

Backup Mode is direct appliance, Seems that I can get better performance via network for some reason :S

We backup exchange by just doing a backup of the VM.

Let me know if this is not right way of going about it.

CPU is fairly high, which is a issue, as its taking away from other services.
I will have a look at changing our veeam install to another server. May be a better solution.

cheers
Abdre

Vitaliy S.
Product Manager
Posts: 23001
Liked: 1557 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Slow backups

Post by Vitaliy S. » May 15, 2011 10:05 am

Andre,

In order to have maximum performance with Virtual Appliance mode, you need to configure a backup VM with 4 vCPUs at least. As to Exchange best practises, please make sure you've enabled "Application-aware image processing".

On top of that, what backup mode are you using? Choosing forward incremental mode should give you better performance rates due to the fact that with forward incrementals you'll have three times less I/O, and thus much lower IOPS during incremental runs.

Thanks.

andre.lang
Novice
Posts: 9
Liked: never
Joined: May 13, 2011 12:25 am
Full Name: Andre Lang

Re: Slow backups

Post by andre.lang » May 16, 2011 12:45 am

Vitaliy S. wrote:Andre,

In order to have maximum performance with Virtual Appliance mode, you need to configure a backup VM with 4 vCPUs at least. As to Exchange best practises, please make sure you've enabled "Application-aware image processing".

On top of that, what backup mode are you using? Choosing forward incremental mode should give you better performance rates due to the fact that with forward incrementals you'll have three times less I/O, and thus much lower IOPS during incremental runs.

Thanks.
Hi there.

Yes it has access to all 8 CPU's
And also yes forward incremental is the method i'm using.

I will make sure to enable application aware and see how we go :)

thanks
Andre

andre.lang
Novice
Posts: 9
Liked: never
Joined: May 13, 2011 12:25 am
Full Name: Andre Lang

Re: Slow backups

Post by andre.lang » May 16, 2011 1:47 am

Sorry. All 4x Vcores.

Our License doesn't allow 8.

Cheers

andre.lang
Novice
Posts: 9
Liked: never
Joined: May 13, 2011 12:25 am
Full Name: Andre Lang

Re: Slow backups

Post by andre.lang » May 16, 2011 3:30 am

I have noticed all other VM's

Backup mode: HOTADD with changed block tracking

Seems to work no problem, however for main VM where Veeam is installed on

Backup mode: HOTADD without changed block tracking

When I change the mode to network,Im getting a 100mb transfer rate... so obviously its MUCH faster.

Anyideas as to why? And what I can do. The issue with this is its using up alot of network trafic on our switch

Vitaliy S.
Product Manager
Posts: 23001
Liked: 1557 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Slow backups

Post by Vitaliy S. » May 16, 2011 8:07 am

Well...Virtualized Veeam Backup server cannot be used to backup, replicate or copy itself in virtual appliance mode as per our Release Notes. Jobs configured with VA mode will automatically failover to Network processing mode.

That said, I'm not sure why you see difference in performance rates by switching from VA to Network, as in fact you're using the same mode all the time. As to 100Mb transfer rate, seems like changed block tracking does work for you. Please be aware that "with changed block tracking" message appears only while processing virtual disks and not other VM files, which seems to be the case in your situation.

If you want to avoid having network traffic from the source to backup server, you need to deploy a separate VM (4 CPUs) for a backup server and use VA mode. Thanks.

andre.lang
Novice
Posts: 9
Liked: never
Joined: May 13, 2011 12:25 am
Full Name: Andre Lang

Re: Slow backups

Post by andre.lang » May 16, 2011 9:53 pm

Thanks guys. Shall do.

Yes your right, its my bad.

Also, Its using the Vnetwork now, so its not going out over the switch :)

Cheers

mnario
Novice
Posts: 7
Liked: never
Joined: May 19, 2011 6:27 pm
Full Name: Mike Nario
Contact:

Trying out the Veeam Backup and Recovery.

Post by mnario » May 19, 2011 6:32 pm

[merged into existing discussion]

I have some questions to ask the users out there or admins.

- I am trying to get the best throughput for my backup w/o losing data for recovery. I know the Veeam has the ability to backup incremental or Reverse incremental. What I use right now is a USB External Hard Drive (2) connected to a Physical Server.
- I trying to backup about 8 VMs out of 50 or so that we have and it is taking over 20 hours right now to do this. I would like to know what are some of the things that I can look for to see what could be slowing down my backup, which are usually about 1-4mbps in transfer speed others go over 30mbps.

Vitaliy S.
Product Manager
Posts: 23001
Liked: 1557 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Slow backups

Post by Vitaliy S. » May 19, 2011 9:36 pm

Hello Mike,

First of all, you may want to choose any local drive on your physical backup server as a destination target for your backups. Secondly, please provide us with more details regarding your setup (backup mode/type, CBT usage, CPU load on the backup server etc.)

Thanks.

mark_firestone
Novice
Posts: 5
Liked: never
Joined: Jan 19, 2011 11:17 am
Full Name: Mark Firestone
Contact:

very slow backup performance

Post by mark_firestone » Jun 07, 2011 8:26 am

[merged into existing discussion]

Hello,

We are running Veeam on esx 4. The veeam backup server is a VM with 8gb memory and 4 cores.

We used to get thoughput on backups of windows servers of between 150 - 200 mb/sec. Recently, this has dropped to about 30 mb/sec or less, sometimes as low as 6 mb/sec.

Also, memory use seems to have increased. The server used to have 6gb assigned, but seemed to be very low on memory so I increased it to 8gb, but this doesn't seem to have made any difference. By way of comparison, we also backup a Linux VM, which just backed up at over 400 mb/sec, and is consistently fast.

I'm at my wits end. Any troubleshooting tips / advice?

Signed,

Frustrated in Earby.

Shogan

Re: very slow backup performance

Post by Shogan » Jun 07, 2011 8:58 am

Just out of interest which backup mode are you using for VMs? Network, SAN or otherwise?

Vitaliy S.
Product Manager
Posts: 23001
Liked: 1557 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Slow backups

Post by Vitaliy S. » Jun 07, 2011 9:10 am

Hello Mark,

Could you please provide us with more details regarding your setup? Besides, be aware that it is expected to have various processing rate for different VMs. Please have a look at this topic for more info: Very different backup speed on VM's

Thanks.

mark_firestone
Novice
Posts: 5
Liked: never
Joined: Jan 19, 2011 11:17 am
Full Name: Mark Firestone
Contact:

Re: very slow backup performance

Post by mark_firestone » Jun 07, 2011 9:52 am

Shogan wrote:Just out of interest which backup mode are you using for VMs? Network, SAN or otherwise?
Sorry, I forgot to day. These are being backed up in "virtual appliance" mode.

mark_firestone
Novice
Posts: 5
Liked: never
Joined: Jan 19, 2011 11:17 am
Full Name: Mark Firestone
Contact:

Re: Slow backups

Post by mark_firestone » Jun 07, 2011 9:54 am

Vitaliy S. wrote:Hello Mark,

Could you please provide us with more details regarding your setup? Besides, be aware that it is expected to have various processing rate for different VMs. Please have a look at this topic for more info: Very different backup speed on VM's

Thanks.
I'd be happy to. What other details would be helpful?

I understand that you can get different speeds with different VMs... but performance has really fallen, from nearly 200mb/sec to 30mb/sec or less for the same VMs...

Vitaliy S.
Product Manager
Posts: 23001
Liked: 1557 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Slow backups

Post by Vitaliy S. » Jun 07, 2011 2:16 pm

Mark, please provide us with the backup mode you've chosen (Reversed/Forward Incremental), destination storage type, connection to the the target, VMs (roles) that are affected by this performance degradation etc.

theflakes
Enthusiast
Posts: 33
Liked: never
Joined: Jun 09, 2010 11:16 pm
Full Name: Brian Kellogg
Contact:

Re: Serious Issue with 5.0.2 Restore Point Retention!!

Post by theflakes » Jun 07, 2011 7:27 pm

We had similar issues and moving our backup server OS from 2008 64bit to 2008 R2 64 bit fixed the problem for us. We would see the backups start at normal speed and then degrade over time.

mooreka777
Influencer
Posts: 18
Liked: 1 time
Joined: Jan 20, 2011 1:05 pm
Full Name: K Moore
Contact:

Veeam taking longer than 24 hours for nightly backups

Post by mooreka777 » Jun 09, 2011 5:39 pm

[merged]

Hello All,

my backups take about 30 hours to finish and I need some help what best practice is to get it closer to 12.

VMware 4.1 esx host
Veeam Server is a virtual machine in the cluster

Backup Server - dl380g5 with 10T of disk storage on an fiber connected AX4-5
3.5T of vm's backing up over share

Has anyone run into this problem and what the did to correct it? There are about 80vm's and I plan to grow to about 250 VMs by the end of the year. The physical esx server has 4 1G interfaces etherchanneled and the backup server has two 1gb interfaces etherchanneled. When I check the switch utilization it is 400Mb to 500Mb. I don't have jumbo frames turned on, but can if anyone has seen that dramatically reducing backup times.

Any direction or things I can try I would love to hear them.

Kindly,
Kelly

Vitaliy S.
Product Manager
Posts: 23001
Liked: 1557 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Veeam taking longer than 24 hours for nightly backups

Post by Vitaliy S. » Jun 09, 2011 9:33 pm

Hello Kelly,

Could you please describe your job setup? Are you running a Virtual Appliance mode? What is the backup mode you use? Besides, what about CPU usage on the backup server?

Thanks.

mooreka777
Influencer
Posts: 18
Liked: 1 time
Joined: Jan 20, 2011 1:05 pm
Full Name: K Moore
Contact:

Re: Slow backups

Post by mooreka777 » Jun 28, 2011 7:39 pm

sorry for the delayed response. I have tried just about every way possible to get backups under 24 hours and cant do it.

180vm's
tried virtual server and virtual appliance
physical server and direct san access

both took over 24 hours for the full backup.

I now split up the vcenter into three configs
prod linux
prod windows
development

I want to build three veeam servers and backup all three with different vm's. Do I need three databases or can two run as "agents" and let the main veeam server manage the db?

I am backing up to an HP dedup appliance with a 10G interface, so I know the bottleneck isn't the appliance.

Any direction or help would be much appreciated.

Thanks,
kelly

Gostev
SVP, Product Management
Posts: 24804
Liked: 3566 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Slow backups

Post by Gostev » Jun 28, 2011 8:55 pm

Bottleneck could very well be the dedup appliance... having 10Gb interface means nothing, what matters is actual storage speed. Most dedupe appliances are generally pretty slow targets because of heavy data processing they perform on incoming data. Try backing up to raw disks as a start, and see if performance improves...

Generally speaking, you should be able to do 3.5TB with a single server and 3 parallel jobs (according to your VM groups) just fine. You are way below numbers from where I would look at deploying second Veeam Backup server. You are not planning to perform full backup every day, right?

jdpal_ccs
Novice
Posts: 3
Liked: never
Joined: Jul 11, 2011 1:29 pm
Contact:

Re: Slow backups

Post by jdpal_ccs » Jul 11, 2011 3:42 pm

Vitaliy S. wrote:Andre,

In order to have maximum performance with Virtual Appliance mode, you need to configure a backup VM with 4 vCPUs at least. As to Exchange best practises, please make sure you've enabled "Application-aware image processing".

On top of that, what backup mode are you using? Choosing forward incremental mode should give you better performance rates due to the fact that with forward incrementals you'll have three times less I/O, and thus much lower IOPS during incremental runs.

Thanks.
Hi Vitaliy,

I am curious how using forward incremental mode uses 3x less I/O (than reverse incremental, I assume)? It seems like this is a very useful thing to know and could really affect the design of Veeam backup jobs. Could you elaborate more on this?

Thanks!

Vitaliy S.
Product Manager
Posts: 23001
Liked: 1557 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Slow backups

Post by Vitaliy S. » Jul 11, 2011 4:07 pm

Yes, of course, see Anton's reply over here: Guidance on best Backup Method based upon %% Change?

daveneil
Lurker
Posts: 1
Liked: never
Joined: Feb 21, 2012 2:14 pm
Full Name: David Neil
Contact:

Re: Slow backups

Post by daveneil » Feb 21, 2012 2:25 pm

Same problem as mark

We are running Veeam on esx 4.1 The veeam backup server is a VM with 12gb memory and 3 cores.

We used to get thoughput on backups of windows server 2008 R2 of between 150 - 200 mb/sec. this has dropped to 10 mb/sec.

Also, memory use seems to have increased. The server used to have 6gb assigned, but seemed to be very low on memory so I increased it to 8gb, but this doesn't seem to have made any difference.

Whe are using reverse incrimental.

Did you solve this mark ?

foggy
Veeam Software
Posts: 18278
Liked: 1564 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Slow backups

Post by foggy » Feb 21, 2012 4:45 pm

David, this is quite an old thread and Mark didn't answer the last Vitaliy's question about his setup, so I doubt that he can help you. Please describe you environment (what are your source and target storages, connection type, what backup mode and transport mode do you use, whether all VMs are affected or not, what is CPU usage on the backup server) and also try to remember what kind of changes have you performed to your environment recently. Btw, what VBR version do you use?

Post Reply

Who is online

Users browsing this forum: Baidu [Spider], Bing [Bot] and 10 guests