Slow backups

[andre.lang]

Hi guys.

I have the following setup

1x Server DL380g6 2xQuads 24Gig Ram which runs 5x VM's
1x Server which is used to replicate all VM's onto nightly

3x External hard drives

Now what we do is have 3x different jobs which backup onto each external hard drive, EG Job 1 destination is E, Job 2 is F and Job 3 is G.
Now if the device is not connected, it is told not to retry.

The issue is, I feel all of our backups are going slow. Veeam is installed on our main Vm, which is 110gb, and runs exchange and our CRM. a incremental backup usually takes 2-3 hours for this machine per job. The process rate is around the 9 mb for both backup and replication jobs.

I feel this is a little slow. Is it because veeam is installed on the main large server, and if so, where should it be installed on!!

Let me know.
Thanks
Andre

[Alexey D.]

Hello Andre,

Please provide more details on your setup. Which backup mode are you using? What is the configuration of that "main VM" (number of vCPUs, memory)? What is the CPU utilization during backup cycle?

[Vitaliy S.]

Veeam is installed on our main Vm, which is 110gb, and runs exchange and our CRM.

Yes, more details would definitely help here. But first of all I would recommend moving backup server from Exchange/CRM servers, as it seems like you backup servers might be starving on CPU resources.

By the way, how do you backup your Exchange now?

[andre.lang]

Hi Guys, Its got access to 2x Quads and 16gb memory.

Backup Mode is direct appliance, Seems that I can get better performance via network for some reason :S

We backup exchange by just doing a backup of the VM.

Let me know if this is not right way of going about it.

CPU is fairly high, which is a issue, as its taking away from other services.
I will have a look at changing our veeam install to another server. May be a better solution.

cheers
Abdre

[Vitaliy S.]

Andre,

In order to have maximum performance with Virtual Appliance mode, you need to configure a backup VM with 4 vCPUs at least. As to Exchange best practises, please make sure you've enabled "Application-aware image processing".

On top of that, what backup mode are you using? Choosing forward incremental mode should give you better performance rates due to the fact that with forward incrementals you'll have three times less I/O, and thus much lower IOPS during incremental runs.

Thanks.

[andre.lang]

Andre,

In order to have maximum performance with Virtual Appliance mode, you need to configure a backup VM with 4 vCPUs at least. As to Exchange best practises, please make sure you've enabled "Application-aware image processing".

On top of that, what backup mode are you using? Choosing forward incremental mode should give you better performance rates due to the fact that with forward incrementals you'll have three times less I/O, and thus much lower IOPS during incremental runs.

Thanks.

Hi there.

Yes it has access to all 8 CPU's
And also yes forward incremental is the method i'm using.

I will make sure to enable application aware and see how we go

thanks
Andre

[andre.lang]

Sorry. All 4x Vcores.

Our License doesn't allow 8.

Cheers

[andre.lang]

I have noticed all other VM's

Backup mode: HOTADD with changed block tracking

Seems to work no problem, however for main VM where Veeam is installed on

Backup mode: HOTADD without changed block tracking

When I change the mode to network,Im getting a 100mb transfer rate... so obviously its MUCH faster.

Anyideas as to why? And what I can do. The issue with this is its using up alot of network trafic on our switch

[Vitaliy S.]

Well...Virtualized Veeam Backup server cannot be used to backup, replicate or copy itself in virtual appliance mode as per our Release Notes. Jobs configured with VA mode will automatically failover to Network processing mode.

That said, I'm not sure why you see difference in performance rates by switching from VA to Network, as in fact you're using the same mode all the time. As to 100Mb transfer rate, seems like changed block tracking does work for you. Please be aware that "with changed block tracking" message appears only while processing virtual disks and not other VM files, which seems to be the case in your situation.

If you want to avoid having network traffic from the source to backup server, you need to deploy a separate VM (4 CPUs) for a backup server and use VA mode. Thanks.

[andre.lang]

Thanks guys. Shall do.

Yes your right, its my bad.

Also, Its using the Vnetwork now, so its not going out over the switch

Cheers

[mnario]

[merged into existing discussion]

I have some questions to ask the users out there or admins.

- I am trying to get the best throughput for my backup w/o losing data for recovery. I know the Veeam has the ability to backup incremental or Reverse incremental. What I use right now is a USB External Hard Drive (2) connected to a Physical Server.
- I trying to backup about 8 VMs out of 50 or so that we have and it is taking over 20 hours right now to do this. I would like to know what are some of the things that I can look for to see what could be slowing down my backup, which are usually about 1-4mbps in transfer speed others go over 30mbps.

[Vitaliy S.]

Hello Mike,

First of all, you may want to choose any local drive on your physical backup server as a destination target for your backups. Secondly, please provide us with more details regarding your setup (backup mode/type, CBT usage, CPU load on the backup server etc.)

Thanks.

[mark_firestone]

[merged into existing discussion]

Hello,

We are running Veeam on esx 4. The veeam backup server is a VM with 8gb memory and 4 cores.

We used to get thoughput on backups of windows servers of between 150 - 200 mb/sec. Recently, this has dropped to about 30 mb/sec or less, sometimes as low as 6 mb/sec.

Also, memory use seems to have increased. The server used to have 6gb assigned, but seemed to be very low on memory so I increased it to 8gb, but this doesn't seem to have made any difference. By way of comparison, we also backup a Linux VM, which just backed up at over 400 mb/sec, and is consistently fast.

I'm at my wits end. Any troubleshooting tips / advice?

Signed,

Frustrated in Earby.

[Shogan]

Just out of interest which backup mode are you using for VMs? Network, SAN or otherwise?

[Vitaliy S.]

Hello Mark,

Could you please provide us with more details regarding your setup? Besides, be aware that it is expected to have various processing rate for different VMs. Please have a look at this topic for more info: Very different backup speed on VM's

Thanks.

[mark_firestone]

Just out of interest which backup mode are you using for VMs? Network, SAN or otherwise?

Sorry, I forgot to day. These are being backed up in "virtual appliance" mode.

[mark_firestone]

Hello Mark,

Could you please provide us with more details regarding your setup? Besides, be aware that it is expected to have various processing rate for different VMs. Please have a look at this topic for more info: Very different backup speed on VM's

Thanks.

I'd be happy to. What other details would be helpful?

I understand that you can get different speeds with different VMs... but performance has really fallen, from nearly 200mb/sec to 30mb/sec or less for the same VMs...

[Vitaliy S.]

Mark, please provide us with the backup mode you've chosen (Reversed/Forward Incremental), destination storage type, connection to the the target, VMs (roles) that are affected by this performance degradation etc.

[theflakes]

We had similar issues and moving our backup server OS from 2008 64bit to 2008 R2 64 bit fixed the problem for us. We would see the backups start at normal speed and then degrade over time.

[mooreka777]

[merged]

Hello All,

my backups take about 30 hours to finish and I need some help what best practice is to get it closer to 12.

VMware 4.1 esx host
Veeam Server is a virtual machine in the cluster

Backup Server - dl380g5 with 10T of disk storage on an fiber connected AX4-5
3.5T of vm's backing up over share

Has anyone run into this problem and what the did to correct it? There are about 80vm's and I plan to grow to about 250 VMs by the end of the year. The physical esx server has 4 1G interfaces etherchanneled and the backup server has two 1gb interfaces etherchanneled. When I check the switch utilization it is 400Mb to 500Mb. I don't have jumbo frames turned on, but can if anyone has seen that dramatically reducing backup times.

Any direction or things I can try I would love to hear them.

Kindly,
Kelly

[Vitaliy S.]

Hello Kelly,

Could you please describe your job setup? Are you running a Virtual Appliance mode? What is the backup mode you use? Besides, what about CPU usage on the backup server?

Thanks.

[mooreka777]

sorry for the delayed response. I have tried just about every way possible to get backups under 24 hours and cant do it.

180vm's
tried virtual server and virtual appliance
physical server and direct san access

both took over 24 hours for the full backup.

I now split up the vcenter into three configs
prod linux
prod windows
development

I want to build three veeam servers and backup all three with different vm's. Do I need three databases or can two run as "agents" and let the main veeam server manage the db?

I am backing up to an HP dedup appliance with a 10G interface, so I know the bottleneck isn't the appliance.

Any direction or help would be much appreciated.

Thanks,
kelly

[Gostev]

Bottleneck could very well be the dedup appliance... having 10Gb interface means nothing, what matters is actual storage speed. Most dedupe appliances are generally pretty slow targets because of heavy data processing they perform on incoming data. Try backing up to raw disks as a start, and see if performance improves...

Generally speaking, you should be able to do 3.5TB with a single server and 3 parallel jobs (according to your VM groups) just fine. You are way below numbers from where I would look at deploying second Veeam Backup server. You are not planning to perform full backup every day, right?

[jdpal_ccs]

Andre,

In order to have maximum performance with Virtual Appliance mode, you need to configure a backup VM with 4 vCPUs at least. As to Exchange best practises, please make sure you've enabled "Application-aware image processing".

On top of that, what backup mode are you using? Choosing forward incremental mode should give you better performance rates due to the fact that with forward incrementals you'll have three times less I/O, and thus much lower IOPS during incremental runs.

Thanks.

Hi Vitaliy,

I am curious how using forward incremental mode uses 3x less I/O (than reverse incremental, I assume)? It seems like this is a very useful thing to know and could really affect the design of Veeam backup jobs. Could you elaborate more on this?

Thanks!

[Vitaliy S.]

Yes, of course, see Anton's reply over here: Guidance on best Backup Method based upon %% Change?

[daveneil]

Same problem as mark

We are running Veeam on esx 4.1 The veeam backup server is a VM with 12gb memory and 3 cores.

We used to get thoughput on backups of windows server 2008 R2 of between 150 - 200 mb/sec. this has dropped to 10 mb/sec.

Also, memory use seems to have increased. The server used to have 6gb assigned, but seemed to be very low on memory so I increased it to 8gb, but this doesn't seem to have made any difference.

Whe are using reverse incrimental.

Did you solve this mark ?

[foggy]

David, this is quite an old thread and Mark didn't answer the last Vitaliy's question about his setup, so I doubt that he can help you. Please describe you environment (what are your source and target storages, connection type, what backup mode and transport mode do you use, whether all VMs are affected or not, what is CPU usage on the backup server) and also try to remember what kind of changes have you performed to your environment recently. Btw, what VBR version do you use?