Comprehensive data protection for all workloads
Post Reply
mdangel1
Influencer
Posts: 10
Liked: never
Joined: Dec 17, 2013 7:16 pm
Contact:

VEEAM 7 backup speeds

Post by mdangel1 »

I would like to get an idea on what speeds to expect. We have implemented VEEAM backup & replication 7R2
Our environment is as follows:

2 VMware 5.1 CLuster running on IBM HS23 blades and IBM 3850x5's
Source datastore FB NetApp 3270
Target DR site 10GB pipe between buildings , NetApp 3220 which is where our CIFS reside.
1 VEEAM B&R Server and 2 Proxys per cluster all 2012 4vCPU's 24GB RAM

DEDUP turned on on Source, setting is (Automated-Deduplication happens automatically)

I feel my backups are taking too long. I would like to know what numbers people in the community are getting.
To give you an example, a 1.3TB VM is taking 10 hours to complete a full backup. (not a synthetic full)
ANy thoughts?
Gostev
Chief Product Officer
Posts: 31456
Liked: 6647 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: VEEAM 7 backup speeds

Post by Gostev »

mdangel1 wrote:To give you an example, a 1.3TB VM is taking 10 hours to complete a full backup.
Are you backing up only this VM alone, or are there other VMs being processed in parallel as well?
mdangel1
Influencer
Posts: 10
Liked: never
Joined: Dec 17, 2013 7:16 pm
Contact:

Re: VEEAM 7 backup speeds

Post by mdangel1 »

I usually have 4 to 6 jobs running at a time, but please note I have 4 Proxys and 4 CIFS share to spread out the load. I also have a 10GB link to my CIFS.
Gostev
Chief Product Officer
Posts: 31456
Liked: 6647 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: VEEAM 7 backup speeds

Post by Gostev »

Let's only talk about the job that processes the VM in question then (assuming the 4 CIFS shares all point to different physical devices, otherwise there is no any "load spreading" really). Are there other VMs being processed in parallel by this job when the 1.3TB VM is being backed up? This is easy to see from the job's email report, which lists start and end times for each VM in the job.

Also, what does this backup job indicates as the bottleneck in its log?
mdangel1
Influencer
Posts: 10
Liked: never
Joined: Dec 17, 2013 7:16 pm
Contact:

Re: VEEAM 7 backup speeds

Post by mdangel1 »

4 CIFS point to 2 separate NetApps.
On this last full the bottleneck was the target, but when I look back to all other jobs for this server the bottleneck was the source. When I look at the VEEAM jobs, the bottleneck is Source.

We are not using the email report, but this is the only VM in this job. when I run a report for the job it states the following
backup size 1.3TB
Dedupe 1.4x
Compression 1.0x
start time 3:46AM
End time 1:37PM
Duration 9:51
Gostev
Chief Product Officer
Posts: 31456
Liked: 6647 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: VEEAM 7 backup speeds

Post by Gostev »

You can get the same report by right-clicking the job and selecting Report. See if there are any other VMs being processed in parallel with the VM in question. Be sure to also look up additional VMs in all other jobs which are writing to the same physical device at the same time.

1.3 TB in 10 hours = 38 MB/s definitely not too shabby if you have multiple VMs processed in parallel (up to 8 per NetApp box if you kept concurrency defaults on your backup proxies and backup repositories intact).
mdangel1
Influencer
Posts: 10
Liked: never
Joined: Dec 17, 2013 7:16 pm
Contact:

Re: VEEAM 7 backup speeds

Post by mdangel1 »

That is exactly what I did. Right click the job and choose report. 1 of 1 VM's processed.

No other jobs are running when this server performs a full backup. I am positive. This is a tier 1 server and we only perform fulls on SUndays.
Gostev
Chief Product Officer
Posts: 31456
Liked: 6647 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: VEEAM 7 backup speeds

Post by Gostev »

In that case, considering the bottleneck reported is Target, something is very wrong with that NetApp box, or the storage network. Active full is sequential workload, meaning you should be getting around 100 MB/s even when backing up to a desktop with a single spindle of some 5400 rpm eco hard drive.

In fact, this is going to be a good troubleshooting step: do a test run to any Windows or Linux server you have (add the server as backup repository to Veeam first).
mdangel1
Influencer
Posts: 10
Liked: never
Joined: Dec 17, 2013 7:16 pm
Contact:

Re: VEEAM 7 backup speeds

Post by mdangel1 »

will do!!! I will need to set it up and will get back to you with the results. Thank you!!!!
mdangel1
Influencer
Posts: 10
Liked: never
Joined: Dec 17, 2013 7:16 pm
Contact:

Re: VEEAM 7 backup speeds

Post by mdangel1 »

I set up a windows server as a repository in our source site and the numbers were great, but that is expected.

My environment is 2 buildings with a 10GB link between them. 2 NetApps in each building and We are backing up from one building to another.

I checked with my network team and the 10GB pipe has no contention.

Any thoughts?

BTW, Thank you for all your assistance. I have spent hours with support and they stated our numbers above average, but I feel we can be faster.
tsightler
VP, Product Management
Posts: 6009
Liked: 2842 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: VEEAM 7 backup speeds

Post by tsightler »

Do you have a server acting as a CIFS gateway on the target side or are you writing CIFS directly over that 10Gb link? from the shared proxies? Are the proxies connected to each other via 10Gb as well?
mdangel1
Influencer
Posts: 10
Liked: never
Joined: Dec 17, 2013 7:16 pm
Contact:

Re: VEEAM 7 backup speeds

Post by mdangel1 »

writing directly to CIFS created on NetApps from the shared proxys. yes, proxys connected to each other via 10GB.
I created a new volume on the source NetApp with DEDUP and compression disabled, I created a new VM and restored it to the PROD CIFS and still got the same numbers.
Please note that 99% of my jobs say the bottleneck is the source.
Gostev
Chief Product Officer
Posts: 31456
Liked: 6647 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: VEEAM 7 backup speeds

Post by Gostev »

mdangel1 wrote:I have spent hours with support and they stated our numbers above average
Well, this is certainly above average result for my laptop lab, but ridiculously low for your hardware on 10Gb ethernet :|
mdangel1 wrote:Please note that 99% of my jobs say the bottleneck is the source.
But as far as I understand we are troubleshooting the job that says the bottleneck is the target, so the other 99% does not matter ;)
I would try backing up to the local NetApp box (same site) as the next troubleshooting step.
mdangel1
Influencer
Posts: 10
Liked: never
Joined: Dec 17, 2013 7:16 pm
Contact:

Re: VEEAM 7 backup speeds

Post by mdangel1 »

the same job which run incremental on weekdays and full on Sundays, last night the bottleneck was Source.
Gostev
Chief Product Officer
Posts: 31456
Liked: 6647 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: VEEAM 7 backup speeds

Post by Gostev »

This part is actually easy to explain. First, incremental is random I/O on the source storage, but on the target I/O is still sequential in case of classic incremental backup. Of course, random I/O will always be many times slower. But even more importantly, due to our source-side dedupe and compression, during the incremental runs we read MUCH more data than we transfer and store (with sequential writes!) in the backup file. Hope this make sense.
mdangel1
Influencer
Posts: 10
Liked: never
Joined: Dec 17, 2013 7:16 pm
Contact:

Re: VEEAM 7 backup speeds with NetApp

Post by mdangel1 »

First , let me say thank you for all your help.
Last nights job was an incremental and it actually transferred 139.2GB and it took 1 hour and 8 minutes. Do you feel those are appropriate numbers?
Gostev
Chief Product Officer
Posts: 31456
Liked: 6647 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: VEEAM 7 backup speeds

Post by Gostev »

Multiple that by at least 2-3 times due to the source-side dedupe and compression, and this gives us about 100MB/s non-sequential read speed on source storage. Sounds pretty good to me, I assume the bottleneck reported was Source?
mdangel1
Influencer
Posts: 10
Liked: never
Joined: Dec 17, 2013 7:16 pm
Contact:

Re: VEEAM 7 backup speeds

Post by mdangel1 »

yes, bottleneck Source
Gostev
Chief Product Officer
Posts: 31456
Liked: 6647 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: VEEAM 7 backup speeds

Post by Gostev » 1 person likes this post

Just to follow up this with some information from out internal forums.

This is for the cases when the bottleneck reported is target NetApp CIFS storage. One of our huge customers have been troubleshooting CIFS target performance issues with NetApp support for a very long time, and they have finally came out with the following recommendations which have really improved the performance (use at your own risk):

Code: Select all

options cifs.max_mpx 1124
options cifs.tcp_window_size 2096560
options cifs.neg_buf_size 65535
options raid.mirror_read_plex_pref alternate
Hope this helps!
Post Reply

Who is online

Users browsing this forum: Google [Bot] and 158 guests