Discussions specific to the VMware vSphere hypervisor
Post Reply
Ross_1976
Novice
Posts: 8
Liked: never
Joined: Mar 28, 2014 10:07 am
Full Name: Ross_1976
Contact:

Hotadd speed

Post by Ross_1976 » Mar 28, 2014 10:31 am

Hi all,

earlier this year we implemented Veeam B&R 7.0 R2 on a physical Windows 2008 R2 server (2 CPUs, 16GB RAM, local storage). The virtual infrastructure has a couple of ESXi 5.1 hosts sharing a SAN storage with two dozen VMs on it. All NICs are set to 1Gbs and everything works a treat.

However, we haven't seen very fast backup jobs, about 15-18 MB/s when using Network mode. We then installed a new proxy VM and switched to Hotadd mode... we could see a little improvement but no real major gain, just 28-30 MB/s for most jobs and, even worse, just 15 MB/s for bigger file servers... sometimes the bottleneck is Source, sometimes is Target... can't really understand...

My question is: what is your average speed in Hotadd mode? Which performance should I get in my pure 1Gbs setup??

Thanks!

dellock6
Veeam Software
Posts: 5732
Liked: 1622 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: Hotadd speed

Post by dellock6 » Mar 28, 2014 10:38 am

Hi,
there are few informations about your backup repository to tell you what can be observed. Any more detailed informations about the production san and the physical server where you store your backups?
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2019
Veeam VMCE #1

Ross_1976
Novice
Posts: 8
Liked: never
Joined: Mar 28, 2014 10:07 am
Full Name: Ross_1976
Contact:

Re: Hotadd speed

Post by Ross_1976 » Mar 28, 2014 10:48 am

Hi Luca,
the backup server is a small Dell server with the integrated PERC H700 controller with BBU -- 8 SATA disks in a RAID5 config (Write back write policy)

veremin
Product Manager
Posts: 16884
Liked: 1430 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Hotadd speed

Post by veremin » Mar 28, 2014 10:55 am

Also, it might help if you provide the full bottleneck statistics for both cases - "Source", "Target" bottleneck. The one with the percent distribution. Thanks.

Ross_1976
Novice
Posts: 8
Liked: never
Joined: Mar 28, 2014 10:07 am
Full Name: Ross_1976
Contact:

Re: Hotadd speed

Post by Ross_1976 » Mar 28, 2014 11:13 am

Just an example:
VM #1
last night's backup: 55.0 GB read at 16 MB/s [CBT] --- Busy: Source 27% > Proxy 3% > Network 14% > Target 94%

VM#2
last night's backup: 17.3 GB read at 18 MB/s [CBT] --- Busy: Source 39% > Proxy 22% > Network 17% > Target 92%

dellock6
Veeam Software
Posts: 5732
Liked: 1622 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: Hotadd speed

Post by dellock6 » Mar 28, 2014 11:32 am

Ok, the bottleneck so is the target. What kind of job are you using, reversed incremental maybe?
In general, with 8 sata disks in raid5, 30 MBs in somewhat expected. If it's a reversed incremental, even more is a really good result, but I doubt looking at the numbers that's it a reversed.

Luca.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2019
Veeam VMCE #1

Ross_1976
Novice
Posts: 8
Liked: never
Joined: Mar 28, 2014 10:07 am
Full Name: Ross_1976
Contact:

Re: Hotadd speed

Post by Ross_1976 » Mar 28, 2014 11:36 am

That's right, reversed incremental jobs with 14 restore points.

foggy
Veeam Software
Posts: 18245
Liked: 1558 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Hotadd speed

Post by foggy » Mar 28, 2014 11:37 am

You can switch to forward incremental then or try RAID10, since it has less I/O penalty.

dellock6
Veeam Software
Posts: 5732
Liked: 1622 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: Hotadd speed

Post by dellock6 » Mar 28, 2014 11:54 am

and just to clarify Ross, 30 MBs in reversed means the storage is running at 90Mbs, that is not bad at all looking at hardware specs.
follow Alexander advices if you need to improve performances.

Luca.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2019
Veeam VMCE #1

Ross_1976
Novice
Posts: 8
Liked: never
Joined: Mar 28, 2014 10:07 am
Full Name: Ross_1976
Contact:

Re: Hotadd speed

Post by Ross_1976 » Mar 28, 2014 1:52 pm

I've just kicked an Active Full of two small VMs and the processing rate is now at an amazing 105 MB/s..... :o

Gostev
SVP, Product Management
Posts: 24787
Liked: 3520 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Hotadd speed

Post by Gostev » Mar 30, 2014 7:29 pm

Even my laptop's hard drive can do 100 MB/s sequential writes... this is exactly why storage performance is measured in IOPS, not in sequential I/O speed.

dellock6
Veeam Software
Posts: 5732
Liked: 1622 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: Hotadd speed

Post by dellock6 » Mar 31, 2014 3:56 pm

This is a trick Anton, a single disk has no raid penalty!
A fair comparison would be against a raid0 or jbod configuration in the backup storage, but then there would a "small" problem about data protection :)

Kidding apart, Anton is right, there are several design aspects in sizing a backup repository. Even more than pure IOPS, I would consider IOPS with a certain block size. It's easy to squeeze huge numbers with 4k iops tests, more difficult with 512k for example (that is more or less the size of a LAN target dedupe in Veeam after compression).
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2019
Veeam VMCE #1

JoshRountree
Enthusiast
Posts: 34
Liked: 1 time
Joined: Feb 23, 2012 5:49 pm
Full Name: Josh Rountree
Contact:

Re: Hotadd speed

Post by JoshRountree » Mar 31, 2014 8:57 pm

I've moved all my backups to forward incremental with Saturday synthetic full, soooo much faster.

If I remember correctly I think Veeam internally does forward incrementals as well. Backing up to my DR site made the biggest different.

Ross_1976
Novice
Posts: 8
Liked: never
Joined: Mar 28, 2014 10:07 am
Full Name: Ross_1976
Contact:

Re: Hotadd speed

Post by Ross_1976 » Apr 07, 2014 9:11 pm

I was wondering whether we should switch to Forward incremental -- 14 restore points and one Full backup every 2 weeks... is it feasible?
My biggest concern is a couple of huge file servers (3TB and 5TB respectively) which will require ages (a whole weekend) for a complete Full...

Vitaliy S.
Product Manager
Posts: 22973
Liked: 1555 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Hotadd speed

Post by Vitaliy S. » Apr 08, 2014 8:02 am

Given that you have enough free space to store the backup chain then switching to forward incremental mode should give you a better performance rates.

Post Reply

Who is online

Users browsing this forum: Bing [Bot] and 34 guests