Host-based backup of VMware vSphere VMs.
Post Reply
Ross_1976
Novice
Posts: 8
Liked: never
Joined: Mar 28, 2014 10:07 am
Full Name: Ross_1976
Contact:

Hotadd speed

Post by Ross_1976 »

Hi all,

earlier this year we implemented Veeam B&R 7.0 R2 on a physical Windows 2008 R2 server (2 CPUs, 16GB RAM, local storage). The virtual infrastructure has a couple of ESXi 5.1 hosts sharing a SAN storage with two dozen VMs on it. All NICs are set to 1Gbs and everything works a treat.

However, we haven't seen very fast backup jobs, about 15-18 MB/s when using Network mode. We then installed a new proxy VM and switched to Hotadd mode... we could see a little improvement but no real major gain, just 28-30 MB/s for most jobs and, even worse, just 15 MB/s for bigger file servers... sometimes the bottleneck is Source, sometimes is Target... can't really understand...

My question is: what is your average speed in Hotadd mode? Which performance should I get in my pure 1Gbs setup??

Thanks!
dellock6
Veeam Software
Posts: 6137
Liked: 1928 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: Hotadd speed

Post by dellock6 »

Hi,
there are few informations about your backup repository to tell you what can be observed. Any more detailed informations about the production san and the physical server where you store your backups?
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
Ross_1976
Novice
Posts: 8
Liked: never
Joined: Mar 28, 2014 10:07 am
Full Name: Ross_1976
Contact:

Re: Hotadd speed

Post by Ross_1976 »

Hi Luca,
the backup server is a small Dell server with the integrated PERC H700 controller with BBU -- 8 SATA disks in a RAID5 config (Write back write policy)
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Hotadd speed

Post by veremin »

Also, it might help if you provide the full bottleneck statistics for both cases - "Source", "Target" bottleneck. The one with the percent distribution. Thanks.
Ross_1976
Novice
Posts: 8
Liked: never
Joined: Mar 28, 2014 10:07 am
Full Name: Ross_1976
Contact:

Re: Hotadd speed

Post by Ross_1976 »

Just an example:
VM #1
last night's backup: 55.0 GB read at 16 MB/s [CBT] --- Busy: Source 27% > Proxy 3% > Network 14% > Target 94%

VM#2
last night's backup: 17.3 GB read at 18 MB/s [CBT] --- Busy: Source 39% > Proxy 22% > Network 17% > Target 92%
dellock6
Veeam Software
Posts: 6137
Liked: 1928 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: Hotadd speed

Post by dellock6 »

Ok, the bottleneck so is the target. What kind of job are you using, reversed incremental maybe?
In general, with 8 sata disks in raid5, 30 MBs in somewhat expected. If it's a reversed incremental, even more is a really good result, but I doubt looking at the numbers that's it a reversed.

Luca.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
Ross_1976
Novice
Posts: 8
Liked: never
Joined: Mar 28, 2014 10:07 am
Full Name: Ross_1976
Contact:

Re: Hotadd speed

Post by Ross_1976 »

That's right, reversed incremental jobs with 14 restore points.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Hotadd speed

Post by foggy »

You can switch to forward incremental then or try RAID10, since it has less I/O penalty.
dellock6
Veeam Software
Posts: 6137
Liked: 1928 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: Hotadd speed

Post by dellock6 »

and just to clarify Ross, 30 MBs in reversed means the storage is running at 90Mbs, that is not bad at all looking at hardware specs.
follow Alexander advices if you need to improve performances.

Luca.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
Ross_1976
Novice
Posts: 8
Liked: never
Joined: Mar 28, 2014 10:07 am
Full Name: Ross_1976
Contact:

Re: Hotadd speed

Post by Ross_1976 »

I've just kicked an Active Full of two small VMs and the processing rate is now at an amazing 105 MB/s..... :o
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Hotadd speed

Post by Gostev »

Even my laptop's hard drive can do 100 MB/s sequential writes... this is exactly why storage performance is measured in IOPS, not in sequential I/O speed.
dellock6
Veeam Software
Posts: 6137
Liked: 1928 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: Hotadd speed

Post by dellock6 »

This is a trick Anton, a single disk has no raid penalty!
A fair comparison would be against a raid0 or jbod configuration in the backup storage, but then there would a "small" problem about data protection :)

Kidding apart, Anton is right, there are several design aspects in sizing a backup repository. Even more than pure IOPS, I would consider IOPS with a certain block size. It's easy to squeeze huge numbers with 4k iops tests, more difficult with 512k for example (that is more or less the size of a LAN target dedupe in Veeam after compression).
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
JoshRountree
Enthusiast
Posts: 34
Liked: 2 times
Joined: Feb 23, 2012 5:49 pm
Full Name: Josh Rountree
Contact:

Re: Hotadd speed

Post by JoshRountree »

I've moved all my backups to forward incremental with Saturday synthetic full, soooo much faster.

If I remember correctly I think Veeam internally does forward incrementals as well. Backing up to my DR site made the biggest different.
Ross_1976
Novice
Posts: 8
Liked: never
Joined: Mar 28, 2014 10:07 am
Full Name: Ross_1976
Contact:

Re: Hotadd speed

Post by Ross_1976 »

I was wondering whether we should switch to Forward incremental -- 14 restore points and one Full backup every 2 weeks... is it feasible?
My biggest concern is a couple of huge file servers (3TB and 5TB respectively) which will require ages (a whole weekend) for a complete Full...
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Hotadd speed

Post by Vitaliy S. »

Given that you have enough free space to store the backup chain then switching to forward incremental mode should give you a better performance rates.
Post Reply

Who is online

Users browsing this forum: Google [Bot], karsten123 and 74 guests