Host-based backup of VMware vSphere VMs.
Post Reply
umenz
Enthusiast
Posts: 52
Liked: never
Joined: Jun 15, 2011 9:24 am
Full Name: Ulrich Menz
Contact:

Whats your Backup to NAS reversed Incremental Perfomance ?

Post by umenz »

Hi All,

I would not discuss that rev inkr is slow. I practice it on a little Synology (411+) 4bay with 2MB/s and a bigger one (1812+ with 13/MBs. In both scenarios is the target the bottleneck with > 90%.

My Question is another NAS System able to handle the File access rapidly smarter/faster ? Is this a Synology issue ?

Please post me your speed

thanks

Ulrich
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Whats your Backup to NAS reversed Incremental Perfomance

Post by Vitaliy S. »

Hi Ulrich,

I don't think it is a Synology issue, any high-end NAS device should provide you with more IOPs power, thus better job performance. Let's see what other community member backup jobs numbers are.

Thanks!
Gostev
Chief Product Officer
Posts: 31457
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Whats your Backup to NAS reversed Incremental Perfomance

Post by Gostev »

For reversed incremental backup performance on consumer and SOHO grade NAS, rule of thumb is 2 MB/s per spindle (regardless of NAS make and model). Thanks!
Andreas Neufert
VP, Product Management
Posts: 6707
Liked: 1401 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert
Location: Germany
Contact:

Re: Whats your Backup to NAS reversed Incremental Perfomance

Post by Andreas Neufert »

My customers use NAS CIFS targets with 1000+ VMs and more than 100TB of data very successfull with Reverse Incremental.

For Synology, check with Synology the health of your System and Volume. One of my customer had to fix some things and recreate raid and volume and doubled performance.

Also Consumer NAS Systems have different performance over different protocols.
Check out iSCSI (Enable Jumboframes on Switch an NAS), CIFS-SMB (use Veeam Repository CIFS and select data mover proxy on same network/site as storage system is), or NFS together with a Linux System as Repository ( check userguide).
NFS has in many cases best performance because consumer NAS are based on Linux very often.

Check also you virus scanner that network traffic and Repository backup files are excluded.
umenz
Enthusiast
Posts: 52
Liked: never
Joined: Jun 15, 2011 9:24 am
Full Name: Ulrich Menz
Contact:

Re: Whats your Backup to NAS reversed Incremental Perfomance

Post by umenz »

Update: changing backup repository at one of my installations to max 1 Task increment from 3 to max. 20 MB/s
Yesterday I have checked a bigger NAS (Rackstation) and have the same bottleneck with 2 ore more tasks.

My need is to make a Backup with 100 MB/s - this looks not possible ? That is the question starting this thread...

offtopic:
The Problem is when a 10TB SQL Database a week long runs with a snapshot. In some situation it needs 5 TB for the Deltafiles.
And have only one Backup in 5 ore more Days and blockted all replication Jobs.

Replication is in the same environment 80MB/s. So my idea is first replicating for a smaller timeframe with a snapshot on the production site and than backup the replica vm. Is this possible ? Have anyone do this ?

Ulrich
Butha
Enthusiast
Posts: 39
Liked: 20 times
Joined: Oct 03, 2012 10:59 am
Full Name: Butha van der Merwe
Contact:

Re: Whats your Backup to NAS reversed Incremental Perfomance

Post by Butha »

Hi Ulrich,

At the moment using a replica target as " source" is a feature coming in the new release (V8) only. In Theory yes you can run a replication job to another environment, then setup a 2nd job to backup the replica - but I don't think it's ideal.

In my experience achieving sustained throughput of large datasets @ 100MB/s (Megabytes) to shared storage is not common. There are many other factors like dedupe/compression etc that makes your overall backup take a lot less time, rather than running at a maximum speed.

Do you have 10TB of "changed" data per day, or is the dataset combined 10TB? It seems your snapshots for 7 days is roughly 5TB in size - so about 900Gb per day change?

With regards to the NAS as a target - yes the smaller SOHO devices are not geared to handle multiple writes at the same time - very similar to trying to copy 3 different large files to it at the same time VS a single copy.

But to answer your one question: Is it possible to get the backups running at 100MB/s? Of course:

Have some jobs running at 300MB/s +. It's perhaps worth noting this is to internal local disk on a Dell server (not to SAN).

Butha
umenz
Enthusiast
Posts: 52
Liked: never
Joined: Jun 15, 2011 9:24 am
Full Name: Ulrich Menz
Contact:

Re: Whats your Backup to NAS reversed Incremental Perfomance

Post by umenz »

Do you have 10TB of "changed" data per day
not so much, but if a byte in the Database files is changed, a 16MB big Block is moved to the Deltafile.
If anyone runs a bigger query the whole Database file is moved to the Deltafile in a very very shot time.
If a block is in the Deltafile and changed again the deltafile does not grow anymore.

If you have Database Files on a Volume with small freespace (<30%) the risk grow up with every hour a snapshot is existing.

The effective changed Data is 30GB to 100GB typically per Day and not the problem.
The Problem is long Backuptime at RI Backups so your solution is good to make faster Backups to local store and copy this later to the archive. That can be a solution i think.

thanks
Ulrich
Andreas Neufert
VP, Product Management
Posts: 6707
Liked: 1401 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert
Location: Germany
Contact:

Re: Whats your Backup to NAS reversed Incremental Perfomance

Post by Andreas Neufert »

backup as replica source will come with v8

backup to local disk and backup copy job to NAS is a good idea.

backup from replica today is possible but no change block tracking possible. Snap and Scan backup processing read out all VM data and detect the changes. (Read all VM data to Proxy, Proxy detect changes and process like CBT backups afterwards.

If you need fast backups the cheepest thing is to use a physical server with local disks or JBOD with a good controler.
(Instead of buying slow NAS boxes)

My experience with this NAS boxes are better than yours.
Can you please tell us advanced settings of your backup job and repository config.

inline Dedup enabled?
What compression level?
What block size(local/lan/wan)?
repository advanced config Block allingment and uncompression set or not?
At job statistics, click on the vm... which processing mode is viewed for the disks? SAN/Hotadd/NBD?
umenz
Enthusiast
Posts: 52
Liked: never
Joined: Jun 15, 2011 9:24 am
Full Name: Ulrich Menz
Contact:

Re: Whats your Backup to NAS reversed Incremental Perfomance

Post by umenz »

Hi Andreas,
inline Dedup enabled?
Yes
What compression level?
Optimal
What block size(local/lan/wan)?
LAN
repository advanced config Block allingment and uncompression set or not?
No/No
At job statistics, click on the vm... which processing mode is viewed for the disks? SAN/Hotadd/NBD?
nbd in my lab
at customer sites we use proxy, but not really faster

V8 sounds interesting, you mean Backup from replica as source ?
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Whats your Backup to NAS reversed Incremental Perfomance

Post by veremin »

umenz wrote:V8 sounds interesting, you mean Backup from replica as source ?
Nope. It's the opposite - replica from backups. As to backup the replicated VMs, please see the corresponding topic; might be helpful.

Thanks.
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Whats your Backup to NAS reversed Incremental Perfomance

Post by Vitaliy S. »

umenz wrote:My need is to make a Backup with 100 MB/s - this looks not possible ? That is the question starting this thread...
It should be possible, but with high-end NAS device and forward incremental backup mode.
Gostev
Chief Product Officer
Posts: 31457
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Whats your Backup to NAS reversed Incremental Perfomance

Post by Gostev »

With forward incremental mode, you can do 100 MB/s backing up to my old notebook ;) no need for high-end NAS for sure, any NAS would do.
BrandonH
Influencer
Posts: 21
Liked: 2 times
Joined: Jul 10, 2013 4:37 pm
Full Name: Brandon Hogue
Contact:

Re: Whats your Backup to NAS reversed Incremental Perfomance

Post by BrandonH » 1 person likes this post

So I'm not sure what's considered "High End" but I back up to three Drobo 1200i's using tiered storage (3x 800GB SSD's, 9X 3TB). My backup server is one of our retired VM host servers (24 Core, 128 GB/Ram). I'm running two 10G into the network fabric (Brocade VDX) Two 16G into the SAN Fabric (Brocade 5th Gen) with 3PAR Storage snapshots.

With all of the above, I average around 10-25 MB/s using Reverse Incremental with four (Yes 4) jobs running at the same time. From time to time I'll spike to 75 MB/s on new jobs (Eager Zeroed drives).

Veeam support states my speeds are normal and good, I can tell you that I managed the same performance on a 4 core 8GB ram server with two 1 gig connections.

I've also tried backing up directly to one of our 3PAR devices to see if that would improve the performance (To eliminate the Drobo's being the issue). Running 2x16G to 4x8G MPIO I saw less than 10% improvement when backing up to a 10K SAS drives via FC.

I have 1-2TB of changed data a day.

Hope this helps
umenz
Enthusiast
Posts: 52
Liked: never
Joined: Jun 15, 2011 9:24 am
Full Name: Ulrich Menz
Contact:

Re: Whats your Backup to NAS reversed Incremental Perfomance

Post by umenz »

Hi Brandon,

some test later.. ,i have Setup an older HP Server with a P411/1GB and 12x 3TB SAS/7.2k drives in RAID 5 as backup repository.
Use Windows Server 2008R2 2vCPU and 8GB RAM and replace the NAS with this storage as CIFS Share. no other changes.

This litte HW Change boosts a Active Full from 10MB/s to 50 MB/s and the Proxy is now the bottlenek.

the next test are switching from RI to Incremental do not speed up again, now inkr. is a littel bit slower* ~ 40MB/s at active Full ?!?

So my experiences shows the Synology NAS (i have used 3 types) is not usable as repository for VEEAM Backup.
But SQL Server Backup to this NAS to the same share runs with 100MB/s.

* @Gostev for the next steps to run real RI vs I i need your old Notebook to top the 50MB/s

thanks
Ulrich
tsightler
VP, Product Management
Posts: 6009
Liked: 2843 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Whats your Backup to NAS reversed Incremental Perfomance

Post by tsightler » 2 people like this post

umenz wrote:some test later.. ,i have Setup an older HP Server with a P411/1GB and 12x 3TB SAS/7.2k drives in RAID 5 as backup repository.
Use Windows Server 2008R2 2vCPU and 8GB RAM and replace the NAS with this storage as CIFS Share. no other changes.
I believe the P411/1GB is a flash backed write cache controller which should make a pretty significant difference. Even just a good sized battery backed write cache will make a significant difference. Most of these NAS devices do not support write back caching, although a few do offer it, but is recommended only when the unit is used with a UPS, and even then it needs to support a controlled shutdown.

The math for calculating the performance of any given set of disks for reverse incremental is pretty simple and you can get a pretty good estimate using the RAID IOPS calculator. You pick your drive (or you can manually enter your drive specs), RAID levels, stripe size, set a 33/66 read/write mix and set the average I/O size to 512K (assuming default optimizations are used for the job). This will give you and average random IOP result, and a expected throughput. Divide this number by 3 and that's going to be pretty close to your "worst case" estimated throughput for reverse incremental for that array. The final result will probably be slightly higher, but this is a good place to start.

Write caching will then play a huge role in increasing the performance above these expectations. Write caching effectively turns the random I/O nature of reverse incremental into a more sequential operation with significnatly less overall disk contention. While writes are going to the cache, reads are still random, but not being interrupted by constant RAID rewrites as writes are being cached. Once the cache reaches it's high water mark it will flush writes in a more sequential way until the cache hits the low water mark. Assuming the caching system is well designed this can offer a huge boost, but even a perfect cache system is unlikely to get you beyond 100MB or so with only 12 disks in RAID6. The RAID caculator actually has places to enter estimated cache hits. You won't have a very high read cache hit, but write hits will be significant. It's much more difficult to calculate the benefit though because it depends not just on the hit ratio, but also the performance of the cache device.

When I personally refer to a "high-end" device, I'm referring to devices that offer write back cache designs, ideally of significant size and with decent flushing algorithms (not all caching techniques are equal). Battery backed RAID controllers with 512K or larger caches are usually good (assuming a high percentage is dedicated to the write cache). Flash backed write caches or devices that offer flash caching can also be very useful, although yet again many lower end devices offer write around or write through cache which is of only minimal benefit.

Other factors that can have a large impact are RAID stripe size. Because Veeam is doing pretty large I/O, having small stripe sizes can lead to a sharp increase in total I/O. For a simplistic example, imagine if Veeam needs to retrieve a single 512K block. For a device with a 128K stripe size I need only 4 I/Os to read the data from 4 drives. However, if the RAID is using 8K stripe size, suddenly reading a 512K block requires 64 m I/Os, meaning if you only have 12 drives each drive must perform at least 5 I/Os. The calculator will let you play with various size stripe settings as well.

Using different RAID levels have a huge impact. For example, RAID5 offers around 60% more IOPs for a given configuration, and using RAID10 can offer a 300% increase from RAID6 using the very same disks.

Of course, none of this is to say that a Synology can't be a usable target, it's all about whether it meets the desired backup window. A device can be slow and still accomplish this since, when using reverse incremental, only a very small subset of data is transferred each day. It's all about whether the device meets your requirements or not.
dellock6
Veeam Software
Posts: 6137
Liked: 1928 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: Whats your Backup to NAS reversed Incremental Perfomance

Post by dellock6 »

Also ,to add to Tom post, the daily change rate is one of the biggest value to evaluate for using reversed incremental. Again, average environments are seeing around 5-10% of the full backup, so there are chances the daily increment will not impact the performances, but as soon as daily change becomes important (for example if there is an exchange 2010-2013 around with its volume optimiazions that change a lot of blocks) suddenly RI becomes a problem.
A solution can be to isolate "problematic" VMs in jobs with forward incremental, and to use RI only for little changing VMs. You can find out the daily aount of changes by looking at the size of VRB files, or by using Veeam One which has a dedicated report.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
smartsys
Enthusiast
Posts: 32
Liked: 4 times
Joined: Sep 14, 2010 8:27 am
Full Name: Jeroen Leeflang
Contact:

Re: Whats your Backup to NAS reversed Incremental Perfomance

Post by smartsys » 2 people like this post

When you read the Veeam user manual there is a recommendation to use Raid 10 configurations for backup locations.

At first we used dedicated backup servers using lots of spindels. For example a HP Proliant DL380p Gen8 with 4 disks for OS and Application and a hdd enclosure with multiple SATA drives for backup. If possible a direct SAN connection using FC, iSCSI or Direct SAS. Most of the times a RAID 5 was to be used, because the designers only looked at capacity and 4x 3TB equals 9TB RAID 5 backup space is much cheeper than 10x 1TB RAID5 or even 18x 1TB RAID 10.
Then there was the day the backup server failed. Not only did we loose the backup data, but also the complete backup application containing all job configurations and so on. It took several days to be able to backup again at that site.

At that time I started to look for a different solution.
I came up with a virtual Veeam "server" combined with FAST Enterprise Direct or NAS storage.
We use the Netgear ReadyData 5200 series when NAS storage is used.
The Veeam "Server" is always virtual and the proxy(s) are virtual or fysical with a dedicated LAN (preferably 10Gbps) to the repositories.

I preferably use a RAID 10 with a minimum of 10 SATA drives and a SSD write cache or 10k rpm SAS drives as a repository for backup jobs. For archive jobs RAID 5 configurations can be used with large SATA drives.
Because Reversed Incremental is very random IO and not like the Forward Incremental mainly sequential IO the backup repository must be able to perform high IOP rates. When troubleshooting backup performance issues at a clients site I noticed a 80% read / 20% write storage load on the repository server. This is very logical, because with a Reversed Incremental backup, each block of data has to be compared and possibly moved to the rollback file, so there is a lot of comparing overhead. The disk array used for backups could not cope up with load and this caused an intollerable low backup performance.
Now I recommend a large (100-200GB) SSD write cache combined with SATA drives in RAID 5 or 10 or a decent array controller with flash backed cache and high rpm SAS drives in RAID 10 to get proper backup performance.

When budget is an issue. You CAN use RAID 5 for backup job's, but you should not use Reversed Incremental. In these situations I use Forward Incremental jobs with a synthetic full rollback job in weekends or other "quiet" days.
When using Veeam for Archive tot disk or tape there is no need for reversed incremental, because Veeam will copy the right blocks of data to the tape or archive repository. There is no need to secure the last full backup file using 3th party software.
umenz
Enthusiast
Posts: 52
Liked: never
Joined: Jun 15, 2011 9:24 am
Full Name: Ulrich Menz
Contact:

Re: Whats your Backup to NAS reversed Incremental Perfomance

Post by umenz »

very interresting Posts, now I learn a lot and will change the concepts at medium sites one or two steps bigger.

But in very small customer Environments (1 Host + 1 Lowend NAS) i will use VEEAM too and change to syntetic Full at weekend.
If the custome have 2 Hosts i will replicate daily and Backup only weekly - this helps.

thanks all

Ulrich
Post Reply

Who is online

Users browsing this forum: Lewpy, RyanJ and 72 guests