array cach settings for reverse incremental

[kte]

I'm having a backup server with 2 arrays in it with each 2 GB cache, each array has 12 sata disks
I use reverse incrementals so do I put more write cache or more read cache on the array ?? Windows 2012 has 32 GB of ram and the veeam logs say he is using allready 8 GB or filecache

Thanks
Koen

[tsightler]

It can be hard to predict this due to the way different controllers flush writes. Some controllers practically stop all reads when the write cache hits the high-water mark and forces a flush which can cause high read latency and very poor, bursty performance. If your controller behaves this way then you'll probably want to lean toward read-cache. On the other hand, if the controller is a little smarter then a heavy dose of write cache can improve performance significantly. I'd strongly suggest running some IOmeter testing with a good 33/66 read/write mix (256-512KB chunks) and test various combinations.

That being said, I believe that Veeam by default always sets the max OS cache to 25% of the available memory, this does not indicate that it is using all of this memory, only that it is setting the OS cache limits to this value which helps to keep Windows from uses all available memory for cache.

[kte]

It is an HP smartaray P20i with 2GB cache and P421 with 2 GB cache, and a windows raid 0 over bothe array's

[yizhar]

It is an HP smartaray P20i with 2GB cache and P421 with 2 GB cache, and a windows raid 0 over bothe array's

Hi.

Using RAID 0 is a bad idea.
it incurs dependency on BOTH arrays for no need, and much higher risk for data loss or other problems.

I suggest that you delete this RAID0 drive and go for something else.
For example, you can setup 2 independent repositories, one on each backup target array.

Yizhar

[kte]

the disk are protected with hardware raid, and I copy all the backups to another server with veeam
It is the only way to access all the iops of all my disk on both array's

[tsightler]

What stripe size are you using for the RAID0? If you are not very careful you can actually eat into your IOPS significantly by striping with RAID0 in the OS. For example, if you're using a 64K stripe for the RAID0, and Veeam creates a 512K write (quite common) then the write will be split into 8 chunks at the OS level and thus create 4 write I/Os to each array, which can potentially massively increase the write I/O penalty and uselessly eating IOPS.

Understanding chunk/segment size is already of vital importance for building a high performance RAID repo for Veeam, and is even more important when you start layering RAID on top of RAID or you'll just end up with a poor performing setup.

[yizhar]

the disk are protected with hardware raid

Still RAID 0 = very bad idea.

and I copy all the backups to another server with veeam

This is good.
But still RAID 0 = bad idea.

It is the only way to access all the iops of all my disk on both array's

No, it is not the only way.
You can use use 2 repositories (one on each array), and run 2 jobs in parallel.

Yizhar

[yizhar]

In addition to above -
With the RAID0 configuration
A performance problem (networking, raid problem, controller problem) in either of the arrays will affect the whole volume.

Yizhar

[kte]

Still RAID 0 = very bad idea.

It is software raid 0 on top of 2 HW raid card , if you're CPU mortherboard or memory has an issue I'm in the same situation. And I have a 4 h HP intervention on that backup server. It is a pore man solution, but good performance low price, some risks

You can use use 2 repositories (one on each array), and run 2 jobs in parallel.

then in only can use half the iops and i loose 2 time free space on my luns.

[yizhar]

Hi.

Can you provide more details about your whole environment?
Production storage?
Kind of connection to those 2 arrays on the backup server (are they DAS connected, SAS, ISCSI, FC)?
Number of VMs backuped up?
Are you using windows server 2012 dedup?
More detailed info that will help us help you.

Yizhar

[kte]

Production storage a StoreServ 3PAR 7400 2 nodes 8xSSD200 GB + 80x900GB SAS 10K + 20 x 3TB MDLSAS connected in FC 8 GB
to 23 ESXsserver running 300 VM's
backup server veeam 7 DL380 gen 8 ,32GB ram + 8 core CPU intel 2650 LFF ,2x450 (R1) sas for OS + SQL + veeam + 10x4TB (R6) on P420i with 2GB cache DAS
+ extension enclosure with 12 x 4 TB (R6) on P421 with 2GB cache DAS also in the same server
Windows 2012 R2 cofigured in striping over both arrays on the 4 TB disks
FC connected for direct san backup in 2x 8GB and network in 2x10 GBit for restore and replication on te same server on another site
full backups are running @ 400-500 MB/s , reverse incementals between 10-12 MB/s for exchange (4TB) and 40-60 MB/s for other VM's (22 TB)
Synthetique full that 20 hours of 3 TB of exchange data

So can I change anything in the design to keep 60 TB of netto data capacity (formatted in 256 k blocks in windows) and get higer reverse incremental troughput.
chache settings for example there are both @ 30 % read /70% write

K

[tsightler]

(formatted in 256 k blocks in windows)

It sounds like you are referring to NTFS cluster size. This doesn't have anything to do with the segment size used by the RAID 0 stripe.

You might want to run some bechmarks against various configurations. You want to format your RAID with a large chunk/stripe size, and then use a stripe size roughly equal to that for the RAID0 if you feel it's critical to do that (I agree with the advice to just have two repositories).

Here's a link to an IOmeter provide that you can use to perform some benchmarks with various configurations.

[kte]

how can I find the segment size ?

[yizhar]

Hi.

First of all I must be modest and honest - I work for SMB customers so my experience is with much smaller systems (2-3 hosts max, few TB).
So my insights are based on common sense and smaller scale experience.

Thanks for the detailed info.
I would also like to further ask the following:
How many VMs?
When you mention 60tn neto capacity - do you mean the size of production or backups?
Which Veeam version and patch level?
How many jobs?
How many jobs running concurrently?
Size of VBK files?
How are jobs configured in general?

I still highly recommend that you get rid of the RAID0 volume, and go with 2 separate volumes one on each controller.
There are several reasons, I have mentioned most before.
Do not go for wide striping on different hardware - it is not the same like 3PAR...
Go for concurrency instead (several concurrent jobs running against separated target repositories).

Improving the performance can be achieved by techniques such as:
Configure several jobs with around 5tb VBK file max, or even less.
Very large servers such as Exchange 4tb = single jobs.
Other VMs = combined by whatever you decide, but not too many VMs or GBs in single job.
Reasonable size VBK files will be easier to manage and maintain then very few large ones.
Running several jobs in parallel but not to many. Limit each repository to 2-4 concurrent jobs max. Play with this setting and test the results.
Note that you should first check how a single job with single VM performs actions such as full/reverse/synth/etc... when nothing else uses the same volume (to get some baselines).
Then - next step is to check how several jobs perform concurrently.
For example you mention that reverse incementals are slow between 10-12 MB/s, but I don't know what is the VBK size that it runs against, and what other activity if any was running on the same disks when you checked?

Just for the test - please try with a single job that has VBK smaller then 1tb,
running exclusively, when nothing else is using the same target disks.
What results do you get?

Have you also contacted Veeam support for advice?
If not - I think that they can help in addition to the interesting conversation here.

Yizhar

[kte]

total Vm's to backup is 26 TB devide into 250 VM's
netto 60TB is the backup destination , the source production san is arround 100 TB but only 40% used , I'll add an new backup server if the becaome lot bigger the backups
veeam 7 patch 1
I have 7 jobs with parrallel processing enabled and I run 1 job after the other one so concurency 1 for the jobs
VBK full backup are between 2 TB and 6,5 TB
Every job is doing reverse incremental and an active full every 3 months + a copy to a secondary site of every backup with veeam copy on a 10 GB link

The hardware is two times the same Smart Array controller also so I don't see any issue in my situation (I'm a storage engineer , it is only backup and I have a 24x7 4h contract on that server and I have 3par storage snapshots also) the full backups go @ 500MB/s for the reverse of the exchnage 2007 is taking 3 to 4 hours on a regular day without maintenance tassk on the exchnage server (who changes alot of blocks) and he create an output VRB of 50 to 100 GB. The 3 exchnage VM size are 3 TB all exchange databases -> 10-12 MB/sis the performance, This jobs runs first so nothing else runs on the storage and I chnaged the dedup level to WAN , to get smaller increments ( devided the backup file by 3 to 5 whare before I used dedup local)
Other jobs with storage target put in local run between 20 and 45 GB/s for filservers or other things so thats good, only exchange is slow in reverse incremental
synthetic full + transform to reverse is taking to mutch time in the weekend 20 hours for the exchange backup.

The question is what is the SA cache settings to get max performance on the reverse incremental backups??
100% write over 100% read or what to take between them, windows read caching in ram is 8 GB.
So how can I get max performance from this config.
And what are the best practices to get the most out of reverse incremental, even for the future for other customers, disk raid level, SAS vs MDL SAS,.. direct san with physical proxies,...

K

[yizhar]

Hi K.

Thanks for the detailed info.

> I chnaged the dedup level to WAN , to get smaller increments
Can you try also with "LAN"?

> And what are the best practices to get the most out of reverse incremental
If you have enough disk space, you can configure select jobs (such as the exchange job) with forward incremental instead of reverse.
This can give you better performance and reduce load, in the expense of more disk space needed for that job.
Can you test?

Other then that let see what other people here can tip.

Keep sharing your findings.

Yizhar

[kte]

I tried wan but the reverse backup files are almost 2 time as large and the backup time betweeen local, lan and wan keeps the same

[tsightler]

As stated above, reverse incremental is heavily random write I/O and thus write cache is most likely the best use. The exact results can vary greatly based on hardware so it's hard to make a 100% universal "best practice" recommendation other than "test your hardware". RAID 6 is the absolutely worst case for reverse incremental because you will get roughly 7.5% of the normal sequential write performance using reverse incremental with RAID 6 so an array that provide 400MB/s full backup performance will be lucky to provide 30MB/s, potentially worse if the RAID is formatting using segments that are too small.

For high change rate VMs you can choose to use forward incremental and synthetic full every day. Although the transform will take several hours, the portion of the backup where the VM has a snapshot will be complete much faster.