DELL ME4012 for Repository - Questions

[TinchoB]

Hello,
I´m installing a new VEEAM physical Server in "appliance" mode (Server, Repository and Proxy all in one).
It´s my first VEEAM installation and I would like to ask your for advice for best storage config or if someone has experience with this kind of storage (or equivalents like IBM/Lenovo DS2200, HPe MSA, etc.)..

Production Hosts: ESXi 6.7 with a dedicated 10G mgmt network for Backup purposes. They are using a dedicated FC SAN for VM storage.
Veeam Server: Windows 2019 on DELL R730 with 4 10G network interfaces (2 for backup network and 2 for ISCSI ME4012 connection).
Connected to a Storage (for repository) DELL ME4012 iSCSI 10G dual controller with 12 NL-SAS 7.2k HDD.
Transport mode: NBD 10G (I plan to use SAN transport mode in the future but not today because I´ve not FC cards available for the backup server).

I´m attaching the ME4012 to the Server via ISCSI in direct attach or "back to back 10G" using patch cords (without network switch).

1) With 12 x 12TB NL-SAS disks, should I define a RAID6 volume (chunk 128, 256, 512) or ADAPT (chunk 512) ? Which is best ?
2) The Storage Disk Group should be Linear or Virtual ? (in Virtual the chunk size is fixed by default and cant be set, but according to DELL the amount of HDD in an array should follow certain rules like 8 data + 2 parity or 4 data + 2 parity for RAID6).
3) Should I format the volume with NTFS of ReFS file system ? I plan to use incrementals with weekly or monthly fulls.
4) I´m having problems enabling JUMBO frames in the iSCSI Server to Storage connection. I´ve enabled it in the storage (just a click), and then in Windows Server set MTU 8830 using NETSH commands in the ISCSI NIC interfaces, but after restarting the Server, Remote Desktop service and other ones hangs or not respond... after that, to gain access to the Server, the only solution is disable MTU in the storage. If the MTU is left by default in both sides (1500) there is no problem.

I would appreciate any advice or comment you could give me regarding this topics.
Thank you in Advance !!
Regards,

Tincho

[HannesK]

Hello,


1) RAID: chunk / stripe size 128 / 256 should be best. ADAPT sounds nice according to the DELL website. I would try it out and test with diskspd or real backup jobs (preferred) https://bp.veeam.com/vbr/VBP/3_Build_st ... lock-sizes
https://www.veeam.com/kb2014
2) whatever you prefer for RAID rebuild times. I would probably go for 8+2. Following the vendor recommendations is usually a good idea.
3) REFS if you like performance and good price / value ratio (assuming you use synthetic fulls). With active fulls there is no advantage in REFS
4) I don't fully understand. But I recommend to keep network settings default unless you have a good reason (and configure the whole network accordingly).

Best regards,
Hannes

[TinchoB]

Hello Hannes,
thanks for your answer !

Let me add some comments regarding these issues.

1) It seems ADAPT uses a 512k fixed chunk size that cant be changed, so 1 VEEAM block will be written in 1 Storage I/O operation. Is it good for performance purposes ? In the other hand if I choose RAID6, chunk size can be changed.

Regarding size, ADAPT has a 20% of space overhead over RAID6, which will give less amount of usable space (but the theory says its has faster rebuild times...which I dont know if true).

Example for an ADAPT disk array:
In a 12 x 12TB disk array as the one I have:
10 disk used for Data (Raw).
2 disk for Parity.
10 x 12TB = 120TB (Data Raw) ----> 20% of it = 24TB go for ADAPT Overhead.
Usable space = 120TB - 24TB = 96TB with chunk 512k

With RAID6 usable = 120TB (in a 10+2 configuration) with chunk 128 or 256k.

2) I´ve got just 12 disks, and I would like to make the better use of all of them in a single volume. RAID6 8+2 (10 disks) is a thing since I will leave 2 disks out of the array. So the options I have is to go with RAID6 10+2 (which will not follow the "power of 2" rule suggested by DELL), or 2 x RAID6 4+2 (what will eat 4 of the 12 disks for parity) or just use 12 disk ADAPT with a 512k chunk. What do you think ?

3) Regarding ReFS, based in your experience, have you noticed big backup/restore performance degradation as the weeks go on ?

4) Regarding JUMBO FRAMES, in the following link (VEEAM HCL for DELL ME4) they suggest to set it on from 1500 to 8860:
https://www.veeam.com/kb3113
But when I do that Windows Server hangs... I´ve tried 8860 and 8830 in windows... Do you think its necessary ?

Thank you very much !!
Regards,

Martin

[HannesK]

Hello,
1) ah ok, then I would with with classic RAID-6. 10+2 does not give you space disks.

2) depending on the service level, I would go with spare disks (8+2) or 10+2 without hot-spare disks, yes

3) for file level restore: we improved a lot by using asynchronous read in V10. For 11, there will be more async read. So I would go for REFS.

4) Ah, interesting I didn't work with changed MTUs since many years. But every part of the chain (server, switch, storage) must be configured correctly. If one piece is still on 1500, then it ends up pretty bad... I never tested different MTUs with Veeam workloads. With 12 disks, it's probably irrelevant. Hard to say.

Best regards,
Hannes

[nitramd]

Hello TinchoB.

~A year ago, I asked our senior network engineer about using jumbo frames to "speed things up" - I had a few repos that were ingesting tens of TBs of data. The essential answer was: jumbo frames are a pain to set up, to test, and to maintain - which corroborates Hannes's point #4.

The other essential answer was for me to be patient

[TinchoB]

Hello,
regarding JUMBO FRAMES, I did some quick tests with the DiskSPD tool you suggested me.

I set the DELL ME4012 with 1 ADAPT array composed of 12x12TB NL-SAS disks (mandatory chunk 512k cant be changed...). Array Controller A is active and Controller B is in standby.

Storage connected to Server DELL R730 with 2 x 10G Broadcom 10 Base-T interfaces using Cat6 UTP Patch Cords (back to back without network switch). NIC 1 to PORT 0 controller A and NIC 2 to PORT 0 controller B. Each pair in a different IP subnet. MPIO installed. ISCSI set on both sides. Server recognizes ok the volume as LUN0 to MS initiator and multipath is ok.

From the server side, the ADAPT array is recognized as disk V: and formated as NTFS with 64K.

DiskSPD tests results:
--------------------------
The goal at this moment is to make a simple test simulation like if it were an Active FULL (100% write) to note changes.

Windows CMD Command Line: diskspd -c25G -b512K -w100 -Sh -d600 v:\testfile.dat

With MTU set to default (1500).
I/Os | MiB/s | I/O per s
296823 | 247.35 | 494.70
ISCSI NIC peak 2.1Gb/s (Windows Task Manager)

With MTU set to JUMBO (9014) - enable on Storage console (just a click) and then the server (set MTU 9014 in the Advanced properties of the NIC config window, and then open a CMD and execute the NETSH commands to set both nics from 1500 to 9014).
I/Os | MiB/s | I/O per s
1071391 | 892.82 | 1785.64
ISCSI NIC peak 7.8Gb/s (Windows Task Manager)

Could it be possible such an improvement ??

Tincho

[nitramd]

Why would it not be so? You're getting ~3.61 times more data with jumbo frames enabled.

Have you measured sustained throughput? That would be a good number to have.

Have you been able to test throughput via Veeam yet?

[TinchoB]

Hi nitramd,
Using DiskSPD writing 3 files to Storage makes 10G ISCSI NIC get saturated... Same for 3 x Read. About +1GB/s.
I got similar results changing the DELL Storage disk array config from ADAPT 512k (12) to RAID6 256k (10+2).

Regarding VEEAM, I installed it, but as I´m still integrating this server to the VMWARE hosts, I could only do minor tests like active Full using NBD (max 100MB/s per VM), or deploy one HotAdd proxy. In the following days I will try to deploy a couple more proxys to try to saturate the Storage. I´ll tell you.

To be honest, I´m now a trying to define if I´ll go with RAID6 or stay with ADAPT as the Storage array config. VEEAM recommend RAID6, but DELL (according to this https://www.veeam.com/kb2014 ) validated ADAPT 512k...

And then what File System to choose for the Repo. I read a lot of comments in this forum and on the internet about users complaining about problems with ReFS and Win 2019 LTSC, ReFS and Tape Drives and son on... As I plan to use incrementals with active fulls (weekly), and then copy to tape drive, I´m a little worried about if to go with ReFS or then opt for NTFS.

Any suggestion ?

Regards

[TinchoB]

... regarding ReFS vs NTFS, the VEEAM server has 64GB RAM..

[ian0x0r]

Some great info on the PowerVault ME devices here.

Not strictly related but I thought I would point this out for anyone else reading. As you start to grow disk groups by adding shelves or use something like a ME4084 which has 84 drive capacity, then the RAID type selection may become more important for you. RAID 6 can only accommodate 16 drives per disk group, whereas ADAPT scales from 12 to 128 drives per disk group. This is true for both virtual and linear disk groups.

The maximum volume size is also capped at 128TiB, so if you have more storage than that, a scale-out backup repository can help you out.

A colleague of mine also tested adding additional drives to an existing RAID 6 group in virtual mode, it was not possible. It had to be created as a separate disk group which will upset the power of two number of disks in a group mentioned above if you plan to expand in the future. Adapt would allow additional disks to be added.

More info here https://www.dell.com/support/manuals/en ... lang=en-us

Ian

[FECV]

I use an ME4084 along with a SCv3000 for my Veeam backup environment. The main difference between my ME unit and yours other than capacity is I went with the SAN controller versus the iSCSI controller to connect to my Veeam server. In a smaller unit with 12 drives the main question of RAID versus ADPT is expandability. Do you plan to add more drives later on in an expansion controller? If yes ADPT is the way to go. I run all my ME40xx units with Linear Pools versus Virtual Pools. May be something to look into. I feel it gives better performance and I actually had issues with functions like encrypting on the drive until I enabled this which requires a reformat.
dell.com/support/manuals/en-us/powervault-me4012/ME4_Series_AG_Pub/linear-pools-and-disk-groups?guid=guid-467d81d3-298e-4cc3-be36-49071f082fcc&lang=en-us

Jumbo Packets is the way to go for sure. I used to run a lot of direct attach ISCSI stuff for years. Why are you setting to MTU to 8830? Why not set those two 10GB ports to 9000 to 9014 via the network adapter GUI?

[TinchoB]

Yes, expansion is always an option. Perhaps 12 more disks next year (with an expansion chassis). Not only to increase the volume size, but more important to add more I/O performance by adding spindles. The 128TiB volume limit pointed out by @ian0x0r is something to take into account...
I´m using 12TB disks (10.5TiB). From the performance point of view, for 12 drives, do you think a ADAPT is similar to RAID6 ?

My choices are: ADAPT (12)--> 84TiB (expandable in the future... Chunk 512k... theoretically "faster" rebuilds...)
or RAID6 (12)-->106TiB (not easy future expansion due to 16 drive limit... Chunk 256k).
Sacrifice some space now to grow something more in the future or not... ¿?

I agree with you that Linear should be better than Virtual (anyway virtual features are not needed for a VEEAM server).

About Jumbo frames, I tried to set up 8830/8860 with no luck, because it was stated here by DELL/VEEAM ---> https://www.veeam.com/kb2014 (look at the end...)
After some problems I ended up setting MTU 9014 which is the default jumbo size for the ME4 and the Server NICs. In the server I set it up using the NIC advanced properties and then with the NETSH command.

[m1ke78]

For what it's worth we've been running a Dell ME4012 in ADAPT for both production VM's and backups. 12x12 TB NL-SAS disks. I had a drive fail a few months ago and the rebuild took less than a day which surprised me. Also no data loss or any other issues. I also Just installed another 12 bay expansion yesterday which has gone smoothly so far and the full 24 disk group finished rebalancing the data and is now expanding.

So far I've been very pleased with the setup, and don't regret going with ADAPT over a standard raid config.

[JonJR]

We've just got a ME4084 running in ADAPT. Didn't really have a choice as the other RAID options wouldn't handle 42 disks in a single pool. Performance is fairly ordinary, compared to the 4 DAS shelves it replaced.

However, it has a massive limitation for us, the maximum volume size is ~130TB. With 600TB raw, we were hoping for a single 400TB volume, but instead have had to carve it into 3 volumes

[badabing888]

Has anyone used this is a hybrid setup with tiering?
virtual pool accross tiers idea is hotdata is buffered by ssd etc before being pushed down, any advantages?

Anyone gone past the 130TB limit on a single repo i.e using lvm with xfs to span multiple volumes on a single repo?

[badabing888]

For what it's worth we've been running a Dell ME4012 in ADAPT for both production VM's and backups. 12x12 TB NL-SAS disks. I had a drive fail a few months ago and the rebuild took less than a day which surprised me. Also no data loss or any other issues. I also Just installed another 12 bay expansion yesterday which has gone smoothly so far and the full 24 disk group finished rebalancing the data and is now expanding.

So far I've been very pleased with the setup, and don't regret going with ADAPT over a standard raid config.

How’s ur sustained througput with just running large disks only?

Are you running it as iscsi?

[TinchoB]

Hello badabing888,

Sorry for the delay, I´ve been off the forum for some time.
Just for your information we produce one active full per week and six increments.

Our Active FULL JOBs are getting an average of 750/800MB/s of processing speed (sometimes it could have peaks of 1GB/s or more).
Enabling Jumbo Frames made the trick for us !!

Take into account that we have directly attached the storage to the Server using iSCSi (without network switch) with a MTU of 9014 (jumbo frames enables in both sides).

An important fact is that to get these numbers we HAVE to run a Windows DEFRAG once a week, if not the performance will degrade.... We try to run defrag on Monday in order to let it run for a couple of days, and not mess with the FULL backup that starts on Friday !!

Very important: we have a traditional point of view regarding backups, so we defined a RAID6 Linear Array (12 disks x 12TB), and use NTFS windows disk format ! (I´m not sure if it could apply to ReFS...).

Tincho

[badabing888]

Thanks for the reply.

I've got a ME4084 being used across 10Gb/s iSCSI on the network jumbo frames enabled.
I need to go back and check the bench marks for for 100% read / writes or 50/50 non random IO it basically maxes the network card.

I need to go back and rerun the tests as it depends alot on block size, how random the IO is, read / write mix