Cisco Veeam Machine?! S3260

[l0stb@ackup]

@BigJack

I agree the Cisco doco and naming conventions are a bit lacking and inconsistent, however I can answer your main queries:

Firstly, our strip size is 64KB. I won't change it as we're getting the desired performance.

As to Drive Cache, the doco states the following about Unchanged:

The controller uses the caching policy specified on the physical drive. This is the default option.

The obvious question is - "how do I specify the caching policy on the physical drive?? ".
Since I could not find the answer I simply went with the safe option of "Disabled", which hopefully overrides the elusive physical drive caching policy.

With regards to the Strip/Stripe question, the virtual drive settings page has a "Strip Size" setting, no "stripe", so that should clarify this question as well.

[dalvarez17]

Great info!
In order to change those settings in the virtual disk, do you have to format them to apply changes?
Or there is no data loss in the process? Same disk/data remains available to the OS?

Thanks in advance!

Hello, just wanted to chime in a share a hugely important tip we learned today:

Cisco released a great document titled "Veeam Availability Suite on Cisco UCS S3260":
https://www.cisco.com/c/dam/en/us/solut ... 739852.pdf

Page 20 discusses the recommended virtual drive settings for the Veeam repository.

Our settings were:
Cache Policy = "Direct IO"
Read Policy = "No Read Ahead"
Write Policy = "Write Through"
Disk Cache Policy = "Unchanged"

We then applied the recommended settings:
Cache Policy = "Cached IO"
Read Policy = "Always Read Ahead"
Write Policy = "Write Back Good BBU"
I also set Disk Cache Policy to "Disabled"

Since we made the recommended changes, our backup performance saw a 600-1300% improvement, which is quite amazing.

Our specs are:
UCS S3260 M4
2x Intel Xeon E5-2620 v4
128GB RAM
Cisco UCS C3000 RAID Controller for M4 Server Blade with 4G RAID Cache
Veeam repository virtual drive: 23x 8TB SAS HDDs in RAID6

[l0stb@ackup]

@dalvarez17

You can change those settings live with no downtime or any disruption. I did it while a backup was running, no problem.

However be sure to not touch the Select RAID Level to migrate drop-down menu at the top - you do not want to migrate the RAID level.

[ferrus]

I'm a bit confused by that Cisco document. When setting up the storage via the CIMC, Cisco recommends a strip size of 512KB (page 20.) When doing the same thing with UCS Manager, Cisco recommends a different strip size of 64KB and also recommends setting Drive Cache to disable (page 60.) Cisco didn't mention anything about the Drive Cache in the page 20 CIMC config, and the screenshot didn't specify disable.

So what is Cisco's recommendation? 64KB/512KB? Unchanged/Disable?

I suspected our Cisco C240 servers have had really poor disk performance recently. I placed a support call #03336103, and discovered that they've actually dropped 85% of IO performance since migrating to our new datacentre.

The only notable difference (apart from ReFS), was that we used UCS Manager to configure the storage at the new location, whereas it was standard RAID BIOS tools at the original DC.

UCS Manager's storage defaults, are all completely wrong for a Veeam repository. They can all be reconfigured without data loss - except for the Strip Size.
As you pointed out, Cisco recommend 512KB at the top of the document, and 64KB further on.
The reason for this, as far as I can tell - is that UCS Manager only allows 64KB (Platform Default), when it's configured centrally.

I'm not sure if this is a bug in our version of UCSM, of if it's just how it treats C240 servers - but there's no way of changing this default Strip Size centrally. So we're having to destroy and reconfigure our repositories manually.

For info although amending the Read, Write, IO and Caching Policies all helped with performance - the Strip Size gave us the largest overall increase. 65% quicker with 512KB, instead of 64KB.

[jhoughes]

ferrus, that is just a limitation of the management from UCSM which has existed for years unfortunately. Even in the newest 4.0 versions of firmware (doc here), you cannot change from the default stripe size for the VD:

Code: Select all

Name	                             Description
Strip Size (KB) field	             Stripe size for a virtual drive. This can only be Platform Default.

BIOS and cli configuration is definitely superior in this instance, just one of the caveats in UCSM with regards to storage management.

[B@ckMeUp]

Hi. I wanted to say that about a year ago, I moved to Veeam. I purchased 3 Cisco UCS M5L servers to act repositories. These repositories had always performed well reading data, but performed poorly on back jobs that required a lot of writes. For the most part of this past year, it wasn't an issue because the majority of my backups consisted of read IO. Well, recently, that changed and I had an issue where my servers threw up LSA_SIS event id 129 errors and the VM repositories would routinely stop responding during portions of the backup job that required a lot of write IO. I made it my mission this weekend to finally resolve this. I updated everything I could think of and almost lost hope. Then I ran across this post. Bingo, I changed my virtual disk configuration from "Write Through" to "Write Back Good BBU" while a backup job was running with a high about of write IO. It was an immediate switch. I was averaging abround 30-40 MB/s writes before. Almost immediately, the job started hitting 500-600 MB/s writes. Other jobs afterwards started reaching 700+ MB/s processing rate where before they would averaging 50 MB/s to 200 MB/s depending on the percentage of writes. I just wanted to put this out here in case anyone in the future has the same performance problem.

[SkyDiver79]

I've done some reading, but it's no clearer.

The MS ReFS page seems unequivocal:
https://docs.microsoft.com/en-us/window ... eployments

But, as mentioned in the Veeam thread here, there's a Veeam Deployment Guide that covers my hardware and storage EXCATLY, on HW RAID6 ReFS - https://www.veeam.com/wp-cisco-ucs-c240 ... guide.html

I thought this was resolved a couple of months ago - but I'm unsure if it just covered SAN attached storage now.

For an very imminent server rebuild - should I be using HW RAID6 ReFS, or JBOD Dual Parity Storage Space (Std) ReFS?

The "'ReFS is not supported with RAID controllers in non-passthrough mode.'" limitation is only for Storage Spaces. When you use your Raid as Basic Disk, all is fine.

[BackItUp2020]

Blast from the past...

I'm trying to do this Cache change, but my option to go to Cached IO from Direct IO is grayed out - even from a new virtual disk config. Wondering if you've seen that.

I have a TAC case, but I'm wondering if it just something they've gone away from in recent firmware updates.

[SkyDiver79]

This is actually only when there is no cache battery or it has a defect.

[BackItUp2020]

Blast from the past...

I'm trying to do this Cache change, but my option to go to Cached IO from Direct IO is grayed out - even from a new virtual disk config. Wondering if you've seen that.

I have a TAC case, but I'm wondering if it just something they've gone away from in recent firmware updates.

FYI - this option is grayed out if you do not have a feature-supported RAID controller installed.