Very slow backups to HPE DS3600

[ejenner]

Hi everybody

I've got an unusual issue with backups being very slow to write to a pair of HPE 3600 external drive boxes. I've logged a ticket and we've tried some things but I wonder if anybody else has any ideas. My Case # 05461192

Firstly, we have this setup a number of times for different repositories throughout our organisation and this problem is unique to this implementation.

I have x2 external storage arrays which are the HPE DS3600 type.

In Windows server 2019 I've created a spanned volume using ReFS as the file system.

This problem is not Veeam specific but as is often the case, underlying OS issues can affect backups and sometimes require looking at even if they're not just within Veeam.

If I copy a 100GB file from the C: drive to the drive on the external (direct attached) storage it'll begin the copy at 1.25 GB/s but after about 50% of the copy the speed will suddenly plummet to about 40 MB/s. This also happens if backing up in Veeam.

I've tried reformatting the volume as a single volume on either of the drive arrays with NTFS instead of ReFS.

With my tests I've eliminated the following:

1. It's not the spanned volume as the problem still exists with 'simple volumes'
2. It's not ReFS as the problem still exists with NTFS
3. Both drive arrays do the same thing so it's unlikely to be a hardware fault specific to one physical enclosure

It seems a bit like a cache is filling up when you see it can copy 50-60GB at a respectable speed but drops afterwards.

[Mildur]

Hi EJ

It seems a bit like a cache is filling up when you see it can copy 50-60GB at a respectable speed but drops afterwards.

That was something which came to my mind too.

You say, you have the same behavior when you are copying a file with windows explorer instead of Veeam?
What does HPE Support say to this issue? Probably better to also ask them, they can access the array logs and analyze them.

Thanks
Fabian

[mkh]

I know this is probably not the case here, but I instantly think of SMR drives when you have drive performance the plummets after a certain amount of data is written.

I was bitten rather hard at some customers when WD silently changed the "WD Red" to use SMR.
https://www.anandtech.com/show/15878/we ... s-hdd-mess

[ejenner]

I'm beginning to think this is the fault of the controller card.

We're using a HBA H241 in this system. The other places where I have a similar implementation it is using the Smart Array P441 controller card.

The theory goes that this card is firstly not verified for use in a DL380 chassis and 'not a proper RAID card'

It's also a significant difference from this system compared with the others we have.