Veeam Hardened Repository + Ubuntu + Multipath

[mkeating44]

Hi everyone,

I have a strange issue with a new storage repository we're trying to put into production. I have a case open at the moment (#05459009) and happy to work through the process, but was hoping some in the community might have some advice or knowledge on what we're trying to do.

We're wanting to use hardened repository with immutability, so we have a Dell R550 server and Dell EMC Powervault ME4084. The server has a 20TB RAID 5 SSD array, and the ME4084 has ~220TB of storage configured with the ADAPT parity setting. The storage is directly connected to the server using dual path SAS cables (one to each controller on the ME4084). Installed Ubuntu 20.04.4 on the server, all updates applied. Multipath in Ubuntu is configured using the Dell Linux configuration document https://dl.dell.com/manuals/common/powe ... _en-us.pdf and looks to be working fine - formatted the volume with XFS and can access it fine. Both storages are formatted with XFS, set up as per the Veeam hardened repository/immutability guide, added to Veeam with no issues. I've then added both into a Scale Out Repository with no problems. Set Ubuntu to mount both, with the PowerVault volume mounted through the Multipath device.

Set up a couple of test jobs to the SoR, set the PowerVault-based extent to Maintenance mode and ran without issue. Set the SSD-based extent to Maintenance, ran the other job to the PowerVault with no problems - did full and incrementals without issue. However, once I rebooted the server, jobs to the PowerVault extent fail repeatedly with:

Code: Select all

Error: Update of the storage is not supported. Storage version: [-1721682645]. Failed to open storage for read/write access. 
--tr:Failed to upgrade/fix the storage

The files look fine, have the right attributes, I can log in as the local non-privileged user used by Veeam and access the location, write new files with no problems. I even reformatted the volume to test in case I set up permissions wrong, but had the same issue - works fine until a reboot.

Does anyone have a similar setup, or run into a similar issue that might have a suggestion about where to look next? I don't know where the issue is but we're running out of time before this needs to be in production. Thanks!

[Mildur]

Hi Michael

Welcome to the forums.
Thanks for Case number. I see, that you found together with our support a storage issue.
Let us know if it works now after doing the xfs repair.

Thanks
Fabian

[mkeating44]

Thanks Fabian,

Looks ok now after multiple reboots - I had a concern that I hadn't set up the volume properly (I didn't partition it prior to formatting with XFS) so not sure if that has caused an issue needing the XFS repair, or if there was an issue in dismount on reboots.

It's survived multiple reboots so far, going to continue testing into next week to see if it continues to work. Happy to get any other suggestions on provisioning filesystems under Linux in the meantime!

[mkeating44]

All solved - thought I would document the answer for future reference (for any non-Linux gurus out there).

Everything in the Veeam doco was followed, and lots of help from https://www.starwindsoftware.com/blog/v ... ory-part-1

All the doco assumes locally attached storage with a single path to it, which is likely most scenarios. But for local DAS with dual SAS, or iSCSI with multiple connections, Multipath will set up the connection to the paths and manage the volume over the paths.

Assuming you have that set up correct, format with XFS as in the guides, but when it comes time to mount ensure you use the Multipath mapped location to the volume - not the GUID! In my case the path was /dev/mapper/mpatha - map this in fstab with the options required to ensure it is mounted using Multipath for consistency.

We caught this by monitoring the console as root which produced an XFS consistency error when running the job. Running xfs_repair on the volume got it working again, remounting it made it available with all files intact and Veeam was happy afterwards.

[Mildur]

Thanks for letting us know.
I‘m glad it is working now.

Thanks
Fabian