Is support for data immutability on HPE Storeonce on the Veeam roadmap?

[Andreas Neufert]

The beta for v12 will be out soon.

[thynes]

+1 on this feature!
We have a lot more customers requesting immuatability now a days, is there any more information if this will be included in V12?

[Gostev]

Yes, we plan to include it in V12.

[Florent Foulquier]

I confirm and I already got the opportunity to see a demonstration with VEEAM 12 beta and StoreOnce VSA running 4.3.2 beta that permit to have ISV managed Data immutability.
By the way StoreOnce FW 4.3.2 is avaible officialy as control release and brings a lot of interresting enhancments with VEEAM:

• Supports dual authorization that requires approval of two separate users to authorize execution of an operation. This feature protects data loss caused by malicious actions, such as a ransomware attack.
• Supports the ability to set the iLO security state to High Security or Production.
• Supports a passphrase for additional security in the Time-based password option.
• Supports TLS 1.2 version to secure SMTP connections.
• Supports VMWare vSphere ESXi 7.0 for the VSA product.
• ISV Controlled Data Immutability for Catalyst stores.
• Housekeeping performance improvements.
• Random read performance improvements.
• Improved housekeeping performance with Cloud Bank stores.
• Catalyst database performance improvements to support higher object counts per store
• Veeam 4KB Fixed Block

[mike_stewart]

Hello,
I received an email from HPE saying that StoreOnce 4.3.2 FW is not available any more (control release) due to some issues.

"Please be informed that after upgrading StoreOnce systems to the latest available HPE StoreOnce 4.3.2 Controlled Release Software, there is a remote possibility of stores becoming inaccessible intermittently. The StoreOnce Engineering team has advised to stall all upgrades until further notice. We are working on a resolution and will keep you informed on the same."

Hope the StoreOnce version is stable and released in time for the Veeam V12 version.

Regards,
SSA

[foggy]

Hi Sergio, I believe it is one of the purposes of the HPE controlled release approach - to be able to withdraw in case any issues are caught during this stage and address them in the GA version.

[Luca82]

is there any news about immutability support on storeonce? Is it possible to integrate with soreonce in ISV-DI or SRV-DI

[foggy]

As mentioned above, v12 will support ISV managed Data immutability for StoreOnce.

[Regnor]

FYI; StoreOnce firmware 4.3.2 is now available and you can test immutability with the v12 beta.
However I wasn't able to get it working out of the box and need to look at it again when I have more time.

[mike_stewart]

Good morning,
A few weeks ago, I had the opportunity to test the new StoreOnce Firmware v4.3.2 with Veeam v12 Beta and the immutability works as expected.
The only thing is that the immutability is configured in repository settings and not per Job. So, all Jobs that use this repository will have the same immutability settings.
Still waiting for the final version and validate if there will be changes in this settings.

SSA

[veremin]

The immutability will be a repository setting if that's what you are confirming. Thanks!

[foggy]

The concept here is more or less similar to Linux Hardened Repository.

[Luca82]

Hi everyone, is there any news about integration with storeonce immutability?

[Regnor]

Immutability for StoreOnce will be shipped with v12.

[Regnor]

I played a bit with the immutability on StoreOnce systems and noticed something which is, in my opinion, not working as expected. I did alter the date on the VBR server to a point where the immutability would have been expired. Afterwards I deleted the backups inside Veeam, which finished with a warning as the immutability is controlled by the StoreOnce itself. The odd part is, that afterwards the restorepoints are no longer available in Veeam (not under Disk, imported, orphaned) and a rescan only shows "skipped" backups. Via the StoreOnce GUI I could verify that the backup file is still there and only the VBM is gone, which is probably not made immutable as it needs to be writeable.
So the question is, how can you re-import the backups at that point?

[foggy]

Hi Max, there's currently no UI capability for the import of backups from the Catalyst repository. There's a PS cmdlet that allows that but we've recently identified an issue with its operation that we're going to address further down the road. We have no confidence if the issue is a global one or the cmdlet doesn't work in some specific cases only, so you can give it a try (unless your question is theoretical).

[Regnor]

Hi Alexander, well it's half theoretical
I was just trying out some scenarios in my lab to get a better feeling about the implementation of immutability with the StoreOnce.
With changing the clock on the backup server I wanted to find out if the immutability can be circumvented and so stumbled over this 'issue'.

I tried to use the Powershell cmdlet you've linked.
An import in the same configuration database wasn't possible as Veeam complained that the files already existed; though they weren't listed anywhere.
Importing in a new database succeeded but I can't restore anything from the VBK as it errors out with a credentials error.

As it's only my lab and has no priority, I won't open a case. But maybe you can reproduce this behaviour and see if it's a general issue.
I would say most important for me or any StoreOnce user is, that in such a case the backup files are still usable and can be imported.
So if it's only a problem with the current implementation of the cmdlet, but Veeam Support will be able to restore the files, then it's ok in my view.

[foggy]

Another option is to copy the files out of the Catalyst Store via the Files node to any regular repository and import from there.

We will definitely look into the behavior you're seeing. Could you please elaborate on the particular error message you got upon the restore?

[Regnor]

Accessing the VBK via the files tab worked. I could copy it locally, import it and do a file level restore. Didn't event think about this, thanks.

So in the existing backup configuration I got this one during the import:

Failed to import backup Error: Backup 'storeonce://10.9.0.137:test2@/|Agent_VBRServer_StoreOnce2|localhost|Agent_VBRServer_StoreOnce2_-_localhostD2023-04-15T171143_7D0D.vbk' already exists.

In the new configuration the import succeeded but FLR failed with this error:

Failed to create backup mount source. Credentials for object store 'test2' on storage '10.9.0.137' was not found.
Failed to open storage for read access. Storage: [10.9.0.137:test2@/\Agent_VBRServer_StoreOnce2\localhost\Agent_VBRServer_StoreOnce2_-_localhostD2023-04-15T171143_7D0D.vbk].
Failed to restore file from local backup. VFS link: [summary.xml]. Target file: [MemFs://frontend::CDataTransferCommandSet::RestoreText_{40c0d0f5-8286-4fe4-9538-8db84e406249}]. CHMOD mask: [0].
Agent failed to process method {DataTransfer.RestoreText}. Error: Failed to create backup mount source. Credentials for object store 'test2' on storage '10.9.0.137' was not found.

I'll try to build a new lab in the next time to rule out that I've broken something during my tests.

[Regnor]

So, I did deploy a new StoreOnce VSA and started with a fresh Veeam configuration database on the same server.
This time the Import-VBRBackup succeeded after deleting the Backups in the Veeam Console.
The FLR still fails with the Credentials was not found error from above.

[foggy]

The restore part is not clear to me. Can you maybe open a case for a closer look?

[Regnor]

Yes of course. I will open a case next week and share the case number here.

[Regnor]

@Alexander: I just opened a case #05973755 (it's a free/CE case as I couldn't select NFR).

[Luca82]

Is there any other partner that support isv immutability with veeam? Or just storeonce?

[Gostev]

1. Veeam's own Hardened Repository (you can use any general purpose server).
2. There are loads of paid and free S3-compatible object storage systems (see the Compatible + Immutability section).
3. EMC DataDomain ISV immutability support is coming in the next minor release before the end of this year.