MinIO Immutability Unexpected empty versionId for block with id [0]

[mengl]

I recently moved my SOBR cpaycity tier to a new MinIO bucket in order to use immutability.

This was my procedure:
1. Put the old Capacity Tier in maintenance mode --> this instance was not using immutability
2. Create a bucket with object lock enabled in a new MinIO instance using the same name

Code: Select all

mc mb -l local/bucket1

3. Copied all files from old to new instance using mc client
4. Took over IP/Hostname
5. Edit Repository options and enabled immutability

Now the SOBR offload jobs throws this error:

Code: Select all

Unexpected empty versionId for block with id [0]

And in the log files:

Code: Select all

[26.11.2020 23:05:50] <31> Error    Unexpected empty versionId for block with id [0] (Veeam.Backup.Common.CCppComponentException)
[26.11.2020 23:05:50] <31> Error       in c++: Failed to UpdateBlocksImmutability in CloudArchRepo, backup/index id [71401d1f-5659-4b6f-a244-da620faf8918]/[61075436-2f77-4724-50ca-ef2f633b3793], inside folder path [/e1f70016-2b2c-41e3-9df0-c9d959a0553a (vm-3345)], item name [summary.xml], block count [1], immutable till [2020-12-22 10:48:54 UTC]
[26.11.2020 23:05:50] <31> Error       in c++: Failed to update immutability for [124bcb93-632d-b5b3-b950-e231fc54b6ad:41].
[26.11.2020 23:05:50] <31> Error       in c++: Failed to update immutability for storages.
[26.11.2020 23:05:50] <31> Error       in c++: Failed to process method 'ArchRepo.IncreaseImmutabililtyForStorages'
[26.11.2020 23:05:50] <31> Error       at Veeam.Backup.Common.CVcpInvoker.Invoke(CVcpStreamBase stream, String command, CVcpCommandArgs inArgs, Boolean noLog)
[26.11.2020 23:05:50] <31> Error       at Veeam.Backup.AgentProvider.CClientAgentProtocol.Invoke(String command, CVcpCommandArgs inArgs, Boolean noLog, Boolean agentNoLog)
[26.11.2020 23:05:50] <31> Error       at Veeam.Backup.AgentProvider.CBackupClient.Invoke(String command, CVcpCommandArgs inArgs, Boolean noLog)
[26.11.2020 23:05:50] <31> Error    --- End of stack trace from previous location where exception was thrown ---
[26.11.2020 23:05:50] <31> Error       at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
[26.11.2020 23:05:50] <31> Error       at Veeam.Backup.Common.ExceptionFactory.ThrowNecessaryAggregateException(IEnumerable`1 exceptionsCollection)
[26.11.2020 23:05:50] <31> Error       at Veeam.Backup.AgentProvider.CBackupClientImpl.ConstructException(Exception exception, String error)
[26.11.2020 23:05:50] <31> Error       at Veeam.Backup.AgentProvider.CBackupClient.ConstructException(Exception ex, String error, Object[] args)
[26.11.2020 23:05:50] <31> Error       at Veeam.Backup.AgentProvider.CBackupClient.OnInvokeError(Exception e, String command, CVcpCommandArgs inArgs)
[26.11.2020 23:05:50] <31> Error       at Veeam.Backup.AgentProvider.CBackupClient.Invoke(String command, CVcpCommandArgs inArgs, Boolean noLog)
[26.11.2020 23:05:50] <31> Error       at Veeam.Backup.AgentProvider.CBackupClient.ArchRepoIncreaseImmutabilityForStorages(String archRepoId, String backupId, String indexId, CArchiveTierStgVersionsInfo stgs)
[26.11.2020 23:05:50] <31> Error       at Veeam.Backup.Core.ArchiveTier.SArchiveImmutabilityManager.UpdateImmutabilityForPreviousStorages(CStorage storage, CBackupArchiveIndex index, Guid archiveExtentId, CRepositoryAccessor repositoryAccessor, CArchiveBackupAgents agents)
[26.11.2020 23:05:50] <31> Error       at Veeam.Backup.Core.ArchiveTier.CCopyStorageDehydrateAlg.DehydrateStorage(CArchiveBackupAgents agents, CArchiveIndexAgent indexAgent, CArchiveBackupCheckpointManager checkpointManager, CArchiveBackupStorageTask storageTask, CBackupArchiveIndex index, Int32 compressionLevel, ITaskSession taskSession, IStopSessionSync stopControl)
[26.11.2020 23:05:50] <31> Error       at Veeam.Backup.Core.ArchiveTier.CArchiveBackupTaskPerformer.DehydrateStg(CBackup backup, CArchiveBackupStorageTask storageTask, CBackupArchiveIndex index, CStorageArchivingLock storageLock, IStorageDehydrateAlg storageDehydrateAlg, CBackupTaskSession taskSession, IStopSessionSync stopControl)
[26.11.2020 23:05:50] <31> Error       at Veeam.Backup.Core.ArchiveTier.CArchiveBackupTaskPerformer.Perform(CBackup backup, CArchiveBackupStorageTask storageTask, CBackupArchiveIndex index, IStorageDehydrateAlg storageDehydrateAlg, CBackupTaskSession taskSession, IStopSessionSync stopControl, CTaskSchedulerSessionLogger logger)
[26.11.2020 23:05:50] <31> Error       at Veeam.Backup.Core.ArchiveTier.CArchiveBackupTaskPerformer.Perform(CArchiveBackupStorageTask storageTask, CBackup backup, CBackupArchiveIndex index, CBackupTaskSession taskSession, IStopSessionSync stopControl, CTaskSchedulerSessionLogger logger)
[26.11.2020 23:05:50] <31> Error       at Veeam.Backup.Core.ArchiveTier.CArchiveBackupTaskPerformer.Perform(CArchiveBackupChainTask task, CBackupTaskSession taskSession, ILogRecordHandler log, CStorageSet storageSet, IStopSessionSync sessionControl, CDisposableLock& indexLock, IDisposable& backupLock)
[26.11.2020 23:05:50] <31> Error       at Veeam.Backup.Core.ArchiveTier.CArchiveBackupTaskPerformer.PerformCycle(CArchiveBackupChainTask task, CBackupTaskSession& taskSession, ILogRecordHandler log, CStorageSet storageSet, IStopSessionSync sessionControl)
[26.11.2020 23:05:50] <31> Error       at Veeam.Backup.Core.ArchiveTier.CArchiveBackupTaskPerformer.Perform(CArchiveBackupChainTask task, IStopSessionSync sessionControl)
[26.11.2020 23:05:50] <31> Error       at Veeam.Backup.Core.ArchiveTier.CArchiveBackupTaskPerformer.PerformSafe(CArchiveBackupChainTask task, IStopSessionSync sessionControl)

Can immutability be activated afterwards?
Could this be caused by the movement?
Or is this just a MinIO related Bug (Version is RELEASE.2020-11-25T22-36-25Z)?

[HannesK]

Hello,
What about doing a normal S3 migration (copy from old bucket to new bucket) and check that everything works before enabling immutability? Because S3 to S3 bucket migrations work in general.

I would also just remove the old bucket from SOBR and add a new one in step 4. Changing IP / Hostname of the object storage is something that could also be the reason.

I don't know which MinIO bug you mean, but I recommend to contact MinIO support for that question.

Best regards,
Hannes

[mengl]

I tried to remove and add the S3 repository as suggested but nothing changed.
Disabling immutability in Veeam when using a bucket with object lock enabled seems to be impossible - it threw an error when this box was not checked.

Finally I had to remove the S3 repository again and started over with a completely new and empty bucket.

This is a lab environment but after that I'm afraid to move a repository in production..

[HannesK]

probably I did not write it clear enough. I suggested a clean migration

- put capacity tier into maintenance mode
- copy all data with an S3 copy tool
- remove old bucket
- add new bucket
- check that everything works
- enable immutability

sure, I also would not put that kind of migration you described before into production