SOBR with S3 Object Capacity Tier Immutability Question

[Lewpy]

For the first time, I have had to reduce the number of restore points in an SOBR that includes local ReFS storage and an S3 Object Storage with Immutability set.
We wanted to enable encryption in Veeam end-to-end, and this requires making completely new backup chains (full backups).
Local disk space was tight, so we needed to reduce the restore points to free up some space.
I initially thought "this is not problem, as the restore points will just show in the Capacity Tier section of Veeam" - but I found out this is not how it works.
I was a bit startled to see the restore points vanish from both local and S3 Object Storage
Particularly when the GUI stops me deleting orphaned restore points in the S3 Object Storage until the immutability period is over.
This kicked me to actually RTFM, and I came across this section https://helpcenter.veeam.com/docs/backu ... ml?ver=120
So although the restore points appear to be gone, there are checkpoints that can be rolled back to that would then contain the missing restore points
Great, so I then issued the command to examine the checkpoints .... and it sat there ... a while ... before just generating an error and it does this each time.
I've timed the command, and it errors after exactly 15 minutes, which feels like some kind of timeout somewhere?

Code: Select all

PS C:\Users\someadmin> Connect-VBRServer -Server localhost
PS C:\Users\someadmin> $repository = Get-VBRBackupRepository -ScaleOut
[Time Now: 27/03/2024 18:23:49]
PS C:\Users\someadmin> Get-VBRCapacityTierSyncInterval -Repository $repository
Get-VBRCapacityTierSyncInterval : Connection to 'Veeam Backup Service' is not available, host 'localhost', port '9392'
At line:1 char:1
+ Get-VBRCapacityTierSyncInterval -Repository $repository
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Get-VBRCapacityTierSyncInterval], CAppException
    + FullyQualifiedErrorId : Veeam.Backup.Common.CAppException,Veeam.Backup.PowerShell.Cmdlets.GetVBRCapacityTierSyncInterval

That took: 00:00:15:00 [Time Now: 27/03/2024 18:38:49]
PS C:\Users\someadmin>

Note: I removed the code that spat out the timestamps for clarity

If I run the command again by itself, I just get

Code: Select all

PS C:\Users\someadmin> Get-VBRCapacityTierSyncInterval -Repository $repository
Get-VBRCapacityTierSyncInterval : Cannot process argument transformation on parameter 'Repository'. Remote connection was terminated. Either re-establish remote connection or use local one, ex. Connect-VBRServer -Server localhost
At line:1 char:45
+ Get-VBRCapacityTierSyncInterval -Repository $repository
+                                             ~~~~~~~~~~~
    + CategoryInfo          : InvalidData: (:) [Get-VBRCapacityTierSyncInterval], ParameterBindingArgumentTransformationException
    + FullyQualifiedErrorId : ParameterArgumentTransformationError,Veeam.Backup.PowerShell.Cmdlets.GetVBRCapacityTierSyncInterval

So my entire PowerShell connection is dropped.

Is there something I am doing wrong, or a timeout somewhere that needs tweaked?
At the moment, this puts a big "risk" sign over the SOBR and immutable store, when it was believed to be a nice safe vault

Veeam B&R 12.1.1.56 (just recently patched last week to the latest version)

Thanks,
Lewis.

[HannesK]

Hello,

$repository = Get-VBRBackupRepository -ScaleOut

do you only have one SOBR? otherwise I would add the -Name parameter

Get-VBRCapacityTierSyncInterval : Connection to 'Veeam Backup Service' is not available, host 'localhost', port '9392'

that sounds like the connection to the backup service itself fails and nothing was even touched in object storage / capacity tier. Can you run any VBR cmdlet successfully?

I was a bit startled to see the restore points vanish from both local and S3 Object Storage

sounds like you are in "copy" mode. "copy" means that everything you do in performance tier will be replicated to capacity tier

At the moment, this puts a big "risk" sign over the SOBR and immutable store, when it was believed to be a nice safe vault

which risk? immutable data is immutable and can be restored once the PowerShell commands are fixed


Best regards,
Hannes

[Lewpy]

Hi Hannes,

do you only have one SOBR? otherwise I would add the -Name parameter

Yes, there is just a single SOBR, so the $repository variable just contains one "object".

that sounds like the connection to the backup service itself fails and nothing was even touched in object storage / capacity tier. Can you run any VBR cmdlet successfully?

I can run other VBR commands just fine, such as Get-VBRJob & and the aforementioned Get-VBRBackupRepository.

sounds like you are in "copy" mode. "copy" means that everything you do in performance tier will be replicated to capacity tier

Correct, the SOBR is in Copy mode, so this was me just understanding the process when immutability is in the equation.

which risk? immutable data is immutable and can be restored once the PowerShell commands are fixed

Immutable data is all fine as long as you can access it. At the moment, we can't access the immutable "checkpoints" so wouldn't be able to use the immutable functionality.
It's like the data is locked in a secure safe, but we don't have a key: it's safe, but unusable.
So it is a "Risk" until we can list the "checkpoints" to know we have the rollback options available.
If this is just a local PowerShell issue that can be resolved, then that is great
If it is an issue with the S3 Object Storage subsystem and the whole thing is just not going to work, then that is a problem we would need to address

Thanks,
Lewis.

[Lewpy]

Just to add to this: when I issue the command

Code: Select all

Get-VBRCapacityTierSyncInterval -Repository $repository

a new VeeamAgent.exe process is launched, and looking in Resource Monitor I can see that this new process is talking to the S3 Object Storage FQDN.
It is only using about 10,000 B/sec (as Resource Monitor displays it).

Once the PowerShell command "times out" after 15 minutes, the VeeamAgent.exe process stays running with the same level of network activity to the S3 Object Storage FQDN.

The Internet connection at this location is 10Gbps, and Resource Monitor shows a latency of about 10-20ms to the S3 Object Storage IP address.

[Lewpy]

Okay, the proverbial rabbit hole opened up before me, and I just couldn't resist jumping headfirst in [I nearly always do!]

So using Process Monitor, I watched the VeeamAgent.exe process that started when my PowerShell cmdlet was issued.
It stayed running for just under 40 minutes in total, long after the PowerShell cmdlet timed out.
I could see it talking to the S3 Object Storage FQDN, and also logging to

Code: Select all

"C:\ProgramData\Veeam\Backup\Utils\ArchiveCheckpointManagement.Gate.log"

When I looked inside that log file, I could see it enumerating/retrieving checkpoints.
I also noticed a log file next to it (but not written to by the same VeeamAgent.exe process)

Code: Select all

"C:\ProgramData\Veeam\Backup\Utils\ArchiveCheckpointManagement.Index.log"

Inside there, I could determine that each VM in the SOBR was having its checkpoints enumerated, and taking about 20 seconds (on average) for each VM. This varied based upon the number of checkpoints on the VM, plus general randomness probably down to S3 Object Storage command latency, etc.
Since there are about 128 VMs in the SOBR, this adds up to around 40 minutes of time: this matches the length of time that the VeeamAgent.exe was running for.

So it appears to be that the PowerShell cmdlet times out after 15 minutes, but the actually amount of time required is a factor of

Code: Select all

[Speed of enumerating S3 Object Storage] x [Number of VMs in SOBR] x [Number of Checkpoints on each VM in SOBR]

And in our case here, the PowerShell cmdlet times out first
I don't see a documented "timeout" parameter for Get-VBRCapacityTierSyncInterval.
Is there a way that we can increase the timeout ourselves by tweaking the Registry or by running another PowerShell cmdlet beforehand?
Or is this something that would need altered by Veeam internally?

Or am I completely on the wrong track?

Thanks,
Lewis.

P.s. It would be really nice if the Get-VBRCapacityTierSyncInterval provided feedback while running, such as Progress Bars as it iterates each VM and enumerates their checkpoints Just sitting their for 15 minutes makes you really wonder if it is doing anything at all.

[Lewpy]

P.s. It would be really nice if the Get-VBRCapacityTierSyncInterval provided feedback while running, such as Progress Bars as it iterates each VM and enumerates their checkpoints Just sitting their for 15 minutes makes you really wonder if it is doing anything at all.

Okay, having just been hampered by the Export-VBRRestorePoint being interactive and displaying Progress Bars, I am feeling this might not be a good idea after all
post515439.html#p515439
Unless it is implemented in such as way it can be disabled with a "-NonInteractive" parameter (or similar), so as not to mess up any pipelines, etc.

[veremin]

Hi, Lewis,

There is indeed a 15-minute timeout interval for calling our services on a remote server, after which the process fails. It appears that you have encountered this limitation.

The timeout can be adjusted by modifying the registry key "RemotingTimeout" (DWORD), so you can try experimenting with different values (should be set in seconds).

However, if you feel that the querying of the checkpoint is taking longer than expected, it might be beneficial to reach out to our support team. They can analyze the debug logs and determine the underlying cause of the process slowdown.

Thanks!

[Lewpy]

Hi Vladimir,

Thanks for the information
A quick search found this article https://www.veeam.com/kb2166
While not the same issue, it does detail the "RemotingTimeout" registry key.
I used that PS code (7200 / 2 hours is good enough for my purposes), then had to wait for a suitable pause in backup activity for a quick reboot, and all now looks good

Code: Select all

PS C:\WINDOWS\system32> Connect-VBRServer -Server localhost
PS C:\WINDOWS\system32> $repository = Get-VBRBackupRepository -ScaleOut
[Time Now: 03/04/2024 07:05:43]
PS C:\WINDOWS\system32> Get-VBRCapacityTierSyncInterval -Repository $repository

StartDateUtc        EndDateUtc
------------        ----------
<some date>         <a later date>


That took: 00:00:35:42 [Time Now: 03/04/2024 07:41:25]
PS C:\WINDOWS\system32>

And we feel less "at risk", now we can "see" the immutable data
That is going to be an important Reg modification to keep track of!

Thanks,
Lewis.