FR: sort Resourcename in RTS log

[mjr.epicfail]

Hi all,

Minor FR here, but as a CloudConnect provider I often use the "RTS.ResourcesUsage.log" log to monitor active connections / sessions between tenants and the repositories and or other Veeam Infrastructure.
I noticed that there is no particular sorting in the output when usage is collected / changed since last interval.
Is it possible to sort (ascending) the resource name?

[HannesK]

Hello,
I guess we should offer a better way of monitoring than parsing the RTS log in general instead of trying to make RTS log "public consumable".

What is the goal you like to achieve? Monitoring the tasks is one thing, but what is the purpose of it? The "real goal"?

Thanks,
Hannes

[mjr.epicfail]

Hi Hannes,

Thank you for your reply.
I'm using this log file because in CloudConnect the running tasks in the VBR console is slow and inaccurate.
The RTS logs gives me (almost in realtime) the consumption of resources.
I'm using this to check for load issues and for CloudConnect MaintenanceMode to find out which tenants are still connected.

Side issue, MaintenanceMode is not honered for restore actions, restores can still be started while CloudConnect is in MaintenanceMode.

[david.domask]

Hi @mjr.epicfail,

Can you elaborate on this part a bit?

the running tasks in the VBR console is slow and inaccurate.

Slow I think can be reviewed by the Support team to understand the behavior a bit, but can you explain a bit more on the inaccurate part with an example if possible?

As for this:

Side issue, MaintenanceMode is not honered for restore actions, restores can still be started while CloudConnect is in MaintenanceMode.

https://helpcenter.veeam.com/docs/backu ... ce&ver=120

Check the blue Note box on this page for further elaborations on Maintenance Mode and tenant restores. Can you elaborate on the workflow you're trying to achieve that is interrupted by tenants being able to perform restores while in Maintenance Mode? I can guess of course but if you can just share your intended workflow a bit we can help advise better.

[mjr.epicfail]

Hi David,

Yes i can elaborate on this part.
For as long as I have been working with Cloud Connect, the SP side of the running task has been slow (probably because we have hundreds and hundreds of tenants configured). Restarts of the server or the SQL server do not improve the speed.
The information displayed on the SP side can be very limited to only a jobname, sometimes there is more info like the task that are running in the jobs but not really anything else like the rich information of the running task if you are on a tenant VBR console.
The RTS log gives me some information about which backupresources (in my case only SOBR extents) are in use and how many task are running on them, and I can see which tenants are still using the resources. This helps me speed up patch installation and or reboots of the extents.

Maintenancemode:
Thank you for the link and the clarification.

2. Prevent tenant jobs from starting.
If a tenant starts a new job session at the time when the SP backup server is operating in the Maintenance mode, the job will fail.

There are edge cases the maintenancemode is ignored for some jobs, I did see some copy jobs fail and then to just start up again and start copying data to the SP side. Ill create a logbundle when I see this happening again.

The helpcenter page starts with:

In some cases, the SP may need to perform service actions with the SP backup infrastructure, for example, upgrade a server whose resources are consumed by tenant VM backups and replicas. Such operations may require that the SP cloud resources become temporarily unavailable to tenants and tenant activities are temporarily put on hold. To make the SP environment ready for maintenance, the SP can put its backup server to the Maintenance mode.

And in the blue note it states:

NOTE
Consider the following:

When the SP backup server is operating in the Maintenance mode, the tenant can access backups created in the cloud repository, for example, restore data from such backups. Thus, the SP should not use the Maintenance mode functionality to cease tenant activities before moving tenant backups to another cloud repository. The SP should disable a tenant prior to performing operations with tenant backups.

Does that mean if i want to reboot all the extents for patches, i need to disable all the tenants first to make sure that no backups are being accessed and or restored by the tenant?
I always thought that maintenance mode was just that, make things off limits to tenant so the SP can work on the infrastructure while not interrupting jobs and or other actions.

In our case disabling hundreds of tenants would be long wait till all tenants are being disabled, and because we are using VSPC, the tenants can enable their own users again. (some resellers don't really read maintenance notifications)

But maybe this maintenance bit should be a different topic.

[david.domask]

Hi Danny,

For as long as I have been working with Cloud Connect, the SP side of the running task has been slow (probably because we have hundreds and hundreds of tenants configured). Restarts of the server or the SQL server do not improve the speed.

I'm going to guess you've had a case or two on this, if so do you remember the case number so I can check the outcome on those cases? If you haven't had a chance for a case yet for the slowness, I think it's worth letting Support take a look. I would reproduce the issue, note the date and time of the reproduce (I know it's just checking the UI, but even a screen recording quick of the behavior with the timestamps visible will help to narrow the investigation), and then provide logs from the SP Server.

The information displayed on the SP side can be very limited to only a jobname, sometimes there is more info like the task that are running in the jobs but not really anything else like the rich information of the running task if you are on a tenant VBR console.

Got it, so sounds like more ideal would be to see if there's a better way to display the activity for Providers. What information specifically would you hope to see here?

There are edge cases the maintenancemode is ignored for some jobs, I did see some copy jobs fail and then to just start up again and start copying data to the SP side. Ill create a logbundle when I see this happening again.

Would be great just to confirm the behavior if you can spot it. A few quick tests in my lab and I could not reproduce such a behavior, but indeed best to open a case once you spot it and get the necessary logs from both Provider and Tenant side.

Does that mean if i want to reboot all the extents for patches, i need to disable all the tenants first to make sure that no backups are being accessed and or restored by the tenant?
I always thought that maintenance mode was just that, make things off limits to tenant so the SP can work on the infrastructure while not interrupting jobs and or other actions.

Ideally it is best to disable the tenants to prevent error messages should they miss the message for the Maintenance Mode, but it will indeed be best to inform Tenants about the upcoming maintenance window in advance, then perform the patching while in Maintenance Mode.

[mjr.epicfail]

Hi David,

1) Thank you for your response and taking the time for this.
I don't have any cases about this, i have had several conversations about this during other cases and support engineers always told me this is what it is.
I'll check if I can make some screenrecordings. Is there any way to get them to you without broadcasting this over the forum?

2) A summary of what the tenant sees with some information for the SP. (on the top of my mind)
- Jobname
- objects in jobs
- current state of job (running backups of objects and or health check or merge etc etc)
- per processed object
- current state in process
- percentage done
- target extent (single extent or extent in SOBR)
And show this information for all jobs, even for jobs that are being processed by tenant VBR's that are not managed by the VCC organization. I understand that there is some privacy involved here, but I can read most of the information in the VBM file itself if I really wanted to know things about the backups.

3) I will if I encounter it again, I normally run maintenance every month or so (windows updates and or Veeam updates).

4) Yeah, that's not working for us, 1) tenants can enable them via the VSPC. 2) did you ever try to collect all tenants and admin state and then run a disable on about 500 - 1000 tenants and afterward revert that change? That maybe works on a small scale but not with these numbers.

[david.domask]

Hi Danny,

1. I think you can just Private Message me a secure link and I'll take a look
2. Thank you for this, let's consider it an enhancement request and I'll handle registering it
3. ++
4. Understood, and again, I suppose let us consider it an enhancement request, as I know that the Maintenance Mode has been receiving improvements over various releases, and I get the use case.

[mjr.epicfail]

Hi David,

I have sent you a PM