Monitoring and reporting for Veeam Backup & Replication, VMware vSphere and Microsoft Hyper-V in a single System Center Operations Manager Console
Post Reply
sandsturm
Veteran
Posts: 278
Liked: 23 times
Joined: Mar 23, 2015 8:30 am
Contact:

Table VeeamBackupMP.SessionTaskStaging is growing fast

Post by sandsturm »

Hi all

After we troubleshooted SCOM infrastructure due to a full disk on the database server, we found that the table VeeamBackupMP.SessionTaskStaging is huge and has been growing very fast recently. In one column of this table are XML files (2MB per XML file) stored and that's the reason why the table is so big. Does anybody know, what the usage of this table is? What about removing older entries in this table? Or does anyone know how large this table will grow?

thx,
sandsturm
Alec King
VP, Product Management
Posts: 1445
Liked: 362 times
Joined: Jan 01, 2006 1:01 am
Contact:

Re: Table VeeamBackupMP.SessionTaskStaging is growing fast

Post by Alec King »

Hi Sandsturm,
Sorry you've experienced this problem. This DB table holds data required for our Veeam Backup MP reports. It can grow, depending on activity on the Backup server, and if SCOM is able to process incoming data in a timely manner.

We introduced a self-healing feature in Veeam MP 8.0 Update 5, which will detect processing errors, and automatically truncate the table by 10%. You can see details in the U5 Release Notes
If you do not yet have update 5, I recommend you install it (it is a relatively lightweight update - MP files only, no need to update our backend collection or UI)

However as the table in your DB is already very large, Update 5 alone may not be enough to solve it.
Please check this Appendix in our User Guide for details on this problem.
Here also are some additional notes to help with the troubleshooting process above -

For VeeamBackupMP.SessionTaskStaging the schemaName will be ‘VeeamBackupMP’. Also, to avoid the re-growth of the table, I would suggest to check the MaxRowsToProcessPerTransactionCount value in StandardDatasetStagingArea table for our dataset

Code: Select all

select MaxRowsToProcessPerTransactionCount from StandardDatasetStagingArea WHERE DatasetId = (SELECT DatasetId FROM StandardDataset WHERE SchemaName='VeeamBackupMP')
go
and decrease that value manually by 2 or 3 times.

Code: Select all

UPDATE StandardDatasetStagingArea
             SET                 
                    MaxRowsToProcessPerTransactionCount = [value]
             WHERE DatasetId = (SELECT DatasetId FROM StandardDataset WHERE SchemaName='VeeamBackupMP')       
go
It should speed up the data grooming.

Hope above helps - any questions please let us know. Thanks!
Alec King
Vice President, Product Management
Veeam Software
sandsturm
Veteran
Posts: 278
Liked: 23 times
Joined: Mar 23, 2015 8:30 am
Contact:

Re: Table VeeamBackupMP.SessionTaskStaging is growing fast

Post by sandsturm »

Hi Alec

Thanks for your fast reply. We increased the value of MaxRowsToProcessPerTransactionCount from 10'000 to 30'000 and installed the latest release of the management pack (v8 update 5). But this did not have the desired effect. The table is still growing...

anything else we can do?

thx,
sandsturm
Alec King
VP, Product Management
Posts: 1445
Liked: 362 times
Joined: Jan 01, 2006 1:01 am
Contact:

Re: Table VeeamBackupMP.SessionTaskStaging is growing fast

Post by Alec King »

Hi Sandsturm,

Actually the recommendation was to DECREASE the MaxRowsToProcessPerTransactionCount. Not increase.

This is because high values can cause timeout of the process with overlapping from previous processes. These timeouts/failures cause the table to grow.
The self-healing process in MP update 5 will only decrease by 10% for each error and with your new starting value of 30,000 this will probably not work.
I suggest you set the value to 5,000 and see if the self-healing process will start to work.

The growth of this table could indicate an event storm from Veeam Backup. The trigger events happen when restore points are created/updated/deleted, in Veeam Backup eventlog on the Veeam Backup server. There are events with IDs – 10010/10090/10050. There are other events but these are most common. Multiple events are expected, however can you check how many events you have at peak times? (when backups are running)

Can you also check please for events 31551 & 31552? This events appear on SCOM Management Servers, OperationsManager eventlog.
Event IDs are 31551 and 31552, and they should contain internal Dataset name in Param[3], in our case it should be “Veeam.Backup.Dataset”.
(Event ID 31554 with same dataset name inform that all is OK)

Also, do you have an Enterprise PLUS license for MP (i.e. includes Reporting); and are you using the Veeam MP for Backup reports?
If you are not using the reports, we can disable the MP rules that populate the dataset. Then this table will not be required at all.

By the way, you may at anytime open a support case, and they will assist you to check and gather all the required data. I'd suggest you also open a case, and post the case ID here.
Thanks!
Alec King
Vice President, Product Management
Veeam Software
sandsturm
Veteran
Posts: 278
Liked: 23 times
Joined: Mar 23, 2015 8:30 am
Contact:

Re: Table VeeamBackupMP.SessionTaskStaging is growing fast

Post by sandsturm »

Hi Alex

Shame on me that I've overlooked "decrease" and increased the value instead :-). After we decreased the value we opened a support case and last night we could finally solve the problem. We had to run a stored procedure to solve the problem. We did the following steps:

1. Configure an override to disable the maintenance procedure for the dataset:
a. Open the Ops Mgr console using the account with administrative rights. Go to the Authoring section.
b. Navigate to Rules and look for Standard Data Warehouse Data Set maintenance rule.
c. Right-click the Standard Data Warehouse Data Set maintenance rule and select Overrides > Override the Rule > For a specific object of class: Standard Data Set.
In the Override Properties window, select the check box next to the Enabled parameter, change the Override Value to False, and click Apply to save changes.
This will prevent dataset maintenance from running automatically for the given dataset type.

2. Restart the System Center Management service on the RMS to disable all running maintenance procedures.
3. Wait 10 minutes, then connect to the SQL server that hosts the Ops Mgr DW DB and open SQL Server Management Studio.
Run the following query:

USE [OperationsManagerDW]
DECLARE @DataSet uniqueidentifier
SET @DataSet = (SELECT DatasetId FROM StandardDataset WHERE SchemaName = 'VeeamBackupMP')
EXEC StandardDatasetMaintenance @DataSet

The query could take several hours to complete depending on how much data has been flooded to the warehouse. Do not stop the query until it completes.
4. Once the query completes, delete the override configured at step 1.


The table VeeamBackupMP.SessionTaskStaging had 1.9Mio rows and a size of 2TB in our case. The above stored procedure stopped unexpectedly several times after some hours, then we increase the Datawarehouse servers memory (and the memory allocated for the MSSQL DB) to 96GB and the procedure finished after 9 hours successfully.
Alec King
VP, Product Management
Posts: 1445
Liked: 362 times
Joined: Jan 01, 2006 1:01 am
Contact:

Re: Table VeeamBackupMP.SessionTaskStaging is growing fast

Post by Alec King »

Hi Sandsturm,
Thanks for the update!
And thanks also for posting the instructions here, hope they'll be useful if other customers encounter this problem.
Glad your system is back up and running now. And with the new self-healing procedures in MP Update 5, this table should not grow so large in future.
Thanks!
Alec
Alec King
Vice President, Product Management
Veeam Software
Post Reply

Who is online

Users browsing this forum: No registered users and 5 guests