Comprehensive data protection for all workloads
Post Reply
manuel.aigner
Enthusiast
Posts: 31
Liked: 5 times
Joined: Sep 25, 2017 8:25 am
Full Name: Manuel Aigner
Contact:

Backup Server: Enormous SQL-Workload every 5min

Post by manuel.aigner »

Hi there,

since v10 we're experiencing some strange workload spikes every 5min on SQL-Server (which is all in one an backup server). After some performance tracking I found the query which causes this. (see below).
It seems to me it have to do something with SQL-AlwaysOn Backups. Is there a possibility to optimize the query a little bit. Our backup server has assigned 16vCores and 96GB of RAM. But no problem, this query runs longer than 1min at nearly 100% CPU-Usage and that approx. all 5min.
To encounter the performance drop on the whole machine I limited SQL-Server to 8vCPUs (one of two available NUMA-Nodes) and reduced "Max degree of parallelism" to 6. But this is only a workaround.

Here the figured out query:

Code: Select all

/*
This query text was retrieved from showplan XML, and may be truncated.
*/
WITH sqloibs_groups
AS (
	SELECT groups.group_id
		,sqloibs.obj_id
	FROM [Backup.Model.AlwaysOnGuestDatabases] dbs
	LEFT JOIN [Backup.Model.AlwaysOnGroups] groups ON dbs.group_id = groups.group_id
	INNER JOIN dbo.[Backup.Model.SqlOIBs] sqloibs ON dbs.group_db_id = sqloibs.group_db_id
	GROUP BY groups.group_id
		,sqloibs.obj_id
	)
	,sqloibs_count_on_stg
AS (
	SELECT b.id AS logbackup_id
		,sqloibs.obj_id AS object_id
		,COUNT(sqloibs.id) AS oibs_count
		,MAX(sqloibs.creation_time) AS creation_time
		,MAX(sqloibs.usn) AS usn
	FROM dbo.[Backup.Model.SqlOIBs] sqloibs
	INNER JOIN dbo.[Backup.Model.Storages] stg ON stg.id = sqloibs.stg_id
	INNER JOIN dbo.[Backup.Model.Backups] b ON stg.backup_id = b.id
	GROUP BY sqloibs.stg_id
		,sqloibs.obj_id
		,b.id
	)
	,points_count
AS (
	SELECT DISTINCT log_backup.id AS id
		,oibView.backup_id AS parent_id
		,s_stg.object_id AS object_id
		,MAX(oibView.secondary_vol) AS sec_vol
		,sqloibs_groups.group_id
		,(
			CASE 
				WHEN MAX(oibView.storage_backup) = 1
					THEN (
							SELECT SUM(oibs_count)
							FROM sqloibs_count_on_stg s
							WHERE s.logbackup_id = id
								AND s.object_id = s_stg.object_id
							)
				ELSE (
						SELECT SUM(oibs_count)
						FROM sqloibs_count_on_stg s
						WHERE s.logbackup_id = id
						)
				END
			) AS points
		,MAX(s_stg.creation_time) AS creation_time
		,MAX(s_stg.usn) AS usn
	FROM sqloibs_count_on_stg s_stg
	INNER JOIN dbo.[Backup.Model.Backups] log_backup ON s_stg.logbackup_id = log_backup.id
	INNER JOIN [dbo].[DbAggregate.OibsView] oibView ON log_backup.parent_backup_id = oibView.fake_b_id
	LEFT JOIN sqloibs_groups ON s_stg.object_id = sqloibs_groups.obj_id
	GROUP BY oibView.backup_id
		,log_backup.id
		,oib_object
		,group_id
		,s_stg.object_id
	)
	,group_filer
AS (
	SELECT id
		,parent_id
		,points
		,usn
		,creation_time
		,sec_vol
		,ROW_NUMBER() OVER (
			PARTITION BY group_id ORDER BY group_id
			) AS in_same_sql_group
	FROM points_count
	WHERE group_id IS NOT NULL
	)
SELECT t.id
	,t.parent_id
	,t.points
	,MAX(t.usn) AS usn
	,MAX(t.creation_time) AS creation_time
	,MAX(sec_vol) AS sec_vol
INTO #tmp
FROM (
	SELECT id
		,parent_id
		,points
		,usn
		,creation_time
		,sec_vol
	FROM points_count
	WHERE group_id IS NULL
	
	UNION
	
	SELECT id
		,parent_id
		,points
		,usn
		,creation_time
		,sec_vol
	FROM group_filer
	WHERE in_same_sql_group >= 1
	) t
GROUP BY t.id
	,t.parent_id
	,t.points
PetrM
Veeam Software
Posts: 3264
Liked: 528 times
Joined: Aug 28, 2013 8:23 am
Full Name: Petr Makarov
Location: Prague, Czech Republic
Contact:

Re: Backup Server: Enormous SQL-Workload every 5min

Post by PetrM »

Hi Manuel,

Looks like that the pasted query represents an SQL stored procedure which retrieves a list of AlwaysOn databases and sorts them by groups.
However, we should collect more information about this behavior before we can make any kind of conclusions regarding possible optimizations.

It's not something that we can fix quickly over forum posts so please open a support request and ask our engineers to take a look at extended debug logs in order to understand what exactly triggers this query at the level of business logic and are there any possibilities to optimize the query itself?

By the way, I would recommend to make sure that the same issue persists after installing this patch (if it's not installed yet) prior to contact our support team.

Thanks!
manuel.aigner
Enthusiast
Posts: 31
Liked: 5 times
Joined: Sep 25, 2017 8:25 am
Full Name: Manuel Aigner
Contact:

Re: Backup Server: Enormous SQL-Workload every 5min

Post by manuel.aigner » 1 person likes this post

Hi Petr,

thanks for reply. We will install CU1 as soon as corona lock down is over.
If the "Problem" persists I will open a SR.
Gostev
Chief Product Officer
Posts: 31561
Liked: 6725 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Backup Server: Enormous SQL-Workload every 5min

Post by Gostev »

Most SQL optimizations are included in CP2, which should be released next week.
manuel.aigner
Enthusiast
Posts: 31
Liked: 5 times
Joined: Sep 25, 2017 8:25 am
Full Name: Manuel Aigner
Contact:

Re: Backup Server: Enormous SQL-Workload every 5min

Post by manuel.aigner »

Hi together,

in the meanwhile there is an open case. We found out this query is part from SP AggregateSqlPointsInfo.
There are two things:
1. yes, the query is not optimal, if the corresponding data becomes too big.
2. Our configuration for SQL-Transaction Backup is a little bit too much.
We have at least 50 RPs for each VM + Active Full overhead and we do TLog-Backups every 15min and keep them up until the corresponding full disappears.
This means at least 4800 RPs for each VM. We have around 200 SQL-VMs. So there are over 1mio. RPs in DB. Our DB is approx. 50GB.
Maybe this is the problem.
alexander sh
Veeam Software
Posts: 40
Liked: 28 times
Joined: Dec 13, 2012 8:52 am
Full Name: Alexander Shelemin
Location: Prague
Contact:

Re: Backup Server: Enormous SQL-Workload every 5min

Post by alexander sh »

Hey @manuel.aigner ! AggregateSqlPointsInfo procedure was significantly improved in CP2, which should be out any day now - so please install it and let us know if the problem persists. Thanks in advance!
manuel.aigner
Enthusiast
Posts: 31
Liked: 5 times
Joined: Sep 25, 2017 8:25 am
Full Name: Manuel Aigner
Contact:

Re: Backup Server: Enormous SQL-Workload every 5min

Post by manuel.aigner » 1 person likes this post

Hi Alexander,
of course I got a new version of that SP from support. Probably it's the same one as in CP2. It looks like much better. The Backup Server load has moved a little bit from CPU to storage backend. Because instead of one massive common table expression there are some select into temp tables for intermediate results.
Now this query takes 5-7 sec instead of 50-90 sec.
Post Reply

Who is online

Users browsing this forum: Google [Bot] and 68 guests