Comprehensive data protection for all workloads
Post Reply
dspjones
Enthusiast
Posts: 45
Liked: 17 times
Joined: May 04, 2012 2:51 pm
Contact:

Veeam (v9) Negatively Affecting SQL Databases

Post by dspjones »

case number is 01714513


I was alerted this morning by our DBA that veeam backup is freezing I/O on our main sql database cluster when it runs. We backup our environment 6x per day 12a/4a/8a/12p/4p/8p. The entire job consists of about 50 servers and takes just under 30 minutes to finish. The specific DB server in questions is completed within 2 minutes. But, as the SQL logs show and the user calls to the helpdesk bear out, every time the backup occurs, the sql database freezes just long enough to kick users out of the application that database is used for.

The first thing I did is turn off log truncation, selecting COPY ONLY. However it is still freezing the database. Next I have disabled application-aware processing altogether. The next job runs at 4pm today so I am still waiting to see if this change "resolves" the problem but in previous versions of veeam that we've used 6/7 and 8 this has not been a problem on any database system. Any ideas on whether this might be a bug or something we are doing wrong?
dspjones
Enthusiast
Posts: 45
Liked: 17 times
Joined: May 04, 2012 2:51 pm
Contact:

Re: Veeam (v9) Negatively Affecting SQL Databases

Post by dspjones »

On the SQL server in question, coinciding with each veeam backup job run is the following VSS error in the application log:

Volume Shadow Copy Service error: Unexpected error calling routine ConvertStringSidToSid(S-1-5-21-1374761507-110244007-91453608-13101.bak). hr = 0x80070539, The security ID structure is invalid.
.

Operation:
OnIdentify event
Gathering Writer Data

Context:
Execution Context: Shadow Copy Optimization Writer
Writer Class Id: {4dc3bdd4-ab48-4d07-adb0-3bee2926fd7f}
Writer Name: Shadow Copy Optimization Writer
Writer Instance ID: {41e94905-ff8f-4c1f-80ec-554752574ce1}
Gostev
Chief Product Officer
Posts: 31459
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Veeam (v9) Negatively Affecting SQL Databases

Post by Gostev »

We did not have similar reports from other v9 users, so this is likely an environmental issue that is not specific to v9. Generally speaking, v9 had no changes in SQL application-aware processing logic or its VSS integration.
dspjones
Enthusiast
Posts: 45
Liked: 17 times
Joined: May 04, 2012 2:51 pm
Contact:

Re: Veeam (v9) Negatively Affecting SQL Databases

Post by dspjones » 2 people like this post

DBA found the issue. Related to some change she made in her sql maintenance plan. In a nutshell it looks like she changed her job to run at the exact same times that the Veeam job was running and the two jobs were in contention with each other causing VSS I/O freeze. Veeam would be backing up sql and her job would also be trying so there was a freeze of I/O while the two processes worked through their argument. In my opinion though it really is, one hand not knowing nor asking what the other is doing and making changes without telling anyone. I've told her a thousand times that Veeam is backing up sql and there is no need for her to do so anymore but I guess that's falling on deaf ears.

I've asked support to close the case.
Gostev
Chief Product Officer
Posts: 31459
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Veeam (v9) Negatively Affecting SQL Databases

Post by Gostev »

Sounds good, and thanks a lot for coming back to post the root cause!
DaveWatkins
Veteran
Posts: 370
Liked: 97 times
Joined: Dec 13, 2015 11:33 pm
Contact:

Re: Veeam (v9) Negatively Affecting SQL Databases

Post by DaveWatkins » 1 person likes this post

You'll really want to make sure that she's not truncating the log file with her backups, and if she's doing it from a maint plan then she probably is. If both Veeam and a native SQL backup are truncating the logs then no one's log replay's will ever work and the only restore option you'll have that will work is from the exact time the backup was taken
Post Reply

Who is online

Users browsing this forum: Amazon [Bot], ante_704, Bing [Bot], Google [Bot], mrmccoy007 and 334 guests