Veeam (v9) Negatively Affecting SQL Databases

[dspjones]

case number is 01714513


I was alerted this morning by our DBA that veeam backup is freezing I/O on our main sql database cluster when it runs. We backup our environment 6x per day 12a/4a/8a/12p/4p/8p. The entire job consists of about 50 servers and takes just under 30 minutes to finish. The specific DB server in questions is completed within 2 minutes. But, as the SQL logs show and the user calls to the helpdesk bear out, every time the backup occurs, the sql database freezes just long enough to kick users out of the application that database is used for.

The first thing I did is turn off log truncation, selecting COPY ONLY. However it is still freezing the database. Next I have disabled application-aware processing altogether. The next job runs at 4pm today so I am still waiting to see if this change "resolves" the problem but in previous versions of veeam that we've used 6/7 and 8 this has not been a problem on any database system. Any ideas on whether this might be a bug or something we are doing wrong?

[dspjones]

On the SQL server in question, coinciding with each veeam backup job run is the following VSS error in the application log:

Volume Shadow Copy Service error: Unexpected error calling routine ConvertStringSidToSid(S-1-5-21-1374761507-110244007-91453608-13101.bak). hr = 0x80070539, The security ID structure is invalid.
.

Operation:
OnIdentify event
Gathering Writer Data

Context:
Execution Context: Shadow Copy Optimization Writer
Writer Class Id: {4dc3bdd4-ab48-4d07-adb0-3bee2926fd7f}
Writer Name: Shadow Copy Optimization Writer
Writer Instance ID: {41e94905-ff8f-4c1f-80ec-554752574ce1}

[Gostev]

We did not have similar reports from other v9 users, so this is likely an environmental issue that is not specific to v9. Generally speaking, v9 had no changes in SQL application-aware processing logic or its VSS integration.

[dspjones]

DBA found the issue. Related to some change she made in her sql maintenance plan. In a nutshell it looks like she changed her job to run at the exact same times that the Veeam job was running and the two jobs were in contention with each other causing VSS I/O freeze. Veeam would be backing up sql and her job would also be trying so there was a freeze of I/O while the two processes worked through their argument. In my opinion though it really is, one hand not knowing nor asking what the other is doing and making changes without telling anyone. I've told her a thousand times that Veeam is backing up sql and there is no need for her to do so anymore but I guess that's falling on deaf ears.

I've asked support to close the case.

[Gostev]

Sounds good, and thanks a lot for coming back to post the root cause!

[DaveWatkins]

You'll really want to make sure that she's not truncating the log file with her backups, and if she's doing it from a maint plan then she probably is. If both Veeam and a native SQL backup are truncating the logs then no one's log replay's will ever work and the only restore option you'll have that will work is from the exact time the backup was taken