SQL Transaction Log backup's failing

[andybishop]

Hi,

We are having issues with a couple of servers running hourly SQL Transaction log backup's. I have a case raised 03171093.

Suggestions have been to disable UAC, Anti-virus and reboot the server. Disabling UAC is not an option, and should not cause an issue when the VM's are accessible on the network. Anti-virus exclusions are already in place for Veeam. Rebooting the servers all the time is not a viable option for production servers.

Number1 Server

Code: Select all

Failed to prepare guest for SQL Server transaction log backup Details: Cannot upload guest agent's files to the administrative share [C:\Windows]. Could not copy host file [C:\Program Files (x86)\Veeam\Backup Transport\GuestInteraction\VSS\sql\VeeamLogShipper_X64.exe] to guest [C:\Windows\VeeamLogShipper\VeeamLogShipper.exe] Could not copy host file [C:\Program Files (x86)\Veeam\Backup Transport\GuestInteraction\VSS\sql\VeeamLogShipper_X64.exe] to guest [vm-103843] file [C:\Windows\VeeamLogShipper\VeeamL
Transaction logs for the following databases were not backed up: master;model;msdb;distribution
Transaction logs from some databases were not backed up for the last 12 intervals

The process for VeeamLogShipper seems to get stuck and cannot be removed, trying to end the Veeamlogshipper.exe process via Task Manager give the following error: "Unable to Terminate process" "The operation could not be completed. Access is denied"

A reboot of this server on Monday morning resolved the issue until the Incremental backup run last night and now we are getting the same error as before.


Number2 Server

Code: Select all

Failed to prepare guest for SQL Server transaction log backup Details: Cannot upload guest agent's files to the administrative share [C:\Windows]. Could not copy host file [C:\Program Files (x86)\Veeam\Backup Transport\GuestInteraction\VSS\sql\VeeamLogShipper_X64.exe] to guest [C:\Windows\VeeamLogShipper\VeeamLogShipper.exe] Could not copy host file [C:\Program Files (x86)\Veeam\Backup Transport\GuestInteraction\VSS\sql\VeeamLogShipper_X64.exe] to guest [vm-103988] file [C:\Windows\VeeamLogShipper\VeeamL
Failed to prepare guest for SQL Server transaction log backup Details: Failed to execute vSphere API command: [Cannot connect to remote SCM database. Machine: []. Requested rights: [0xf0015]. Win32 error:Access is denied. Code: 5].
Transaction logs for the following databases were not backed up: master;model;msdb;distribution
Transaction logs from some databases were not backed up for the last 13 intervals

This server was rebooted last night and is now working again for the time being.

Any suggested on this, we only have six servers in the backup test and these two of the servers are frequently failing with these errors. This is concerning as we were planning to move a large number of SQL Production server to Veeam.

Has anyone else had this issue?

[Dima P.]

Hello Andy.

It looks like guest component cannot be properly uploaded to the mentioned machines (this indeed might be connectivity issue caused by firewall or admin share inaccessibility). Please keep working with our support team in order to precisely identify and fix the root cause. Cheers!

[andybishop]

I am able to access the share from all Veeam servers, permissions are fine, nothing is blocked on the firewall (I have monitored the firewall logs). Following a reboot on Monday morning the Transaction logs processed with no issues until the Incremental backup ran last night. The same happened last week, had issues, rebooted, worked for a few days and then failed until it was rebooted.

Support are suggesting removing Anti-virus from this server, but it has the same software, version and policies as the servers that are working all the time. Also the fact it works for a period of time rules this out, and our IT Security department will not want AV turning off.

[foggy]

But still something prevents the log shipper process deployment. Are there any related messages in the Event Log on the affected VMs?

[andybishop]

The incremental backup that runs at 23:00 completed successfully, then at 23:03 the following starts to show in the Application logs:-

The description for Event ID 1000 from source VGAuth cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.

If the event originated on another computer, the display information had to be saved with the event.

The following information was included with the event:

vmtoolsd: Username and password successfully validated for 'DOMAIN\VEEAM-SVC'

These events happen for around 10 minutes and then stop, they start again when the next Transaction log interval runs an hours later.

[andybishop]

After working all day with no issues, Number2 Server has now started having the same issues again. Incremental completed successfully at 23:00 and now failing every hour with:

Code: Select all

Failed to prepare guest for SQL Server transaction log backup Details: Cannot upload guest agent's files to the administrative share [C:\Windows]. Could not copy host file [C:\Program Files (x86)\Veeam\Backup Transport\GuestInteraction\VSS\sql\VeeamLogShipper_X64.exe] to guest [C:\Windows\VeeamLogShipper\VeeamLogShipper.exe] Could not copy host file [C:\Program Files (x86)\Veeam\Backup Transport\GuestInteraction\VSS\sql\VeeamLogShipper_X64.exe] to guest [vm-103988] file [C:\Windows\VeeamLogShipper\VeeamL
Failed to prepare guest for SQL Server transaction log backup Details: Failed to execute vSphere API command: [Cannot connect to remote SCM database. Machine: []. Requested rights: [0xf0015]. Win32 error:Access is denied. Code: 5].
Transaction logs for the following databases were not backed up: master;model;msdb;distribution

Transaction logs from some databases were not backed up for the last 5 intervals

[foggy]

Please continue working with your engineer on this. You can also ask to escalate the support case, if required. Thanks!

[ejenner]

Isn't this just a simple 'access is denied' error? We had to give specific rights to the account we use for Veeam within our SQL setup to allow it to backup SQL. We used the 'diskadmin' role within SQL as we determined that was the correct role to use for performing backups. We went from 'access denied' to successful backup after giving this right.

[andybishop]

It's not getting that far, it's failing copying and running the veeamlogshipper as a service. On one of the servers there is a process stuck. It's works for a couple of days following a reboot, and then it will stop working after an incremental backup runs. The account used for Veeam has the correct SQL permissions but I will double check.

[ejenner]

the permissions are granted within the SQL environment. It has some built-in users you can use (although I'd be willing to bet you can set up your own). As I say, we used 'diskadmin' and that is one of them. What are you using for yours?