SQL Transaction Log Backup issues

[cerberus]

Hi all,

Support Case #07425710

We recently upgraded our Veeam backup server hardware, and as part of this we also followed best practice and deployed the new server as non-domain joined.

VBR is used to backup VMware vSphere 8 environment, no issues with the VM backup job, runs super-fast with [CBT][fast-clone] and uses storage snapshots, I peak at about 2GB/s.

One of these VMs is a SQL Server 2016, and I am having a hard time getting transaction logs to backup...

Code: Select all

Transaction log backup interval 15 minutes 	
New transaction log backup interval started at 10/2/2024 3:12:26 PM 	
Preparing guest for SQL Server transaction log backup 	03:40
Using guest interaction proxy MYBACKUPSERVER (Different subnet) 	
Required backup infrastructure resources have been assigned 	
Enumerating SQL Server databases 	00:36
Saving 3.9 MB of transaction logs to backup repository 	54:06
Found previously backed up SQL Server transaction log for .... SSISDB;ReportServer;mode
Skipping simple recovery model database: ... ReportServerTempDB;msdb;master 	
Waiting for log shipping server 	26:41
Performing SQL Server transaction log backup for ... ;SSISDB;model;ReportServer 	47:25
Failed to connect to guest agent through VIX via GIP, failing over to guest agent connection via GIP 	
Failed to connect to guest agent via GIP, failing over to guest runtime components through VIX via GIP 	
Failed to inject guest runtime components through VIX via GIP, failing over to guest runtime components via GIP 	
Failed to upload SQL Server transaction log backup for MBAM Recovery and Hardware to MYBACKUPSERVER: Failed to connect to the guest 	
Failed to connect to guest agent through VIX via GIP, failing over to guest agent connection via GIP 	
Failed to connect to guest agent via GIP, failing over to guest runtime components through VIX via GIP 	
Failed to inject guest runtime components through VIX via GIP, failing over to guest runtime components via GIP 	
Failed to upload SQL Server transaction log backup for OPPY_Nielsen to MYBACKUPSERVER: Failed to connect to the guest 	
Failed to connect to guest agent through VIX via GIP, failing over to guest agent connection via GIP 	
Failed to connect to guest agent via GIP, failing over to guest runtime components through VIX via GIP 	
Failed to inject guest runtime components through VIX via GIP, failing over to guest runtime components via GIP 	
Failed to upload SQL Server transaction log backup for NSP_Accounting to MYBACKUPSERVER: Failed to connect to the guest 	
Failed to connect to guest agent through VIX via GIP, failing over to guest agent connection via GIP 	
Failed to connect to guest agent via GIP, failing over to guest runtime components through VIX via GIP 	
Failed to inject guest runtime components through VIX via GIP, failing over to guest runtime components via GIP 	
Failed to upload SQL Server transaction log backup for XXX to MYBACKUPSERVER: Failed to connect to the guest

It basically does not work, the major change here is we moved the server into a different subnet (still basically connected to the same switch, different vLAN) and the server is no longer domain joined.

I have tried with different quest interaction proxy, log shipping server, etc.. nothing seems to work.

Anyone run into anything similar?

[david.domask]

Hi Mirza,

Thank you for the details on the unexpected behavior and the case number.

The issue itself based just on the errors you're reporting look to be connectivity related, though I understand you have done a bit of work on this front already and looks like it was able to resolve the issue for some of the databases, but not all. No known issues at this time match the behavior and it looks more environmental at first blush.

The case is currently with our Advanced Technical Support, so please continue the research along with Veeam Support, looks like there are some steps to check from the most recent correspondence, and hopefully the joint research will identify why these last few DBs are having challenges.

[cerberus]

Yeah, this one is puzzling, in my many years of dealing with VBR I never ran into something this strange.

The transaction log backup job runs for a while, at snail speed (KB/s) for almost 2 hours (when it should really be a few minutes, underlying infra is very fast); I think it just hits some internal timer and eventually gives up.

Code: Select all

10/2/2024 3:50:15 PM :: Failed to inject guest runtime components through VIX via GIP, failing over to guest runtime components via GIP
Failed to upload SQL Server transaction log backup for ...: Failed to connect to the guest 	
Failed to upload SQL Server transaction log backup for ...: Repository agent is stopped 	
10/2/2024 4:54:31 PM :: Transaction log backup completed at 1 KB/s with bottleneck: Guest connection (VIX)

Last night I also updated VMware Tools to the latest and rebooted the SQL server, no effect.

Had no issues when backup server was on the domain and same subnet, now that its off-domain and different subnet, can't get passed this issue; the crazy thing is when the actual VM backup job runs with settings to truncate the database logs instead of shipping them off, zero issues, logs below. Why TLog job is running at snail speed pulling at KB/s speeds for only a subset of databases...

Code: Select all

5:18:36 PM	VMware Tools are not up to date, some functionality may be affected 	
5:18:41 PM	Using guest interaction proxy XXX (Different subnet) 	
5:19:32 PM	Inventorying guest system 	00:37
5:20:58 PM	Preparing guest for hot backup 	00:19
5:21:28 PM	Releasing guest 	00:22
5:21:28 PM	Creating VM snapshot 	
5:21:40 PM	Getting list of guest file system local users 	00:10
5:21:50 PM	Collecting disk files location data 	00:44
5:22:42 PM	Removing VM snapshot 	00:08
5:22:51 PM	Queued for processing at 10/2/2024 5:22:51 PM 	
5:22:51 PM	Required backup infrastructure resources have been assigned 	00:00
5:22:56 PM	Processing XXX 	03:30
5:22:57 PM	VM processing started at 10/2/2024 5:22:57 PM 	
5:22:57 PM	VM size: 1.9 TB (1.3 TB used) 	
5:23:01 PM	Saving [FA-ProdDS-04] XXX.vmx 	00:00
5:23:01 PM	Saving [FA-ProdDS-04] XXX.vmxf 	00:00
5:23:01 PM	Saving [FA-ProdDS-04] XXX.nvram 	00:00
5:23:05 PM	Using backup proxy VMware Backup Proxy for retrieving Hard disk 1 data from storage snapshot on XXX 01:59
5:23:05 PM	Using backup proxy VMware Backup Proxy for retrieving Hard disk 2 data from storage snapshot on XXX 01:59
5:23:05 PM	Using backup proxy VMware Backup Proxy for retrieving Hard disk 3 data from storage snapshot on XXX 01:59
5:25:05 PM	Hard disk 3 (500 GB) 39.3 GB read at 8 GB/s [CBT][fast clone]	00:07
5:25:05 PM	Hard disk 2 (1.2 TB) 46.8 GB read at 2 GB/s [CBT][fast clone]	00:24
5:25:06 PM	Hard disk 1 (200 GB) 2 GB read at 980 MB/s [CBT][fast clone]	00:51
5:26:06 PM	Saving GuestMembers.xml 	00:00
5:26:07 PM	Finalizing 	00:00
5:26:08 PM	Truncating transaction logs 	00:10
5:26:24 PM	Swap file blocks skipped: 582 MB 	
5:26:24 PM	Deleted file blocks skipped: 23.4 GB 	
5:26:28 PM	Busy: Source 86% > Proxy 31% > Network 17% > Target 12% 	
5:26:28 PM	Primary bottleneck: Source 	
5:26:28 PM	Network traffic verification detected no corrupted blocks 	
5:26:28 PM	Processing finished at 10/2/2024 5:26:28 PM 	

Is there any way to ad-hoc stop the actual transaction job backup immediately? It runs when the main VM backup job that has app aware enabled, I know I can disable the main job, but it still takes a while for the transaction job to stop, is there a way to immediately stop it? Losing a lot of time trying different things in just getting this job to stop and re-run with new settings.

[david.domask]

Hi Mizra,

>Is there any way to ad-hoc stop the actual transaction job backup immediately?

While there are "fast" ways to terminate the Log Backup, it's best to follow the normal process, as there is a bit of clean up of the log shipping components that needs to be done.

For what its worth, based on just the above, I'm not convinced it's something specific to any configuration you'd set in Veeam (but naturally I'm working with limited evidence, so will be happy to be corrected), so I think it's best to continue with Veeam Support and let them continue their review, as I see that there are a few tests that Support is asking for most recently in the case.

[cerberus]

What is the way to "fast" terminate the transaction log backup job other than rebooting the backup server? LOL

I am using a test job and backing up a test SQL server, just trying a few different things at this point, while the support ticket is worked on by our techs and your support group.

[JoLam]

Hi,
just wanted to add a +1.
I'm experiencing the same issue and it startet around two weeks ago seemingly out of the blue.
Has your ticket yielded any updates/solution?

[david.domask]

Hi JoLam, welcome to the forums.

Sorry to hear about the challenges -- the topic creator's case is still being researched, and such issues tend to be related to environmental factors, so please open a Support Case as per the Forum Policies.

Be sure to include logs for the affected jobs for Support to review. Please share your case number once opened.

[JoLam]

Thanks, will do that.

[cerberus]

Can you share if there is a way to immediately terminate transaction log backup? Purely for testing, I want the ability to stop the job, without resorting to restarting Veeam services or backup server just to get this thing to stop running.

In our case, it can take hours before it times out, and sometimes I don't have hours - just want to try new settings or run other operations that can't because of "In progress Waiting for transaction log backup jobs to finish".

[PetrM]

Hello,

Did you have a look at this page?

Thanks!