Basically, they use Veeam Agent to backup their Azure servers to our CloudConnect repository. Backup job failed with error message:
"Error: The MERGE statement attempted to UPDATE or DELETE the same row more than once. This happens when a target row matches more than one source row. A MERGE statement cannot UPDATE/DELETE the same row of the target table multiple times. Refine the ON clause to ensure a target row matches at most one source row, or use the GROUP BY clause to group the source rows. "
Both tenants have other servers in Azure using the same backup and network setting without issue.
We first thought it had something to do with SP side database, but it looks more like connection issue now.
Case # 04823585
Logs for 1 of them:
Code: Select all
[25.06.2021 11:42:58] <01> Info [CloudGateSvc **************:6180]Request: [Service.SaveBackupJobInfo] IsRequireRetry:False, IsForeverIncremental:True, IsReverseIncremental:False,
[25.06.2021 11:43:19] <01> Error Exception while connecting to endpoints [203.**.**.**]
[25.06.2021 11:43:19] <01> Error Unable to connect to the service provider. (Veeam.Backup.Core.CSocketInvokerClient+CCloudGateUnavailableException)
[25.06.2021 11:43:19] <01> Error at Veeam.Backup.Core.CSocketInvokerClient.Wait(WaitHandle waitObj, State state, Int32 timeout, String timeoutError, Boolean throwGateUnavailableException)
[25.06.2021 11:43:19] <01> Error at Veeam.Backup.Core.CSocketInvokerClient.TryInvoke(CSocketInvokerParams invokerParams)
[25.06.2021 11:43:19] <01> Error A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond 203.**.**.**:6180 (System.Net.Sockets.SocketException)
[25.06.2021 11:43:19] <01> Error at System.Net.Sockets.Socket.InternalEndConnect(IAsyncResult asyncResult)
[25.06.2021 11:43:19] <01> Error at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)
[25.06.2021 11:43:19] <01> Error at System.Net.Sockets.TcpClient.EndConnect(IAsyncResult asyncResult)
[25.06.2021 11:43:19] <01> Error at Veeam.Backup.Core.CSocketInvokerClient.OnConnected(IAsyncResult result)
[25.06.2021 11:43:19] <01> Error Failed to invoke command. Will try another gate
[25.06.2021 11:43:19] <01> Error Unable to connect to the service provider. (Veeam.Backup.Core.CSocketInvokerClient+CCloudGateUnavailableException)
[25.06.2021 11:43:19] <01> Error at Veeam.Backup.Core.CSocketInvokerClient.Wait(WaitHandle waitObj, State state, Int32 timeout, String timeoutError, Boolean throwGateUnavailableException)
[25.06.2021 11:43:19] <01> Error at Veeam.Backup.Core.CSocketInvokerClient.TryInvoke(CSocketInvokerParams invokerParams)
[25.06.2021 11:43:19] <01> Error at Veeam.Backup.Core.CSocketInvokerClient.Invoke(CSocketInvokerParams invokerParams)
[25.06.2021 11:43:19] <01> Error at Veeam.Backup.Core.CCloudGateSvcImpl.InvokeRpc(CSocketInvokerClient invokerClient, CSocketInvokerParams invokerParam)
[25.06.2021 11:43:19] <01> Error at Veeam.Backup.Core.CCloudGateSvc.InvokeWithSessionRestartOnFail(CSocketInvokerParams spec, Action restartSession)
[25.06.2021 11:43:19] <01> Error at Veeam.Backup.Core.CCloudGateSvc.InvokeWithGateRescanOnFail(CSocketInvokerParams spec)
[25.06.2021 11:43:19] <01> Error A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond 203.**.**.**:6180 (System.Net.Sockets.SocketException)
[25.06.2021 11:43:19] <01> Error at System.Net.Sockets.Socket.InternalEndConnect(IAsyncResult asyncResult)
[25.06.2021 11:43:19] <01> Error at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)
[25.06.2021 11:43:19] <01> Error at System.Net.Sockets.TcpClient.EndConnect(IAsyncResult asyncResult)
[25.06.2021 11:43:19] <01> Error at Veeam.Backup.Core.CSocketInvokerClient.OnConnected(IAsyncResult result)Code: Select all
[25.06.2021 16:27:20] <01> Error Microsoft SQL server hosting the configuration database is currently unavailable. Possible reasons are heavy load, networking issue, server reboot, or hot backup.
[25.06.2021 16:27:20] <01> Error Please wait, and try again later.
[25.06.2021 16:27:20] <01> Error Error information:
[25.06.2021 16:27:20] <01> Error The MERGE statement attempted to UPDATE or DELETE the same row more than once. This happens when a target row matches more than one source row. A MERGE statement cannot UPDATE/DELETE the same row of the target table multiple times. Refine the ON clause to ensure a target row matches at most one source row, or use the GROUP BY clause to group the source rows. (Veeam.Backup.Common.CSqlException)
[25.06.2021 16:27:20] <01> Error at Veeam.Backup.Common.CExceptionUtil.ThrowSqlException(Exception innerException)
[25.06.2021 16:27:20] <01> Error at Veeam.Backup.DBManager.CDbAccessor.ExecNonQuery(CStoredProcedureData procedureData, Nullable`1 identifier)
[25.06.2021 16:27:20] <01> Error at Veeam.Backup.DBManager.CDbAccessor.ExecNonQuery(String spName, SqlParameter[] spParams)
[25.06.2021 16:27:20] <01> Error at Veeam.Backup.DBManager.CCloudProviderDbScope.UpdateGatesAvailiblity(IReadOnlyList`1 priorityList, Guid cloudProviderId)
[25.06.2021 16:27:20] <01> Error at Veeam.Backup.DBManager.CCloudProviderDbScope.UpdateGates(IReadOnlyList`1 priorityList, Guid cloudProviderId)
[25.06.2021 16:27:20] <01> Error at Veeam.Backup.Core.CCloudProvider.UpdateGatePriorityList(CCGateAddressCollection priorityList)
[25.06.2021 16:27:20] <01> Error at Veeam.Backup.Core.CCloudGateSvc.InvokeWithGateRescan(Action action)
[25.06.2021 16:27:20] <01> Error at Veeam.Backup.Core.CCloudGateSvc.InvokeWithSessionToken(CCloudInvokerSpec spec)
[25.06.2021 16:27:20] <01> Error at Veeam.Backup.Core.CCloudDbAccessor.GetPointsByBackupId(Guid backupId)
[25.06.2021 16:27:20] <01> Error at Veeam.Backup.Core.CBackup.FindLastPoint()
[25.06.2021 16:27:20] <01> Error at Veeam.Backup.Core.CLocalBackupTargetRepairAlg.Repair()
[25.06.2021 16:27:20] <01> Error at Veeam.Backup.Core.CBackupTargetEndpointRepairAlg.Repair()
[25.06.2021 16:27:20] <01> Error at Veeam.Backup.Core.CBackupJobTarget.PreProcess(CTasksScheduler tasksScheduler)
[25.06.2021 16:27:20] <01> Error at Veeam.EndPoint.CEndPointBackupJobPerformer.ExecuteTasks(CBackupTaskContext backupTaskContext, CSessionEpDiskFilter sessionDiskFilter)
[25.06.2021 16:27:20] <01> Error at Veeam.EndPoint.CEndPointBackupJobPerformer.PerformIteration(CBackupTaskSession backupSession, CDisposableList innerDisposableList, IBackupCacheStrategy backupCacheStrategy, CBackupTaskContext taskContext, CBackupRepository targetRepository, CSessionEpDiskFilter sessionDiskFilter)
[25.06.2021 16:27:20] <01> Error at Veeam.EndPoint.CEndPointBackupJobPerformer.PerformWithFailoverToCache(CBackupTaskSession backupSession, CDisposableList parentContext, IBackupCacheStrategy backupCacheStrategy, CBackupTaskContext taskContext, CBackupRepository targetRepository, CSessionEpDiskFilter sessionDiskFilter)
[25.06.2021 16:27:20] <01> Error The MERGE statement attempted to UPDATE or DELETE the same row more than once. This happens when a target row matches more than one source row. A MERGE statement cannot UPDATE/DELETE the same row of the target table multiple times. Refine the ON clause to ensure a target row matches at most one source row, or use the GROUP BY clause to group the source rows. (System.Data.SqlClient.SqlException)
[25.06.2021 16:27:20] <01> Error at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
[25.06.2021 16:27:20] <01> Error at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
[25.06.2021 16:27:20] <01> Error at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
[25.06.2021 16:27:20] <01> Error at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString, Boolean isInternal, Boolean forDescribeParameterEncryption, Boolean shouldCacheForAlwaysEncrypted)
[25.06.2021 16:27:20] <01> Error at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite, Boolean inRetry, SqlDataReader ds, Boolean describeParameterEncryptionRequest)
[25.06.2021 16:27:20] <01> Error at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, TaskCompletionSource`1 completion, Int32 timeout, Task& task, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry)
[25.06.2021 16:27:20] <01> Error at System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(TaskCompletionSource`1 completion, String methodName, Boolean sendToPipe, Int32 timeout, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry)
[25.06.2021 16:27:20] <01> Error at System.Data.SqlClient.SqlCommand.ExecuteNonQuery()
[25.06.2021 16:27:20] <01> Error at Veeam.Backup.DBManager.CDbAccessor.ExecSqlCommand(SqlConnection connection, CStoredProcedureData procedureData)
[25.06.2021 16:27:20] <01> Error at Veeam.Backup.DBManager.CDbAccessor.ExecNonQuery(CStoredProcedureData procedureData, Nullable`1 identifier)