Issues since Update 2

[Lewpy]

Since applying Update 2 to a customer's B&R v8 installation, they have started to get a couple of new issues

Backup Jobs
While processing some VMs within a job the following error occasionally occurs

Code: Select all

Processing ******* Error: The current transaction cannot be committed and cannot support operations that write to the log file. Roll back the transaction.

At the end of a job (each job contains multiple VMs) the following error occasionally occurs

Code: Select all

Full backup file merge failed Error: The current transaction cannot be committed and cannot support operations that write to the log file. Roll back the transaction.

This is normally fixed during one or two retries of the backup job.

Backup Copy Jobs
When the Backup Copy Job interval ends and the Backup Copy Job is stopped & restarted, the following error occasionally occurs

Code: Select all

Failed to start job [****** Archive] Error: Another job process already in progress. 

This is resolved after many automatic restart attempts of the Backup Copy Job, each failed restart attempt generating another similar error.
The Backup Copy Job goes from one Veeam Server (full installation) to another Veeam Server (remote Windows Server).

Are these known issues, or do I need to raise Support Calls for them?
They only started to appear after applying Update 2: the first error started that evening

Thanks,
Lewis.

[Lewpy]

Update: just realised that the slight pre-release Update 2 [8.0.0.2018_Update2] we received (to fix a particular issue) is a different build than the official Update 2 release [8.0.0.2021_Update2].
I will apply the "official" Update 2 release and see if it resolved the issues, and then update this thread.

[Gostev]

Actually, please get Update 2a from support (this is the newest build that should go GA in the next few days). Thanks!

[Lewpy]

Hi Anton,
I've opened a request with Support to obtain Update 2a
I will be sure to double-check build numbers this time when it does go public
Thanks,
Lewis.

[Lewpy]

I received Update 2a (build 2029) quickly from Support, and managed to install it before the backups ran the other night.
There have been two nights worth of backups & backup copy jobs, and all the "red" failures have vanished from the History view
So looks promising that the issues have been resolved in newer builds of Update 2 ([more] time will tell, as with most things!).

However, I do have one persisting issue caused by the errors we were receiving
One of the Backup Copy Jobs issued the following errors the other night

Code: Select all

31/05/2015 01:32:37 :: All VMs have been queued for processing 
31/05/2015 01:32:43 :: Queued for processing at 31/05/2015 01:32:43 
31/05/2015 01:32:43 :: Required backup infrastructure resources have been assigned 
31/05/2015 01:32:48 :: Failed to merge full backup file. Terminated at 31/05/2015 08:04:19 
31/05/2015 01:33:29 :: Failed to create synthetic full restore point 04/05/2015 00:00:00 
31/05/2015 08:04:19 :: Failed to generate points Error: The current transaction cannot be committed and cannot support operations that write to the log file. Roll back the transaction. 
31/05/2015 08:04:19 :: Failed to create restore points 
31/05/2015 08:05:21 :: Starting full backup file merge [GFS] 
31/05/2015 08:08:12 :: Deleting full backup 01/02/2015 00:00:00 per retention policy 
31/05/2015 08:08:15 :: Full backup file merge completed successfully 
31/05/2015 08:08:15 :: Waiting for the new copy interval 
01/06/2015 00:00:11 :: Copy interval has expired

Since that time, it has not tried to run the "full backup file merge" process.
The day before it did this

Code: Select all

30/05/2015 01:49:13 :: All VMs have been queued for processing 
30/05/2015 01:49:19 :: Queued for processing at 30/05/2015 01:49:19 
30/05/2015 01:49:19 :: Required backup infrastructure resources have been assigned 
30/05/2015 01:49:24 :: Starting full backup file merge [GFS] 
30/05/2015 01:49:51 :: Restore point 03/05/2015 00:00:00 created successfully 
30/05/2015 02:11:50 :: Full backup file merge completed successfully 
30/05/2015 02:11:51 :: Waiting for the new copy interval 
31/05/2015 00:00:34 :: Copy interval has expired

But every night since the above errors, it has just done this

Code: Select all

03/06/2015 01:29:08 :: All VMs have been queued for processing 
03/06/2015 01:29:16 :: Waiting for the new copy interval 
04/06/2015 00:00:17 :: Copy interval has expired

So basically, the Backup Copy Job chain is increasing in size, and the "Recent" full backup is not merging forward any more.
In fact, looking at the Restore Points of the Backup Copy Job in the Backups->Disk section of the Veeam Console, the "Recent" retention point is not near the end of the list with the other GFS retention points: it is positioned at the day of the above errors and showing as 0.0KB for both size columns (it is definitely not a zero size file though).

Is there a quick way to resolve this (a repository rescan, or triggering the Backup Copy Job Health Check?), or will it need a Support Call to investigate?

Thanks,
Lewis.

[foggy]

Lewis, yes, you'd better address this to support.

[Lewpy]

Thanks foggy, will do

[Lewpy]

I have raised an incident with Support (ID# 00934122).
No immediate resolution, and the issue has been internally escalated.

[Lewpy]

Just to conclude: the issue has been resolved through Veeam Support (thanks Lisa!)
In a nut-shell, the Veeam SQL database got out of sequence with the Backup Copy Job file chain, so was not merging the files as it didn't think it needed to. Left long enough, it may have started to remerge, but it was not clear this would happen and also may happen incorrectly.
So the SQL database was modified to re-align with the file chain, and when the Backup Copy Job was re-enabled it immediately started to merge the backup files
We've had a couple of nights of normal backup operation, so case closed