Reversed incremental backups are getting longer and longer..

[GregorS]

I've got Veeam Backup & Replication version 6.5. I've got a reversed incremental job with ~50 VMs. It's been running since november 2012 without intermediate fulls. There're now ~230 reversed increments. Every run gets a bit longer to finish. For every VM there's now a ~10 min wait time at phase "Using source proxy <backup proxy name> [hotadd]". What'd be the cause?

Kind regards
Gregor

[dellock6]

I'm almost sure the VBK file is heavily fragmented after running for so long time. 230 days is a time too long for a single reverse incremental chain, there are several reasons for not letting this chain go on fo so long time (http://www.virtualtothecore.com/en/?p=4617). Running an active full every 1 or 2 months is a safe procedure.

Luca.

[GregorS]

Thanks Luca. I'll set weekly active fulls and see the difference.

Regards
Gregor

[veremin]

Thanks Luca. I'll set weekly active fulls and see the difference.

Actually, with reversed incremental mode the need for performing active full backup isn’t that strong; it’s considered to be less important. So, weekly Full is definitely an overkill; performing Full once every 1-3 months is more than enough.

Thanks.

[GregorS]

My first intermediate active full backup is now created but the problem didn't go away. Curiously, even during the full there was a more than ten minutes waiting time period for each and every VM at phase "Using source proxy <backup proxy name> [hotadd]". Can someone please elaborate what Veeam Backup is doing during this exact phase?
Some more hints:

• at the beginning there were no noticeable delays
• I can see through history that the delay is getting worse with about 11 minutes at present
• I perceive that the delay is getting a bit longer at every backup run
• the delay is independent of the characteristics of a VM, number of VMs and now even of the type of a backup job and it seems to be equally long for each VM
• it's probably linearly correlated only to the number of increments

Thanks
Gregor

[foggy]

Gregor, debug logs should clearly tell what is being performed during this delay. I suggest to contact our technical team so they could assist you in reading the job logs.

[GregorS]

I checked the logs as suggested:

1. job log files are enormous, the current Job.Primary_Backup_Job.Backup.log is 2.7 GB uncompressed and 300 MB ZIP
2. since this is barely uploadable I rather looked inside the log, at least at the beginning of it:

Code: Select all

===================================================================
Continuing interrupted log...
Log has been started by 'ONKO_NET\svcVeeam' user (Non-interactive)
Logging level: [4 (AboveNormal)]
MachineName: [ONKO-VEE], OS: [Microsoft Windows NT 6.1.7601 Service Pack 1]
Process: [64 bit], PID: [3500], SessionId: [0]
UTC Time: [15.04.2013 16:00:17], DaylightSavingTime: [True]
Culture: [sl-SI], UI culture: [en-US]
...
[16.04.2013 06:01:04] <01> Info     [OIB] Finding corrupted oibs in group: 'e8b17b8a-ae02-4b8c-8203-c0e6379827b8'
[16.04.2013 06:01:04] <01> Info     [OIB] No corrupted oibs in group: 'e8b17b8a-ae02-4b8c-8203-c0e6379827b8' found
[16.04.2013 06:01:04] <01> Info     [BackupMeta] Generate backup meta for 'Primary Backup Job:d42bb7f6-2c5e-4225-808f-c3a5b403a1f1'
[16.04.2013 06:01:47] <01> Info     [BackupMeta] Saving backup meta to 'D:\Primary Backup Job\Primary Backup Job.vbm'
[16.04.2013 06:01:47] <01> Info         [FC] Getting item info 'D:\Primary Backup Job'.
[16.04.2013 06:01:49] <01> Info     [ONKO-VEE] [FC] Saving file content '<BackupMeta Id="f883024d-eabc-4b75-b3f9-8d46083eea6a" JobId="d6503...

The log seems normal from the beginning, it enumerates the VMs for backup, then it starts to deal with the first VM and then it comes to "generating backup meta" where the last line in the output above is absurdly long to the infinity and beyond and it just keeps on being longer with every incremental run. I think this is what hogs my backup. But is this normal behaviour?

Gregor

[foggy]

Please provide logs to our technical support and continue pursuing this issue with them directly. The product management team behind this forum is not involved in technical support and is unable to assist with reading debug logs. Please inform your support engineer about the size of the log files when opening the ticket, and you will be provided with a link to the FTP server to upload this amount of debug logs files.

[fast_cat]

Hi,

Check out this post to see if you have the same issue that I was seeing....(its the posts towards the end of the discussion that demonstrate one reason why backup jobs appear to get slower..at least for my environment)

regards
Rob

[GregorS]

Thanks to Fast Cat for adding a clue. I checked the VMwareSnapshots table, there are 25 rows and I assume 24 are stale but I don't think this is the root cause for my delays. It seems that my backup job is somehow combining data about all previous backup runs into metadata at the beginning of each VM's backup.

[wmoore]

We backup about 15TB per night (that is, about 15TB of source data) using reversed incrementals and we had something similar happen to us - jobs would take longer, jobs would fail - the bigger VMs started failing more often than succeeding. The logs kept telling us "couldn't read the next block" or something equally non-descriptive.

Our backup target was down to about 1.2TB of space and we did some cleanup to recover about 1TB and suddenly the backups are flying again, and completing with 100% success. It seems like Veeam needs a lot of scratch space to work.

[GregorS]

Wayne (wmoore), thank you for your input. My backup repository's capacity is 24.5 TB with 6.6 TB free (more than 25%). My full backup is around 1.5 TB and increments are less than 100 GB. I still think it's something with metadata...

[GregorS]

I would just like to report that the problem is now solved thanks to Alexander Shelemin of Veeam Support. Deleting the whole contents of the table VMwareSnapshots in Veeam backup database in Microsoft SQL Server was sufficient. There were many leftover snapshots journaled in, because my backup job occasionally freezes while removing the snapshot and I had to stop the running job forcefully many times.

G.

[foggy]

Thanks for getting back with the update, Gregor! Much appreciated.

[Yuki]

Can someone please provide step-by-step instructions for this. I would like to see if we are being affected by this (we also had to kill a few jobs that got stuck and manually clean up), so we could potentially be in the same situation.

[Vitaliy S.]

Yuki, I would strongly recommend performing all database tables removal tasks with the help of our support team.

[veremin]

I would like to see if we are being affected by this.

As mentioned in this post, in order to determine whether you have the said issue or not, you might want to query –dbo-VMware Snapshot table, see how many entries it has and compare them with the results from vCenter.

Anyway, feel free to contact our support team and ask them to assist you with this.

Thanks.

[GregorS]

Here's some PowerShell code you can reuse to delete rows from the table that are not associated with real snapshots in the virtual infrastructure. First download and install the following, best on your VBR server:

1. Windows Management Framework 3.0, this'll give you Microsoft Windows PowerShell 3.0 on supported platforms, if you don't have it already
2. VMware vSphere PowerCLI
3. Microsoft Windows PowerShell Extensions for Microsoft SQL Server 2012 SP1, it works well with older versions of SQL Server too

Code: Select all

Add-PSSnapin -Name "VeeamPSSnapin", "VMware.VIMAutomation.Core"
Import-Module -Name "SQLPS" -DisableNameChecking
Connect-VIServer -Server "YourVCServerName"
$SnapshotRef = Invoke-SQLCmd -Query "SELECT * FROM VMwareSnapshots" -ServerInstance "YourSqlServerName" -Database "YourSqlDatabaseName" | ForEach-Object {$_.Snapshot_ref}
$SnapshotId = Get-VM | Get-Snapshot | ForEach-Object {$_.Id.Replace("VirtualMachineSnapshot-", "")}
$SnapshotRef | Where-Object {$SnapshotId -notcontains $_} | ForEach-Object {Invoke-SQLCmd -Query "DELETE FROM VMwareSnapshots WHERE Snapshot_ref = '$_'" -ServerInstance "YourSqlServerName" -Database "YourSqlDatabaseName"}

You can get your SqlServerName and SqlDatabaseName in the following system registry location on your VBR server:
HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication

[foggy]

And one more time I want to stress that:

Yuki, I would strongly recommend performing all database tables removal tasks with the help of our support team.

Also, you should keep in mind that database structure can be changed between product releases. Thanks!