Comprehensive data protection for all workloads
Post Reply
GregorS
Veeam ProPartner
Posts: 48
Liked: 11 times
Joined: Apr 10, 2013 12:11 pm
Full Name: Gregor Smerke
Location: Ljubljana, SI - Slovenia
Contact:

Reversed incremental backups are getting longer and longer..

Post by GregorS »

I've got Veeam Backup & Replication version 6.5. I've got a reversed incremental job with ~50 VMs. It's been running since november 2012 without intermediate fulls. There're now ~230 reversed increments. Every run gets a bit longer to finish. For every VM there's now a ~10 min wait time at phase "Using source proxy <backup proxy name> [hotadd]". What'd be the cause?

Kind regards
Gregor
dellock6
Veeam Software
Posts: 6137
Liked: 1928 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: Reversed incremental backups are getting longer and long

Post by dellock6 » 2 people like this post

I'm almost sure the VBK file is heavily fragmented after running for so long time. 230 days is a time too long for a single reverse incremental chain, there are several reasons for not letting this chain go on fo so long time (http://www.virtualtothecore.com/en/?p=4617). Running an active full every 1 or 2 months is a safe procedure.

Luca.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
GregorS
Veeam ProPartner
Posts: 48
Liked: 11 times
Joined: Apr 10, 2013 12:11 pm
Full Name: Gregor Smerke
Location: Ljubljana, SI - Slovenia
Contact:

Re: Reversed incremental backups are getting longer and long

Post by GregorS »

Thanks Luca. I'll set weekly active fulls and see the difference.

Regards
Gregor
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Reversed incremental backups are getting longer and long

Post by veremin » 1 person likes this post

Thanks Luca. I'll set weekly active fulls and see the difference.
Actually, with reversed incremental mode the need for performing active full backup isn’t that strong; it’s considered to be less important. So, weekly Full is definitely an overkill; performing Full once every 1-3 months is more than enough.

Thanks.
GregorS
Veeam ProPartner
Posts: 48
Liked: 11 times
Joined: Apr 10, 2013 12:11 pm
Full Name: Gregor Smerke
Location: Ljubljana, SI - Slovenia
Contact:

Re: Reversed incremental backups are getting longer and long

Post by GregorS »

My first intermediate active full backup is now created but the problem didn't go away. Curiously, even during the full there was a more than ten minutes waiting time period for each and every VM at phase "Using source proxy <backup proxy name> [hotadd]". Can someone please elaborate what Veeam Backup is doing during this exact phase?
Some more hints:
  • at the beginning there were no noticeable delays
  • I can see through history that the delay is getting worse with about 11 minutes at present
  • I perceive that the delay is getting a bit longer at every backup run
  • the delay is independent of the characteristics of a VM, number of VMs and now even of the type of a backup job and it seems to be equally long for each VM
  • it's probably linearly correlated only to the number of increments
Thanks
Gregor
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Reversed incremental backups are getting longer and long

Post by foggy »

Gregor, debug logs should clearly tell what is being performed during this delay. I suggest to contact our technical team so they could assist you in reading the job logs.
GregorS
Veeam ProPartner
Posts: 48
Liked: 11 times
Joined: Apr 10, 2013 12:11 pm
Full Name: Gregor Smerke
Location: Ljubljana, SI - Slovenia
Contact:

Re: Reversed incremental backups are getting longer and long

Post by GregorS »

I checked the logs as suggested:
  1. job log files are enormous, the current Job.Primary_Backup_Job.Backup.log is 2.7 GB uncompressed and 300 MB ZIP
  2. since this is barely uploadable I rather looked inside the log, at least at the beginning of it:

Code: Select all

===================================================================
Continuing interrupted log...
Log has been started by 'ONKO_NET\svcVeeam' user (Non-interactive)
Logging level: [4 (AboveNormal)]
MachineName: [ONKO-VEE], OS: [Microsoft Windows NT 6.1.7601 Service Pack 1]
Process: [64 bit], PID: [3500], SessionId: [0]
UTC Time: [15.04.2013 16:00:17], DaylightSavingTime: [True]
Culture: [sl-SI], UI culture: [en-US]
...
[16.04.2013 06:01:04] <01> Info     [OIB] Finding corrupted oibs in group: 'e8b17b8a-ae02-4b8c-8203-c0e6379827b8'
[16.04.2013 06:01:04] <01> Info     [OIB] No corrupted oibs in group: 'e8b17b8a-ae02-4b8c-8203-c0e6379827b8' found
[16.04.2013 06:01:04] <01> Info     [BackupMeta] Generate backup meta for 'Primary Backup Job:d42bb7f6-2c5e-4225-808f-c3a5b403a1f1'
[16.04.2013 06:01:47] <01> Info     [BackupMeta] Saving backup meta to 'D:\Primary Backup Job\Primary Backup Job.vbm'
[16.04.2013 06:01:47] <01> Info         [FC] Getting item info 'D:\Primary Backup Job'.
[16.04.2013 06:01:49] <01> Info     [ONKO-VEE] [FC] Saving file content '<BackupMeta Id="f883024d-eabc-4b75-b3f9-8d46083eea6a" JobId="d6503...
The log seems normal from the beginning, it enumerates the VMs for backup, then it starts to deal with the first VM and then it comes to "generating backup meta" where the last line in the output above is absurdly long to the infinity and beyond and it just keeps on being longer with every incremental run. I think this is what hogs my backup. But is this normal behaviour?

Gregor
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Reversed incremental backups are getting longer and long

Post by foggy »

Please provide logs to our technical support and continue pursuing this issue with them directly. The product management team behind this forum is not involved in technical support and is unable to assist with reading debug logs. Please inform your support engineer about the size of the log files when opening the ticket, and you will be provided with a link to the FTP server to upload this amount of debug logs files.
fast_cat
Influencer
Posts: 24
Liked: 1 time
Joined: Nov 06, 2012 11:56 am
Contact:

Re: Reversed incremental backups are getting longer and long

Post by fast_cat » 1 person likes this post

Hi,

Check out this post to see if you have the same issue that I was seeing....(its the posts towards the end of the discussion that demonstrate one reason why backup jobs appear to get slower..at least for my environment)

regards
Rob
GregorS
Veeam ProPartner
Posts: 48
Liked: 11 times
Joined: Apr 10, 2013 12:11 pm
Full Name: Gregor Smerke
Location: Ljubljana, SI - Slovenia
Contact:

Re: Reversed incremental backups are getting longer and long

Post by GregorS »

Thanks to Fast Cat for adding a clue. I checked the VMwareSnapshots table, there are 25 rows and I assume 24 are stale but I don't think this is the root cause for my delays. It seems that my backup job is somehow combining data about all previous backup runs into metadata at the beginning of each VM's backup.
wmoore
Lurker
Posts: 2
Liked: never
Joined: Aug 06, 2012 6:00 am
Full Name: Wayne Moore
Contact:

Re: Reversed incremental backups are getting longer and long

Post by wmoore »

We backup about 15TB per night (that is, about 15TB of source data) using reversed incrementals and we had something similar happen to us - jobs would take longer, jobs would fail - the bigger VMs started failing more often than succeeding. The logs kept telling us "couldn't read the next block" or something equally non-descriptive.

Our backup target was down to about 1.2TB of space and we did some cleanup to recover about 1TB and suddenly the backups are flying again, and completing with 100% success. It seems like Veeam needs a lot of scratch space to work.
GregorS
Veeam ProPartner
Posts: 48
Liked: 11 times
Joined: Apr 10, 2013 12:11 pm
Full Name: Gregor Smerke
Location: Ljubljana, SI - Slovenia
Contact:

Re: Reversed incremental backups are getting longer and long

Post by GregorS »

Wayne (wmoore), thank you for your input. My backup repository's capacity is 24.5 TB with 6.6 TB free (more than 25%). My full backup is around 1.5 TB and increments are less than 100 GB. I still think it's something with metadata...
GregorS
Veeam ProPartner
Posts: 48
Liked: 11 times
Joined: Apr 10, 2013 12:11 pm
Full Name: Gregor Smerke
Location: Ljubljana, SI - Slovenia
Contact:

Re: Reversed incremental backups are getting longer and long

Post by GregorS » 5 people like this post

I would just like to report that the problem is now solved thanks to Alexander Shelemin of Veeam Support. Deleting the whole contents of the table VMwareSnapshots in Veeam backup database in Microsoft SQL Server was sufficient. There were many leftover snapshots journaled in, because my backup job occasionally freezes while removing the snapshot and I had to stop the running job forcefully many times.

G.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Reversed incremental backups are getting longer and long

Post by foggy »

Thanks for getting back with the update, Gregor! Much appreciated.
Yuki
Veeam ProPartner
Posts: 252
Liked: 26 times
Joined: Apr 05, 2011 11:44 pm
Contact:

Re: Reversed incremental backups are getting longer and long

Post by Yuki »

Can someone please provide step-by-step instructions for this. I would like to see if we are being affected by this (we also had to kill a few jobs that got stuck and manually clean up), so we could potentially be in the same situation.
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Reversed incremental backups are getting longer and long

Post by Vitaliy S. » 1 person likes this post

Yuki, I would strongly recommend performing all database tables removal tasks with the help of our support team.
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Reversed incremental backups are getting longer and long

Post by veremin »

I would like to see if we are being affected by this.
As mentioned in this post, in order to determine whether you have the said issue or not, you might want to query –dbo-VMware Snapshot table, see how many entries it has and compare them with the results from vCenter.

Anyway, feel free to contact our support team and ask them to assist you with this.

Thanks.
GregorS
Veeam ProPartner
Posts: 48
Liked: 11 times
Joined: Apr 10, 2013 12:11 pm
Full Name: Gregor Smerke
Location: Ljubljana, SI - Slovenia
Contact:

Re: Reversed incremental backups are getting longer and long

Post by GregorS »

Here's some PowerShell code you can reuse to delete rows from the table that are not associated with real snapshots in the virtual infrastructure. First download and install the following, best on your VBR server:
  1. Windows Management Framework 3.0, this'll give you Microsoft Windows PowerShell 3.0 on supported platforms, if you don't have it already
  2. VMware vSphere PowerCLI
  3. Microsoft Windows PowerShell Extensions for Microsoft SQL Server 2012 SP1, it works well with older versions of SQL Server too

Code: Select all

Add-PSSnapin -Name "VeeamPSSnapin", "VMware.VIMAutomation.Core"
Import-Module -Name "SQLPS" -DisableNameChecking
Connect-VIServer -Server "YourVCServerName"
$SnapshotRef = Invoke-SQLCmd -Query "SELECT * FROM VMwareSnapshots" -ServerInstance "YourSqlServerName" -Database "YourSqlDatabaseName" | ForEach-Object {$_.Snapshot_ref}
$SnapshotId = Get-VM | Get-Snapshot | ForEach-Object {$_.Id.Replace("VirtualMachineSnapshot-", "")}
$SnapshotRef | Where-Object {$SnapshotId -notcontains $_} | ForEach-Object {Invoke-SQLCmd -Query "DELETE FROM VMwareSnapshots WHERE Snapshot_ref = '$_'" -ServerInstance "YourSqlServerName" -Database "YourSqlDatabaseName"}
You can get your SqlServerName and SqlDatabaseName in the following system registry location on your VBR server:
HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Reversed incremental backups are getting longer and long

Post by foggy »

And one more time I want to stress that:
Vitaliy S. wrote:Yuki, I would strongly recommend performing all database tables removal tasks with the help of our support team.
Also, you should keep in mind that database structure can be changed between product releases. Thanks!
Post Reply

Who is online

Users browsing this forum: No registered users and 295 guests