Discussions specific to managed agent-based backups
Post Reply
ctg49
Enthusiast
Posts: 54
Liked: 38 times
Joined: Feb 14, 2018 1:47 pm
Full Name: Chris Garlington
Contact:

Failover cluster full backups

Post by ctg49 » Apr 02, 2019 12:55 pm

Quick question.

We have a failover cluster housing some file servers, and periodically (after patching restarts, usually) we get an error in VEEAM about the cluster membership having changed, and a full backup being required. I'd expect this if the membership was different than the previous incremental, but each time I do patching restarts I put the resources back on their source VM (numbered resources, odds on host01, evens on host02). Am I misunderstanding what this error means? Is there something I can do to avoid getting 15TB worth of full backups every month?

Thanks!

Dima P.
Product Manager
Posts: 10536
Liked: 858 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: Failover cluster full backups

Post by Dima P. » Apr 03, 2019 5:01 pm

Hello Chris,

May I ask if the note becomes inaccessible during backup process or you patch and reboot it in advance? Thank you!

ctg49
Enthusiast
Posts: 54
Liked: 38 times
Joined: Feb 14, 2018 1:47 pm
Full Name: Chris Garlington
Contact:

Re: Failover cluster full backups

Post by ctg49 » Apr 03, 2019 8:04 pm

Hi Dima,

Never do failovers/restarts during backups... I checked the backup on it, and it seems to go something like this:

Resource normally on Host02
Failovers done for patching
Failbacks done once patching complete, so resource is back on host02
Backup performed, full backup taken due to being on wrong node
Confirm in VEEAM that backup is showing up under the 'wrong' node
next backup, VEEAM sees it on the 'correct' node, takes *another* full backup.

It's unlikely that the resource is actually on the wrong node after restarts and failbacks are complete, as I don't touch them very often and veeam seems to 'see' it on the correct node the following day. I'm open to suggestions, though.

Dima P.
Product Manager
Posts: 10536
Liked: 858 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: Failover cluster full backups

Post by Dima P. » Apr 05, 2019 6:52 pm

Chris,

Thank you for the details! I've confirmed with QA folks that full backup should not be performed in the case like yours, so please open a case and share the case ID with me (we need to check your application debug logs to understand the root cause). Thank you in advance!

ctg49
Enthusiast
Posts: 54
Liked: 38 times
Joined: Feb 14, 2018 1:47 pm
Full Name: Chris Garlington
Contact:

Re: Failover cluster full backups

Post by ctg49 » Apr 22, 2019 12:14 pm

Updating as this has continued to occur (once on Friday, once on Saturday... Saturday likely just a recurrence as I deleted the 'full incremental' from Friday).

Case #03521834

Dima P.
Product Manager
Posts: 10536
Liked: 858 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: Failover cluster full backups

Post by Dima P. » Apr 30, 2019 6:50 pm

Hello folks,

Need some clarification on your setups:

1. Which Veeam B&R and agent version are you using?
2. Any chance cluster's storage is configured with Storage Spaces?
3. If the answer to last one is yes - please clarify if that Storage Spaces or Storage Spaces Direct?

Thank you in advance!

ctg49
Enthusiast
Posts: 54
Liked: 38 times
Joined: Feb 14, 2018 1:47 pm
Full Name: Chris Garlington
Contact:

Re: Failover cluster full backups

Post by ctg49 » May 01, 2019 12:32 pm

1) The most recent that comes with VEEAM B&R, we're using 9.5U4a, about shows v9.5.4.2753.
2) Nope, just a basic MSCS file server cluster. Two of these are shared VVOLs, four are shared pRDMs (we're mid-transition to shared VVOLs). There's no correlation between pRDM/VVOL and these 'full backup incidents'.

Dima P.
Product Manager
Posts: 10536
Liked: 858 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: Failover cluster full backups

Post by Dima P. » May 03, 2019 10:11 am

Hello Chris,

Thank you for the update! Please make sure that the agents are up to date as well (you can check that via Inventory node) and keep working with out support team. Cheers!

ctg49
Enthusiast
Posts: 54
Liked: 38 times
Joined: Feb 14, 2018 1:47 pm
Full Name: Chris Garlington
Contact:

Re: Failover cluster full backups

Post by ctg49 » May 22, 2019 7:04 pm

Might have an update on this, tech is running down some other information/going to lab it out.

Consider the following scenario:
Cluster01
Clusterhost01
Clusterhost02
Resource01 hosted on host01

Backup is performed, which successfully backs up resource01 from host02 (basically instant, nothing to back up) and fails for some reason backing up resource01 from host01 (in my case, some network hiccup). The job fails that node, but succeeds the job parameters with a warning or whatever. Now the tricky part, a synthetic full backup is created as scheduled for this job, but the synth full has zero or minimal data within. As a result, the next incremental backup sees multiple TB of 'new' data which needs to be backed up to fulfill the chain.

This appears to be a bug, and should probably cause the synth full to fail if the backup of the resource fails. I don't know how this would work if one job was backing up multiple resources, however... we do one job per resource.

Post Reply

Who is online

Users browsing this forum: No registered users and 4 guests