Agent-based backup of Windows, Linux, Max, AIX and Solaris machines.
Post Reply
gberkshier-cefcu
Influencer
Posts: 19
Liked: 1 time
Joined: Apr 03, 2017 6:31 pm
Full Name: Glenn Berkshier
Contact:

Duplicated nodes in backup copy job of a clustered file share backup

Post by gberkshier-cefcu »

I put a ticket in for this, but maybe that wasn't necessary. I'm just wanting to get some clarification. If the ticket is needed, I can add the support case ID to the this post.

We have a 2 node windows file share cluster. Each node is configured with iSCSI connected disks to our Nimble storage appliance. The nodes are set with active/passive states, so the active node for the cluster has the iSCSI disks connected, while the passive doesn't. We installed the Veeam agent on both nodes a few years ago, when the Veeam agent first came out. It was the only way we could get the backups completed, because Veeam didn't have a way to backup the data in the VM backup. I have these nodes setup in a protection group in Veeam, and I have an agent backup job that runs to backup the nodes, but I'm only backing up the iSCSI disks in that job. The actual VM backup is done in a separate job for these nodes.

The data being backed up with the agent backup job is fairly sizable - we have three separate iSCSI disks being backed up in separate jobs - disk 1 (~250 GB) in job 1, disk2 (~6.5 TB) in job 2, and disk 3 (~4.5 TB) in job 3. The main backup jobs run fine, and behave as I would expect - the active node gets the backup, because it has the iSCSI disks mounted. The backup files themselves show the actual cluster name being backed up, so I can see the data is getting backed up. However, the corresponding backup copy jobs that I have tied to these main jobs are copying the data to our DR repository just fine, but at the DR repository, what I see is that each node is getting backed up with what appears to be all of the data on each one, from the main cluster backup job, regardless of whether or not the node was active or not. Essentially, I'm seeing that I need double the space for storage, because the backup copy job data at DR is being done twice - one for each node - from the main backup job. For every backup job, the backup copy job shows data files for both nodes, almost exactly the same size.

Is this how the backup copy job should work? If so, that's not good, because I now need twice the amount of space at DR than I do at our local site, just for the same data, because it's making both nodes with their own copies in the backup data files.
Dima P.
Product Manager
Posts: 14396
Liked: 1568 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: Duplicated nodes in backup copy job of a clustered file share backup

Post by Dima P. »

Hello Glenn,
have these nodes setup in a protection group in Veeam, and I have an agent backup job that runs to backup the nodes, but I'm only backing up the iSCSI disks in that job.
Can you please clarify if that's a failover cluster job type managed by Veeam B&R or regular server job type? With failover cluster you must add cluster account from AD to the protection group and all the nodes and cluster disks are stored within single backup file. Such backup can be further copied via backup copy job without any data duplication. I assume that you have standalone node backup and each node might have disk state included in the backup, thus when you create a backup copy disks might be processed several types as independent and related to it's node backup.
gberkshier-cefcu
Influencer
Posts: 19
Liked: 1 time
Joined: Apr 03, 2017 6:31 pm
Full Name: Glenn Berkshier
Contact:

Re: Duplicated nodes in backup copy job of a clustered file share backup

Post by gberkshier-cefcu »

I have the cluster added to Veeam in a protection group, and the main backup definitely backs up the cluster, because the backup job clearly shows the cluster name, and I can see "processing cluster node" in the logs for each node. It processes each node, and I have the one file of backup data for the cluster. The backup copy job is set to the related backup job in the "objects to process". If I add new objects, I can choose between "from backups", and "from jobs". I chose "from jobs" when I setup this backup copy job. If I choose "from backups", I see the job and the nodes in there. Selecting that option added the individual nodes to the backup objects to process, instead of the job.

Is there a reason for having two different ways to add these objects, and if so, what is that reason?

UPDATE: I deleted that backup data from the backup copy job, and re-added the objects to the backup job with the "from backups" option instead of the "from jobs" option, and it made no difference. It started the job, and it's creating duplicate restore point data files for each cluster node. I don't understand why this backup copy job wants to create data for each node, when only one node got backed up in the main backup job. I don't need duplicated node backup data - just a backup copy of the cluster file share data. I'm still failing to understand why the backup copy job would be copying the data like this. It's doubling the required amount of space needed for the backup copy. I have these backups going to a repository that's an ReFS formatted volume on a Windows server, and maybe that would technically cut the amount of data being stored, but it still is clearly creating duplicate data for both nodes in the backup copy folder.
Dima P.
Product Manager
Posts: 14396
Liked: 1568 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: Duplicated nodes in backup copy job of a clustered file share backup

Post by Dima P. »

Hello Glenn,

Thank you for the updates!
Is there a reason for having two different ways to add these objects, and if so, what is that reason?
Working with cluster account give us benefits of ownership tracking and node detection. Simply put, whenever you add nodes as regular servers there will be no 'cluster-awareness' but that does not mean you cant use this approach to protect IP DAG or some specific node of your cluster.
I deleted that backup data from the backup copy job, and re-added the objects to the backup job with the "from backups" option instead of the "from jobs" option, and it made no difference. It started the job, and it's creating duplicate restore point data files for each cluster node.
Can you please raise a support ticket? If we have access to debug logs I can ask our RnD team to review those.
gberkshier-cefcu
Influencer
Posts: 19
Liked: 1 time
Joined: Apr 03, 2017 6:31 pm
Full Name: Glenn Berkshier
Contact:

Re: Duplicated nodes in backup copy job of a clustered file share backup

Post by gberkshier-cefcu »

I do have a ticket in - #04868386. I put it in before posting in this forum. I actually got in contact with someone later that day, and now was told that I've been escalated to the Veeam Agent support team, so I can post more details in here as I get them from my ticket.
Dima P.
Product Manager
Posts: 14396
Liked: 1568 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: Duplicated nodes in backup copy job of a clustered file share backup

Post by Dima P. »

Thanks for the update! I'll share your case with RnD folks.
Dima P.
Product Manager
Posts: 14396
Liked: 1568 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: Duplicated nodes in backup copy job of a clustered file share backup

Post by Dima P. »

Glenn,

Based on case details looks like the target repository is running with per-machine backup file option enabled. With this option on every machine will have a shared disk in it's backup file thus you see the data being duplicated.
gberkshier-cefcu
Influencer
Posts: 19
Liked: 1 time
Joined: Apr 03, 2017 6:31 pm
Full Name: Glenn Berkshier
Contact:

Re: Duplicated nodes in backup copy job of a clustered file share backup

Post by gberkshier-cefcu »

But the main backup at our local repository is also set for per-machine backups, and I only get one copy of the data there. Why does this happen in the backup copy job? With the way the agent backuo is backing up the cluster, only the active cluster node has drives attached to it, so those get backed up during the process. The passive node portion of the backup is pretty much non-existent, data-wise. If there is no data being backed up from the passive node in the backup on the main backup job, why does Veeam make a complete backup of that data on the passive node with the backup copy if it doesn't exist in the primary backup job?
Dima P.
Product Manager
Posts: 14396
Liked: 1568 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: Duplicated nodes in backup copy job of a clustered file share backup

Post by Dima P. »

Glenn,

Cluster job currently ignores the per-vm setting while backup copy does not. We will correct this behavior in the next major version: backup copy will keep single backup file for all cluster nodes.
gberkshier-cefcu
Influencer
Posts: 19
Liked: 1 time
Joined: Apr 03, 2017 6:31 pm
Full Name: Glenn Berkshier
Contact:

Re: Duplicated nodes in backup copy job of a clustered file share backup

Post by gberkshier-cefcu » 1 person likes this post

Thanks for that info. I wasn't aware this was the case with the backup copy jobs. After all of these discussions and back and forth with Veeam support, not one person has ever mentioned this. I'm going to re-do my repositories for this, since I just recently created them for this as a test, and I will also make sure I'm using ReFS on 2019 server, just so everything is as robust as possible and optimized for any de-duplication where possible.
Dima P.
Product Manager
Posts: 14396
Liked: 1568 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: Duplicated nodes in backup copy job of a clustered file share backup

Post by Dima P. »

I will discuss with technical writers team how to correctly reflect this behavior in the user guide. Thanks for the feedback Glenn!
Post Reply

Who is online

Users browsing this forum: No registered users and 5 guests