Comprehensive data protection for all workloads
YouAllHaveZoidberg
Novice
Posts: 8
Liked: never
Joined: Sep 25, 2020 3:14 pm
Full Name: Eric Pagan
Contact:

Re: V11 and Rotated Backup Copy Error Email

Post by YouAllHaveZoidberg »

You made a simple change, and now that I understand it I have adjusted the restore points accordingly. It's impact isn't simple though. Now, I have to worry how those restore points get scattered across the drives. They won't be even, and they won't be predictable. In a perfect world, the drive will get rotated every weekday except for Friday. However, there are holidays, hardware failures, missed courier runs, incorrect drive exchanges (wrong day) and the like. To account for that, I now have to reduce the total number of restore points to make sure one drive in particular doesn't overfill. I'm still not quite sure where the number of restore points will fall on Friday (backups run on Saturday and Sunday since some people work weekends) thus adding more confusion and frustration. Even still, if one does start to tip - how do I fix it? I suppose temporarily reducing the restore points for the one drive and then bumping it back up would work, but I shouldn't have to. We're taking an automated process and turning it into something manual.

We've only been using Veeam since v9, so to me the way you had it set before was the right way. There were no hard questions - the only one I had to ask was how much space is there? It was also extremely easy to explain to executives and auditors. I have 12 weeks of restore points on this backup drive (except for Friday). I had no problem with the retention policy overfilling - it was predictable. Space would fill up on all drives at the same rate.

Take it for what you will. I can't add anything of value to this conversation after this or I'd just be repeating myself. If the executives are on board, we'll be moving to Wasabi instead of rotation anyway now that immutability is a go.
Charles.yang
Enthusiast
Posts: 26
Liked: 3 times
Joined: Jun 04, 2021 12:47 am
Contact:

Re: V11 and Rotated Backup Copy Error Email

Post by Charles.yang »

No official docs about retention policy for rotated media so far. However, if the R&B V11's retention points calculation method is by design, not a bug, this should be a big change, which caught Veeam users on the wrong foot. If it's a bug, this issue could be a disaster.

From my point of view, I support retention point is per device strongly !!
Mildur
Product Manager
Posts: 8549
Liked: 2223 times
Joined: May 13, 2017 4:51 pm
Full Name: Fabian K.
Location: Switzerland
Contact:

Re: V11 and Rotated Backup Copy Error Email

Post by Mildur » 1 person likes this post

Hi Charles

Are you looking for that one? Last updated in April 2021.
The Method „retention Policy: keep restore points across all device“ is documented there.

https://helpcenter.veeam.com/docs/backu ... ml?ver=110
When you specify retention settings for a backup job or a backup copy job targeted at a backup repository with rotated drives, you must define the total number of restore points that you want to retain on all drives in the set. For example, if you set retention to 14, the job will keep the total of 14 restore points across all drives.
Product Management Analyst @ Veeam Software
Greg O
Novice
Posts: 7
Liked: 1 time
Joined: Mar 28, 2019 5:33 am
Full Name: Greg Oliver
Contact:

Re: V11 and Rotated Backup Copy Error Email

Post by Greg O »

Gostev wrote: Jun 18, 2021 12:10 pm That is correct. Retention policy is a property of the job and it does not matter where the particular backup file is stored physically. If it fell out of the retention policy, it will be deleted when it becomes reachable.

As far as I understood, V10 behavior is a bug that was fixed because customers complained backups are not deleted according to their retention policy, overfilling their rotated drives.
If it does not matter where the particular backup file is stored physically - then again how I can establish how many retention points are there in total across all of the drives. If choose a restore from this particular job, it tells me I have 8 retention points because it is only looking at the current drive loaded (not the total across all of the drives). If retention points are calculated across all drives, then this value must be stored in the database somewhere ?

Until I changed the retention points value to a higher number, the backup jobs on the other disks were deleted. So now have a uneven number of retention points across the disks, the current one has 8, others in the set have one. Hence the backup with the current disk is just about to fail because it is full. Whereas the others in the set have plenty of space.

If it fell out of the retention policy, it will be deleted when it becomes reachable - if the drive is lost or has failed, it will never becomes reachable because it is dead.

A useful function of the software has become difficult to manage. It would now be easier to forget about rotated drives, backup to a local drive and then upload them to the external drive with a Robocopy mirror.
Charles.yang
Enthusiast
Posts: 26
Liked: 3 times
Joined: Jun 04, 2021 12:47 am
Contact:

Re: V11 and Rotated Backup Copy Error Email

Post by Charles.yang »

Mildur wrote: Jun 19, 2021 4:56 am Hi Charles

Are you looking for that one? Last updated in April 2021.
The Method „retention Policy: keep restore points across all device“ is documented there.

https://helpcenter.veeam.com/docs/backu ... ml?ver=110
Thanks for that.

It seems in V10 and early versions, the retention point for rotated media is per device.
Gostev
Chief Product Officer
Posts: 31456
Liked: 6646 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: V11 and Rotated Backup Copy Error Email

Post by Gostev »

Here are the archived manuals for version 10 and version 9.5, they all have the same note. So it was the intended behavior for at least 5 years now. Perhaps there was some configuration where this functionality did not work correctly, and this bug was fixed in V11.
Charles.yang
Enthusiast
Posts: 26
Liked: 3 times
Joined: Jun 04, 2021 12:47 am
Contact:

Re: V11 and Rotated Backup Copy Error Email

Post by Charles.yang »

Not really, in V9.5 docs: it says "For example, if you set retention to 7, the job will keep 7 restore points on every drive in the set." So it seems the retention policy was changed from V10, the bug was fixed from V11. WOW!!!!
Most of users who are experiencing this problem upgraded B&R from V9.5

Follows are the notes from Veeam documentation.

V9.5
When you specify retention settings for a backup copy job targeted at a backup repository with rotated drives, you must define the number of restore points per drive. For example, if you set retention to 7, the job will keep 7 restore points on every drive in the set.

V10
When you specify retention settings for a backup job targeted at a backup repository with rotated drives, you must define the total number of restore points that you want to retain on all drives in the set. For example, if you set retention to 14, the job will keep the total of 14 restore points across all drives.

V11
When you specify retention settings for a backup job or a backup copy job targeted at a backup repository with rotated drives, you must define the total number of restore points that you want to retain on all drives in the set. For example, if you set retention to 14, the job will keep the total of 14 restore points across all drives.
Gostev
Chief Product Officer
Posts: 31456
Liked: 6646 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: V11 and Rotated Backup Copy Error Email

Post by Gostev »

You are right, sorry I was looking at the first note in each article, which is the same for all three versions (retention across all drives). But apparently, Backup Copy job was a weird exception with its own note and behavior.
m.novelli
Veeam ProPartner
Posts: 504
Liked: 84 times
Joined: Dec 29, 2009 12:48 pm
Full Name: Marco Novelli
Location: Asti - Italy
Contact:

Re: V11 and Rotated Backup Copy Error Email

Post by m.novelli »

So, long story short, please revert to v9.5 behavior 😃
Gostev
Chief Product Officer
Posts: 31456
Liked: 6646 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: V11 and Rotated Backup Copy Error Email

Post by Gostev »

This would mean introducing a bug in the product where the retention policy specified in the job settings does not apply. And we will have no good logical answer to customers who will have an issue with their retention policy not deleting backups.

By the way, after digging this more, part of the reason for this non-standard Backup Copy job retention behavior appears to be the fact that for historical reasons, BCJ used its own and very special backup file chain processing with unique things like "overbuilds". This is no longer the case with V11 where we removed all that code (particularly due to it being a constant source of nasty bugs due to complexity) and replaced everything with the single universal engine taken from primary jobs. So at this point, returning any legacy behaviors which relied on all that removed complexity would be particularly challenging, if not impossible.
m.novelli
Veeam ProPartner
Posts: 504
Liked: 84 times
Joined: Dec 29, 2009 12:48 pm
Full Name: Marco Novelli
Location: Asti - Italy
Contact:

Re: V11 and Rotated Backup Copy Error Email

Post by m.novelli » 1 person likes this post

Ok, anyway I vote to have restore point counted per device with Backup Copy Job

The global count across all rotated media is challenging / complicated / unfeasible

Since v11 I had to configure a pre-script job to format the rotated media before Copy Job due to many errors of media full / error merging / error I don’t know 🤦🏻‍♂️
Charles.yang
Enthusiast
Posts: 26
Liked: 3 times
Joined: Jun 04, 2021 12:47 am
Contact:

Re: V11 and Rotated Backup Copy Error Email

Post by Charles.yang »

Gostev wrote: Jun 21, 2021 12:56 pm This would mean introducing a bug in the product where the retention policy specified in the job settings does not apply. And we will have no good logical answer to customers who will have an issue with their retention policy not deleting backups.

By the way, after digging this more, part of the reason for this non-standard Backup Copy job retention behavior appears to be the fact that for historical reasons, BCJ used its own and very special backup file chain processing with unique things like "overbuilds". This is no longer the case with V11 where we removed all that code (particularly due to it being a constant source of nasty bugs due to complexity) and replaced everything with the single universal engine taken from primary jobs. So at this point, returning any legacy behaviors which relied on all that removed complexity would be particularly challenging, if not impossible.
Thanks Gostev. To my understanding, in V9.5, backup job and backkup copy job use different engines with different retention policies. In V11/V10 they some codes are removed, backup job and backup copy job use same engine/policy.

My question is, why don't count the retention points per device with a normal backup job? A normal backup job basically has no rotated media as a backup copy job.
Gostev
Chief Product Officer
Posts: 31456
Liked: 6646 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: V11 and Rotated Backup Copy Error Email

Post by Gostev »

We simply never had this concept "retention per device" in backup jobs, so the internals are not designed to support this. Even scale-out backup repositories, which do a lot of data management operations on its own like moving some backups between different devices at certain age, still adhere to that single global retention policy specified at the backup job level.
Gostev
Chief Product Officer
Posts: 31456
Liked: 6646 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: V11 and Rotated Backup Copy Error Email

Post by Gostev »

m.novelli wrote: Jun 21, 2021 6:51 pmThe global count across all rotated media is challenging / complicated / unfeasible
I would agree, but you really don't have to be doing this! Just switch to the time-based retention from your current restore point based. Time-based retention is super straight-forward, plus it directly maps to customers' business requirements on how long they want to keep their backups. It is always hard to predict how much time your selected number of restore points would cover due to missed and/or accidental runs, but 4 weeks is always 4 weeks!
Charles.yang
Enthusiast
Posts: 26
Liked: 3 times
Joined: Jun 04, 2021 12:47 am
Contact:

Re: V11 and Rotated Backup Copy Error Email

Post by Charles.yang » 1 person likes this post

In production, for a rotated media, retention point may be better, as a rotated media is often small, free space is critical. If a backup copy job runs more times than usual accidentally, will run out of space. If it's controlled by a retention point policy, it could be easier.

BTW retention point policy starts from V10, doesn't it? Software should meet customers' requirements, not the other way around.
je1000
Service Provider
Posts: 20
Liked: 7 times
Joined: Mar 05, 2021 4:59 pm
Contact:

Re: V11 and Rotated Backup Copy Error Email

Post by je1000 »

Gostev wrote: Jun 22, 2021 12:26 am I would agree, but you really don't have to be doing this! Just switch to the time-based retention from your current restore point based. Time-based retention is super straight-forward, plus it directly maps to customers' business requirements on how long they want to keep their backups. It is always hard to predict how much time your selected number of restore points would cover due to missed and/or accidental runs, but 4 weeks is always 4 weeks!
Time bases policy could be disastrous with rotated media. It could delete your last good backup.

Whatever you want to call it, it was good bug and we need it back. retention per drive on rotated media is a most, If you made a poll on people actually using it, most will want to have per drive, if you don’t agree then make it as option, this is severely impacted my customers who are using rotated media both the false positive and so called fix. It should resolved quickly.
Mildur
Product Manager
Posts: 8549
Liked: 2223 times
Joined: May 13, 2017 4:51 pm
Full Name: Fabian K.
Location: Switzerland
Contact:

Re: V11 and Rotated Backup Copy Error Email

Post by Mildur » 1 person likes this post

Time bases policy could be disastrous with rotated media. It could delete your last good backup.
Time based Policys with Veeam will always keep the last three successful backups, if there isn't any new successful Run of the backup. Or the backup job was stopped for a few days. You will have this three restore points.
This value can be adjusted with a registry key.

https://helpcenter.veeam.com/docs/backu ... ml?ver=110
The minimum number of retained restore points is 3. This number does not depend on the number of days set in the retention policy. For example, the retention policy is set to 5 days. You launch the job after it was stopped for 10 days. Normally, Veeam Backup & Replication would delete all previous restore points. However, due to the minimum number of retained restore points, you will still have at least 3 restore points: the newly created restore point and the two previous ones.
You can change the minimum number of retained restore points with a registry key. For more information, contact Veeam Customer Support.
Product Management Analyst @ Veeam Software
Charles.yang
Enthusiast
Posts: 26
Liked: 3 times
Joined: Jun 04, 2021 12:47 am
Contact:

Re: V11 and Rotated Backup Copy Error Email

Post by Charles.yang »

je1000 wrote: Jun 22, 2021 10:56 am Time bases policy could be disastrous with rotated media. It could delete your last good backup.

Whatever you want to call it, it was good bug and we need it back. retention per drive on rotated media is a most, If you made a poll on people actually using it, most will want to have per drive, if you don’t agree then make it as option, this is severely impacted my customers who are using rotated media both the false positive and so called fix. It should resolved quickly.
Absolutely couldn't agree more. The best solution is to enable a per-device feature or have options.
m.novelli
Veeam ProPartner
Posts: 504
Liked: 84 times
Joined: Dec 29, 2009 12:48 pm
Full Name: Marco Novelli
Location: Asti - Italy
Contact:

Re: V11 and Rotated Backup Copy Error Email

Post by m.novelli »

Charles.yang wrote: Jun 22, 2021 1:59 am In production, for a rotated media, retention point may be better, as a rotated media is often small, free space is critical. If a backup copy job runs more times than usual accidentally, will run out of space. If it's controlled by a retention point policy, it could be easier.

BTW retention point policy starts from V10, doesn't it? Software should meet customers' requirements, not the other way around.
Fully agree

Marco
m.novelli
Veeam ProPartner
Posts: 504
Liked: 84 times
Joined: Dec 29, 2009 12:48 pm
Full Name: Marco Novelli
Location: Asti - Italy
Contact:

Re: V11 and Rotated Backup Copy Error Email

Post by m.novelli »

Gostev wrote: Jun 21, 2021 12:56 pm By the way, after digging this more, part of the reason for this non-standard Backup Copy job retention behavior appears to be the fact that for historical reasons, BCJ used its own and very special backup file chain processing with unique things like "overbuilds". This is no longer the case with V11 where we removed all that code (particularly due to it being a constant source of nasty bugs due to complexity) and replaced everything with the single universal engine taken from primary jobs. So at this point, returning any legacy behaviors which relied on all that removed complexity would be particularly challenging, if not impossible.
Now I have a better big picture

Since v11 update most of my customers with Backup Copy Job to RDX drives started to have issues both during copy and merge phase, failing with error Agent failed to process method {DataTransfer.BackupText}

I even opened a Case # 04800153 , without real help from support. The behavior exhibited by Backup Copy Job to RDX drives after v11 update was the copy throughput going slower and slower after some minutes , from 100 - 115 MB/sec to 15 - 20 MB/sec , leading to failed export

I solved by myself with two workaroud: reduce the parallel copy to RDX drives from 4 to 1 and for bigger customers (about more than 2 TB of data to export) format the drive at each execution of Backup Copy Job

Before v11 I never had so many issues with RDX drives, now I understand that the single universal engine is not really RDX friendly :(

I'm talking about 10 - 15 customers with this issue. Where possibile I'm trying to move from RDX export to Azure Blob Storage export , but this increase the overall costs (both Veeam Enterprise license and Azure storage costs)

Marco
Gostev
Chief Product Officer
Posts: 31456
Liked: 6646 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: V11 and Rotated Backup Copy Error Email

Post by Gostev » 1 person likes this post

je1000 wrote: Jun 22, 2021 10:56 amTime bases policy could be disastrous with rotated media. It could delete your last good backup.
Charles.yang wrote: Jun 23, 2021 1:27 amAbsolutely couldn't agree more.
Please note that this can never happen: time-based retention will never delete your last good backup. As Mildur already explained above, it will always keep last 3 good backups (this is by default, the number is configurable through registry). So, there should be no concerns switching to the time-based retention from this perspective.
YouAllHaveZoidberg
Novice
Posts: 8
Liked: never
Joined: Sep 25, 2020 3:14 pm
Full Name: Eric Pagan
Contact:

Re: V11 and Rotated Backup Copy Error Email

Post by YouAllHaveZoidberg »

Gostev wrote: Jun 21, 2021 12:56 pm This would mean introducing a bug in the product where the retention policy specified in the job settings does not apply. And we will have no good logical answer to customers who will have an issue with their retention policy not deleting backups.

By the way, after digging this more, part of the reason for this non-standard Backup Copy job retention behavior appears to be the fact that for historical reasons, BCJ used its own and very special backup file chain processing with unique things like "overbuilds". This is no longer the case with V11 where we removed all that code (particularly due to it being a constant source of nasty bugs due to complexity) and replaced everything with the single universal engine taken from primary jobs. So at this point, returning any legacy behaviors which relied on all that removed complexity would be particularly challenging, if not impossible.
I'd like to latch onto this for a moment. To me, this is software regression and you don't have an answer for those of us affected by it. It sounds like you know the exact behavior that was giving us the restore point per drive. Given the Veeam team's talent, this "legacy" behavior could be implemented once more as a standalone feature. If you know what it is and how to remove it, then you know how to add it back. You could make it a toggle as suggested earlier. This way, everyone is satisfied. Surely this is within the Veeam's programmers capability. Please understand why you're getting push back on this and help us with a real solution.
Gostev
Chief Product Officer
Posts: 31456
Liked: 6646 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: V11 and Rotated Backup Copy Error Email

Post by Gostev »

m.novelli wrote: Jun 23, 2021 8:13 amBefore v11 I never had so many issues with RDX drives, now I understand that the single universal engine is not really RDX friendly :(
Actually, I was talking purely about the retention engine here. In case of any performance issues, you should look at them separately as this would be unrelated to this discussion or rotated drives in general, but rather to the core engine changes of how we interact with backup storage in V11 to better support enterprise-grade storage(see What's New). There is a big topic about this here, but most likely your performance issue will go away after you enable the Align options in the advanced backup repository setting.
m.novelli
Veeam ProPartner
Posts: 504
Liked: 84 times
Joined: Dec 29, 2009 12:48 pm
Full Name: Marco Novelli
Location: Asti - Italy
Contact:

Re: V11 and Rotated Backup Copy Error Email

Post by m.novelli »

I know Gostev that you feel super guru of Veeam , but sometimes when people write "I solved by myself with two workaroud after opening a support case" please understand that tech people already tried a super simple solution like clicking "Align blocks" and instead there is real bug / regression in Veeam 😉

Marco
Gostev
Chief Product Officer
Posts: 31456
Liked: 6646 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: V11 and Rotated Backup Copy Error Email

Post by Gostev »

I definitely cannot assume this ;) in fact, you're likely the first person to try and do these few things without our instructions. The reality is, most people don't follow the forum discussions closely to even know about these simple solutions!
Charles.yang
Enthusiast
Posts: 26
Liked: 3 times
Joined: Jun 04, 2021 12:47 am
Contact:

Re: V11 and Rotated Backup Copy Error Email

Post by Charles.yang »

Gostev wrote: Jun 23, 2021 12:57 pm Please note that this can never happen: time-based retention will never delete your last good backup. As Mildur already explained above, it will always keep last 3 good backups (this is by default, the number is configurable through registry). So, there should be no concerns switching to the time-based retention from this perspective.
Those have happened in my production for many times, a chain is removed at all in a rotated media when the restore point has reached the retention point number.

Why you always force us to use a time-base retention policy? Restore point is a feature of Veeam and it doesn't work perfectly. You should forward our requests to Veeam development team, thank you in advance.
Gostev
Chief Product Officer
Posts: 31456
Liked: 6646 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: V11 and Rotated Backup Copy Error Email

Post by Gostev »

Restore-point based retention is no longer a default setting, and we will be deprecating this option in future because it is not compatible with many of the new product features. Thus, even if it is available today, eventually it will no longer be a feature of Veeam. As such, we're not planning any enhancements around restore point based retention going forward. But we do want to keep improving time-based retention.
Charles.yang wrote: Jun 24, 2021 1:51 amThose have happened in my production for many times, a chain is removed at all in a rotated media when the restore point has reached the retention point number.
Since there's a code in place that specifically prevents this, it could only be due to some environment-specific bug. If you can reproduce this, please open a support case so we can take a look through debug logs what happens, and address this - then this won't be happening again for you. Thanks!
Charles.yang
Enthusiast
Posts: 26
Liked: 3 times
Joined: Jun 04, 2021 12:47 am
Contact:

Re: V11 and Rotated Backup Copy Error Email

Post by Charles.yang »

Deprecating retention point feature? OK, all of us can go home now.
Gostev
Chief Product Officer
Posts: 31456
Liked: 6646 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: V11 and Rotated Backup Copy Error Email

Post by Gostev »

Correct, as some new features we're building cannot be made working with the restore point-based retention.
Charles.yang
Enthusiast
Posts: 26
Liked: 3 times
Joined: Jun 04, 2021 12:47 am
Contact:

Re: V11 and Rotated Backup Copy Error Email

Post by Charles.yang »

Gostev wrote: Jun 24, 2021 1:45 pm Restore-point based retention is no longer a default setting, and we will be deprecating this option in future because it is not compatible with many of the new product features. Thus, even if it is available today, eventually it will no longer be a feature of Veeam. As such, we're not planning any enhancements around restore point based retention going forward. But we do want to keep improving time-based retention.


Since there's a code in place that specifically prevents this, it could only be due to some environment-specific bug. If you can reproduce this, please open a support case so we can take a look through debug logs what happens, and address this - then this won't be happening again for you. Thanks!
You may take a look the replies in this thread, many users reported the chains are deleted and rebuilt in rotated medias. That's the reason and why we are here. I raised a ticket to Veeam support with logs, was told, this is a known issue and then the case was archived.
Locked

Who is online

Users browsing this forum: Google [Bot], xzvf and 205 guests