V11 and Rotated Backup Copy Error Email

[Gostev]

You may take a look the replies in this thread, many users reported the chains are deleted and rebuilt in rotated medias. That's the reason and why we are here. I raised a ticket to Veeam support with logs, was told, this is a known issue and then the case was archived.

I guess there's a misunderstanding then. We've been talking about time-based retention in the last few posts, so my comments are related to that. And specifically to protecting the last good backups against deletion because their time-based retention has expired, as this has been voiced as the main concern that prevents people from switching to time-based retention.

You're however referring to issues experienced by users whose restore point based retention was simply too short, and the product was deleting older backups according to the effective retention settings. This is not a "known issue" but rather the correct application of the retention policy in V11. The simple fix is to increase the retention to a higher number, as a few folks in this topic already did. The recommended approach going forward though is to switch to the time-based retention policy.

[Charles.yang]

Sorry about the misunderstanding. Yes, I am referring to the restore point retention policy issue rather than time-base retention policy. Unfortunately, we are not using the time-based retention at the moment.

Thank you again, bye.

[YouAllHaveZoidberg]

I understand you're moving away from restore points. If that's the case, then help me understand time based a bit more. To me, it would seem that I'll still have trouble with the Friday drive being full since 1 day = 1 restore point?

[Gostev]

If you are doing daily backups, then switching to time-based retention should make no difference, as the number (and size) of backup files remains the same whether you use time-based or restore point-based retention. It would be 7 backup files in both cases: 1 full and 6 increments.

If you're saying that your drives are not big enough to hold 7 backup files, then you could:
1. Buy bigger drives.
2. Rotate your existing drive more often (every 5 days for example).
3. Disable backups on some days completely (for example, many small customers don't run backups on Saturday and Sunday).

[je1000]

Please note that this can never happen: time-based retention will never delete your last good backup. As Mildur already explained above, it will always keep last 3 good backups (this is by default, the number is configurable through registry). So, there should be no concerns switching to the time-based retention from this perspective.

With rotated media and the current global rotation it will happen. Simple example, a customer having 3 rotated media, one gets encrypted with ransomware, one broke or got corrupted (happens often with rotated media) and one was accidentally plugged in, a day after the ransomware attack as part of the rotation and veeam deleted it since it exceeded the global retention limit without veeam being aware of the external factors.

The idea of increasing your retention is mostly not practical.
1) Backup files are exceedingly big and close to size to probable usb limits.
2) Relying on human interaction to swap drives just in time is never a good strategy.

In the end I don’t see any benefits of having a global retention for rotated media, its only gonna creating a lot of issues, making using rotated media not practical, so i don’t understand why veeam is defending it, giving unpractical theoretical workarounds. It makes me feel like they are not taking real customer feedback seriously. Having a we know better attitude. I would seriously urge veeam to reach out to their clients who are actually using rotated media and ask them what works better for them.

[Gostev]

Except it is completely the opposite. My understanding is, we had other clients complaining that the retention policy specified in the job settings was not applying correctly on repositories backed by external drives, not deleting backups outside of the retention policy and causing drives to overfill - which in their mind was a functional bug that had to be fixed. And there was no good answer we could give them, because their conclusion is correct. So we did take their feedback and fixed the bug to make the retention policy apply in the documented way.

The issue here is that some other customers actually relied on this bug, and now essentially demanding that we return the bug back but we cannot possibly do this, because we will break it again for the customers who had an issue with the retention policy not applying correctly.

The only realistic way to satisfy the latter group is for us to create the whole new special retention mode that is designed to work in their desired way. But this feature request will have to be prioritized among other pending ones and so is not something that can happen fast, which is why we're sharing workarounds that will work today with the current code, like increasing the retention policy to the desired length. And our "unpractical and theoretical" recommendations is actually us trying to help you solve the immediate issue with the only methods available at this time or in foreseeable future.

[je1000]

Are you saying that veeam will put this back as an option sometime in the near future?

While I fail to see the use case for a global retention for rotated media. I will take your word that you had customers demanding it. I would still argue that you didn’t consider the real world implications before you fixing that so called bug. Implications that makes the use of rotated backups with Veeam in most cases unpractical.

We are waiting on upgrading to version 11 for all our clients with usb backups because of this. So hopefully you will add it back before we are forced to upgrade.

[Gostev]

Definitely not in the near future, as this feature was not planned for the next couple of release vehicles, so the required R&D resources are working on some other priority features. Thus, the better path would be to determine the specific real issues with workarounds (such as increasing the retention policy length) and see if any of those concerns can potentially be addressed with simple fixes.

In fact, from the posts above I understand that some customers have already implemented the workaround with increased retention policy - so I'd just await for their real-world feedback first, instead of us here theorizing about any perceived issues with this approach.

[AlexLeadingEdge]

Hi guys,

Ever since we did the upgrade to B&R version 11 each night we are seeing an error message that says:

13/07/2021 12:00:34 AM :: Failed to pre-process the job Error: Rotated drive was changed, restarting backup copy job

I'm not too concerned by it, but it comes up with a failure after running for 14 seconds. The job is set to start a new cycle at 12AM. We did the upgrade on 24/06/2021, which is the date of the first error.

The job logs go as follows:

13/07/2021 12:00:24 AM :: Job started at 13/07/2021 12:00:20 am
13/07/2021 12:00:25 AM :: New copy interval started
13/07/2021 12:00:25 AM :: Rotated drive letter did not change
13/07/2021 12:00:26 AM :: Building VMs list
13/07/2021 12:00:31 AM :: Rotated drive content has changed from the previous job run
13/07/2021 12:00:34 AM :: Failed to pre-process the job Error: Rotated drive was changed, restarting backup copy job
13/07/2021 12:00:39 AM :: Rotated drive was changed, restarting backup copy job
13/07/2021 12:00:40 AM :: Job finished with error at 13/07/2021 12:00:34 am

The backup copy seems to be running correct though.

Any ideas why this is happening?

[AlexLeadingEdge]

Sorry, just saw this thread stating it is a known bug:

veeam-backup-replication-f2/v11-and-rot ... 72491.html

[Natalia Lupacheva]

Hi Alex,

Thank you for working with forum search, let's keep similar issues together.
Moved your post to the existing thread.

Thanks!

[mengl]

We are having the same issue.

1 vote for a fix.

[ale.com]

Hi, there's any update on when the "Failed to pre-process the job Error: Rotated drive was changed, restarting backup copy job" bug is expected to be solved?

Maybe there's no many support ticket raised for this, as it's a "cosmetic" issue, but as someone already said, it's a "cosmetic" issue that involves almost everyone is using rotated drives

[Charles.yang]

Hi poor guys, just updating. As per suggestion/solution, I increased the numbers of restore points, which is the total points in all rotated medias. But Veeam created more restore points than the policy, and one backup copy job even created a new full backup point, now I have got 2 x full backups in a backup chain in the USB drive, funny! . I believe the restore points in Veeam are chaotic.

I gave up and changed to time based retention policy instead.

[Gostev]

Time-based retention is the way to go! It's also the default setting for any new jobs, as restore point based retention is simply not compatible with a number of new features.

Having said that, restore-point based retention is definitely not chaotic. I don't expect it to have too many bugs after 14 years since it existed. Everything you experienced will have an explanation in the debug logs, and will likely match your job settings or something else you did. Our support will be happy to review the debug logs and tell you exactly what happened.

[fpivot95]

Hi, so let's say I have a backup copy job that saves data to a RDX unit.

The previous configuration was:
1) backup copy job with retention policy -> 2 retention points
2) script to delete all data in the RDX unit before the job starts

But this doesn't work anymore as expected because it gives the error "Failed to pre-process the job Error: Rotated drive was changed, restarting backup copy job".

So I don't think I've fully understood how to create a working configuration for a backup job in Veeam 11.
1) backup copy job with retention policy -> how many days??? The customer has 5 cartridges that are changed every day (mon-fri) and I want to keep one point for each cartridge. How many days should I put in the job details?
2) not sure if I still need a script to clear the cartridges. Will Veeam find and remove old restore points automatically?

[Gostev]

What specifically does not work? It says the job is restarted so it should continue normally. They only complaint from OP was that this event is logged as an error (cosmetic issue). If they don't continue for you, then you should open a support case because we can't troubleshoot issues over forum posts.

A script was never needed because Veeam support rotated drive cleanup natively.

[fpivot95]

I probably didn't explain myself well. We also have that "cosmetic" issue (though copy jobs are working regularly). The problem is that we monitor Veeam Events in order to check backup status for all customers and we receive every day one "cosmetic" error for every customer that has a copy job.

According to one of your previously answers "Time-based retention is the way to go!" and I did for one customer (I just changed retention policy from 'retention points' to 'days').
From what I have understood this should make the cosmetic error disappear, right?

[Gostev]

No, this was said in regards to a totally different issue, as the topic got completely derailed. I should probably close it at this point, as this other discussion is causing a confusion with the original question...

I understand the cosmetic issue from the first post should be fixed in the immediate minor release (11a).
@Dima P. @Egor Yakovlev can you please confirm before I close?

[Egor Yakovlev]

Yes, it is fixed in 11a: messages related to the media switch are now "green".