V11 and Rotated Backup Copy Error Email

[agravanea]

im having the same issue on two diff client systems right now. Windows updates appeared to have installed an update for veeam on one and it succeeded afterward, but the other is still failing. v 11 as well, case 04842426

[Dima P.]

Hello Aaron,

Windows updates appeared to have installed an update for veeam on one and it succeeded afterward, but the other is still failing

Neither backup copy nor Veeam B&R update/patch distribution engine relies on Windows Updates logic. Can you please elaborate? Thank you!

[m.novelli]

the issue you are facing is related to the 'unification' of rotated drive retention logic - now it works the same way across all product. Say we have two USB drives A and B
and job retention is set to 14 restore points, driver A has 7 restore points and B has 8 restore points:

I'm not sure if this is good or bad for RDX drives :-/

Marco

[Charles.yang]

Neither backup copy nor Veeam B&R update/patch distribution engine relies on Windows Updates logic. Can you please elaborate? Thank you!

Agree. In theory, B&R checks the backup files and configuration in file level only, Windows updates should not affect it.

Now B&R is killing our backup copies, why doesn't Veeam provide a urgent patch for this issue?

[Dima P.]

Charles.yang,

Can you please check your backup copy job retention? If the backup files on newly connected rotated drive are expired backup job should remove those prior starting new backup chain. Thanks!

[Charles.yang]

My backup copy job retention is 10 restore points, not days. No USB drive has more than 5 points.

[Dima P.]

And the second drive? Retention is calculated based on restore points across all the drives you use for this job.

[Charles.yang]

Retention is often on a backup ( copy ) job setting, all USB drives belong to a repository as rotated media, which is a target of the backup copy job.

[Greg O]

Same issue for me with v11 also. Case #02270263

I confirm, with v10, I had Backup Copy Job (not a normal backup) with a series of rotated drives Mon-Thurs loaded on the respective day each week.
And another set of drives loaded on the respective Friday of each month.
The restore points on each backup were retained, until it reached the specified retention number, then the oldest was merged into the vbk file. The total number of restore points across all disks is not at all relevant to the number of restore points specified in the job.

Now with v11, all of the backups are deleted and a new vbk file is created every time the job is run. Not only are we losing all of the history on each disk, it also means I am backing up 4Tb every night.

In the support link above https://helpcenter.veeam.com/docs/backu ... ml?ver=110, it says
'Backup Copy Jobs

Backup copy jobs are performed in the following way:

Veeam Backup & Replication creates a regular backup chain on the currently attached drive.
When you swap drives, and the attached drive is empty, Veeam Backup & Replication creates a full backup on it. If there is a backup chain on the drive, Veeam Backup & Replication creates a new incremental backup and adds it to the backup chain. The latest incremental backup existing in the backup chain is used as a starting point for the new incremental backup.'

[YouAllHaveZoidberg]

the issue you are facing is related to the 'unification' of rotated drive retention logic - now it works the same way across all product

This is intentional? Can you please reverse this? I don't see how this is an enhancement. It's easier tracking individual drive restore points.

[Dima P.]

Folks,

Thanks for the feedback, we will discuss it with the team! Mind me asking if anyone here is using rotated media with primary jobs or it's just for the backup copy?

[Greg O]

I would have stayed on v10, if I knew this change had been made to rotated drives. Certainly a backward step, or at least give the user the option to retain the restore points vs create a new vbk file every time.

[m.novelli]

Thanks for the feedback, we will discuss it with the team! Mind me asking if anyone here is using rotated media with primary jobs or it's just for the backup copy?

I use RDX drives for Backup Copy Jobs for about 10 - 15 customersNever used for Primary Jobs

I would have stayed on v10, if I knew this change had been made to rotated drives. Certainly a backward step, or at least give the user the option to retain the restore points vs create a new vbk file every time.

Agree, where I've updated Veeam to v11 I've started to have many issues with rotated media. Please revert back to v10 behavior
Thanks!
Marco

[Charles.yang]

Thanks for the feedback, we will discuss it with the team! Mind me asking if anyone here is using rotated media with primary jobs or it's just for the backup copy?

I believe nobody uses rotated media in a primary job, most in backup copy job.
BTW, as it seems this issue can't be fixed soon, is there a way to revert back to V10 from V11?

[Dima P.]

Right now the quickest workaround seems to double the retention on your backup copy job. As for the general behavior in the product - we will discuss it with the team. Still wonder why retention across all drives is ok in primary job (was like that for ages), but not for backup copy, so please keep sharing your feedback!

[YouAllHaveZoidberg]

We use rotated drives for backup copy only.

In the ticket I have open, I asked what happens if a drive misses a rotation or two and becomes full. The answer was that it won't be able to merge and therefore be unable to backup. This should help you guys understand why we're upset. Keeping the restore points even across all drives (by having them be counted individually) is incredibly easy to manage. In order to account for missed rotations now, I have to reduce the total number of restore points. Some people will respond by saying I'll only want the most recent restore point anyway, so what's it matter? To that I say, how do you know? Some companies keep data going back for years.

Thanks for communicating with us on this.

[m.novelli]

That's my same issue, I had to reduce the number of copies for many customers due to rotated drive full since v11

[Dima P.]

The answer was that it won't be able to merge and therefore be unable to backup. This should help you guys understand why we're upset. Keeping the restore points even across all drives (by having them be counted individually) is incredibly easy to manage. In order to account for missed rotations now, I have to reduce the total number of restore points. Some people will respond by saying I'll only want the most recent restore point anyway, so what's it matter?

Can I have you case ID please? This does not sound right - the transform operation must be performed on the connected drive as soon as the backup chain reaches the retention (i.e. retention check across all drives is performed only when you connect the new drive). Thanks!

[Charles.yang]

Apparently, V11 can't identify the backup chain in rotated media, always treat as a new media. This is a critical bug. Very suprised why development team hasn't found and fixed the bug from Mar. If can't be fixed soon, have to consider other products.

[Dima P.]

Reviewing this issue with QA team, stay tuned.

[Greg O]

Right now the quickest workaround seems to double the retention on your backup copy job. As for the general behavior in the product - we will discuss it with the team. Still wonder why retention across all drives is ok in primary job (was like that for ages), but not for backup copy, so please keep sharing your feedback!

I don't understand this comment of doubling the retention on the backup copy job.
Following the support article, that the retention points of the job job should equal total retention points across all drives - then the above statement would not work because the retention number would be the number of points for each drive multiplied by the number of drives ? Or have I misunderstood the comment ?
For example, I had a backup copy job with 6 retention points on each drive using a rotation of 8 drives (one for each of Monday-Thursday, and one for each Friday on the month). This method worked fine under v10.
So now I have changed the job to have 48 retention points. So far, appears ok. The original vbk file is not being overwritten, but don't know if the merge is going to work when the time comes.

[Dima P.]

Greg,

Following the support article, that the retention points of the job job should equal total retention points across all drives

That's correct.

then the above statement would not work because the retention number would be the number of points for each drive multiplied by the number of drives ? Or have I misunderstood the comment ?

For the folks who got used to the per-drive logic with backup copy job and got two drives that seems to be the workaround.

I had a backup copy job with 6 retention points on each drive using a rotation of 8 drives (one for each of Monday-Thursday, and one for each Friday on the month). This method worked fine under v10.

With v11 whenever you connect new drive, Veeam B&R will perform a retention check at the start of your backup copy job. If the backup chain is exceeded and retention can be applied existing backup files will be removed from the newly connected drive and only then new full backup will be created.

So now I have changed the job to have 48 retention points. So far, appears ok. The original vbk file is not being overwritten, but don't know if the merge is going to work when the time comes.

8 drives with 6 restore points each, yup that looks right.

[Greg O]

Before with v10, on the individual drive when the retention point reached the specified value, it would merge the oldest vib into the vbk.
So, are you saying with v11 that won't happen ?

[Dima P.]

Works the same. If the drive was not disconnected but retention was reached we will start merge process.

[Gostev]

Please note that Backup Copy jobs with GFS setting enabled don't do merges in V11 in principle.

[Charles.yang]

Increased the restore point number in the retention policy, it seems the existing backup chain was not deleted, an incremental backup was created in the rotating media then.

In V10, the restore point is per device, I assume V11 calculates the restore points in all the rotating media.

[Charles.yang]

Please note that Backup Copy jobs with GFS setting enabled don't do merges in V11 in principle.

If that's the case, it will be very hard to control the usage of a rotated media.

[je1000]

this was a huge change that I don’t think was thought out properly.
There is many logistical and common sense reasons for rotated media to have restore points should be per drive, I will list a few.

Drives break or get lost, that’s the unfortunate reality for anyone using rotated media.

Rotated media If often used as long term archiving not to be overwritten, like monthly or yearly backup.

This will cause way to many regenerations of full backup jobs, since veeam has a global restore point tracking. this will dramatically slow down the already slow usb jobs.

Optimal USB backup will be like Microsoft Windows built in backup handles the USB drives, but absent of that v10 implementation was quite good but v11 broke it.

[Greg O]

Works the same. If the drive was not disconnected but retention was reached we will start merge process.

Since the retention is now calculated across all rotated drives, how can find I find out how many retention points are out there ?
Would like to know when the maximum value is about to be reached and hence when the merge process will occur.
Again, with v10 and the restore points calculated per drive - this was never an issue.

[Gostev]

In V10, the restore point is per device, I assume V11 calculates the restore points in all the rotating media.

That is correct. Retention policy is a property of the job and it does not matter where the particular backup file is stored physically. If it fell out of the retention policy, it will be deleted when it becomes reachable.

As far as I understood, V10 behavior is a bug that was fixed because customers complained backups are not deleted according to their retention policy, overfilling their rotated drives.

Honestly, I don't see much logic when people complain about backup files which are no longer under retention policy being deleted by Veeam. If they are deleted too early, then just correct your retention policy? Not sure what am I missing here.