Comprehensive data protection for all workloads
tsightler
VP, Product Management
Posts: 5293
Liked: 2146 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Serious Issue with 5.0.2 Restore Point Retention!!

Post by tsightler » Apr 05, 2011 10:31 pm

UPDATED BUILD AVAILABLE FOR DOWNLOAD STARTING APRIL 11 13:00 GMT
Product binaries have been replaced on our web-site with a newer 5.0.2 build (5.0.2.230) where the issue is resolved. “Deleted VMs retention period” feature works as expected in this newer build. Only data belonging to deleted VMs is removed from backup files once the set amount of days after the corresponding VM disappears from infrastructure passes. Backup files themselves are never affected by this data retention setting, and are only handled/deleted according to the main retention policy (number of restore points to keep).

ISSUE SUMMARY
Posted on April 6, 2010 14:49 GMT

SYMPTOMS
After upgrading to version 5.0.2, the next backup job run deletes all restore points older than 14 days.

CAUSE
New “Deleted VMs retention period” feature is behaving incorrectly, causing backup files older than specified amount of days to be deleted even if they contain VMs which had not been deleted from infrastructure.

WHAT IS AFFECTED?
This issue affects any backup jobs with retention policy for data on disk longer than default settings (which are: 14 restore points, daily backups). If you had previously changed default retention policy to large amount of restore points, “Deleted VMs retention period” will take priority and remove restore points older than 14 days, no matter of required restore point count settings.

WHAT IS A WORKAROUND?
Calculate the amount of days matching your restore point count retention policy. Then, open backup job settings and increase the new “Deleted VMs retention period” on “Backup Destination” step to the calculated value. For examples, for 90 restore points and daily backups, set “Deleted VMs retention period” to 90 days. For 100 restore points and backing up 2 times per day, set “Deleted VMs retention period” to 50 days.

RESOLUTION
If you implement the above workaround after upgrading to 5.0.2, but before backup your jobs run, you will avoid losing older restore points. Veeam is working on implementing correct behavior of the “Deleted VMs retention period”. We will update everyone when the new product build fixing the issue is available.

We are very sorry for the inconvenience this may have caused.

Original post starts below this line
=============================

We are experiencing a serious issue after the upgrade of Veeam 5.0.2. Most of our backups are configured as a forward incremental, with an transformation to reverse incremental each night. This allows the backup to complete in the quickest possible time, while the reverse incremental is processed on the secondary storage after the backup is done. These jobs typically have a fairly high number of restore points (typically 90, some 45). Because they are daily backups, that typically gave us 90 or 45 days worth of recovery.

Since the upgrade to 5.0.2, Veeam has removed all of our restore points and is only keep 14 days worth. The jobs still show the correct number of restore points, although there is now a new parameter to set the number of days before purging "deleted VM".

I will open a support case with Veeam, however, I wanted to post this as a warning to others that might also retain a large number of restore points for a large number of days. I would stay away from the 5.0.2 update for a while until this issue can be explained. So far this issue has hit 100% of our jobs that use reverse incrementals or forward incremental with transform. We have some jobs that use forward incrementals only, and run periodic fulls that, at least so far, don't appear to be impacted, although we don't for sure that they can't be.

Gostev
SVP, Product Management
Posts: 24017
Liked: 3254 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Serious Issue with 5.0.2 Restore Point Retention!!

Post by Gostev » Apr 05, 2011 11:30 pm

Thanks for heads up!

tsightler
VP, Product Management
Posts: 5293
Liked: 2146 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Serious Issue with 5.0.2 Restore Point Retention!!

Post by tsightler » Apr 06, 2011 12:17 am

I should point out that it could easily be something unique to our environment, but our jobs are not that special, other than the fact that we use Linux systems as backup targets so it could possibly be something affecting on the Linux agent.

Gostev
SVP, Product Management
Posts: 24017
Liked: 3254 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Serious Issue with 5.0.2 Restore Point Retention!!

Post by Gostev » Apr 06, 2011 12:25 am

I think there is a good chance this is a bug due to new autopurge functionality for deleted/removed VMs, as this is the only big change in retention aside of bug fix. Of course, I am not sure how this could have affected "main" retention that does actual backup file deletion. But if there is issue in code, I could see how this could have slipped pass QC (daily transform coupled with very long retention policy is not a common test case I guess). Although you say this happens on regular reversed incremental jobs too, anyway...

I will update root cause and scope once I hear investigation results from development. If you have the logs from jobs where this massive old restore points deletion happened, please forward them our way, as this may save quite some time trying to reproduce.

tsightler
VP, Product Management
Posts: 5293
Liked: 2146 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Serious Issue with 5.0.2 Restore Point Retention!!

Post by tsightler » Apr 06, 2011 1:13 am

I opened a support ticket but didn't receive an automated email with a ticket number like I normally do. I'll send you the logs.

We have 5 jobs which were affected by this, 3 jobs were forward incremental with daily transform and retention for 45 or 90 restore points, 1 job was forward incremental with weekly transform and retention for 90 restore points, and 1 job was just basic reverse incremental with 90 restore points. Since all of these jobs only ran once a day effectively restore points=days so this has effectively wiped out months of on disk backups (guess we're glad we have tape!!).

mpozar
Enthusiast
Posts: 30
Liked: 1 time
Joined: Jan 01, 2006 1:01 am
Contact:

Re: Serious Issue with 5.0.2 Restore Point Retention!!

Post by mpozar » Apr 06, 2011 5:03 am

We backup the .VBK and the .VRB onto tape and after the upgrade to 5.0.2 I found that I suddenly went from two tapes to one.

My Veeam Backup jobs in Veeam 5.0.1 were setup with Reverse Incremental with a 21 Retention Period and I run one backup per day which then provides me with 21 days of rollback.

Veeam 5.0.2 has a new option in the Backup Job that I did not originally notice after the update to 5.0.2 called Deleted VMS Retention Period and it DEFAULTS to 14 Days and the Restore Points to keep on disk in my environment is set to 21.

I checked the .VBK and .VRB files and I've found that Veeam has DELETED the .VRB files that are older than 14 days. OUCH...

I have NOT deleted any VMs in my environment for MONTHS so I do not understand why .VRBs that are older than 14 days are being deleted. Is this NORMAL (hope not) or a BUG..

Have FUN!

Michael

keatho
Novice
Posts: 6
Liked: never
Joined: Mar 31, 2011 11:14 pm
Full Name: Jye Keath
Contact:

Re: Serious Issue with 5.0.2 Restore Point Retention!!

Post by keatho » Apr 06, 2011 6:20 am

I am also seeing the same problem. All my jobs were set to 30 restore points and have now been cut back to 14.

rgj
Influencer
Posts: 15
Liked: 1 time
Joined: Oct 28, 2010 6:38 am
Full Name: René Jørgensen
Contact:

Re: Serious Issue with 5.0.2 Restore Point Retention!!

Post by rgj » Apr 06, 2011 7:58 am

I'm also at 14 restore points after upgrading to 5.0.2 instead of the configured 30.

Gostev
SVP, Product Management
Posts: 24017
Liked: 3254 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Serious Issue with 5.0.2 Restore Point Retention!!

Post by Gostev » Apr 06, 2011 10:41 am

For now, I suggest you set new parameter (retention in days for larger amount from default 14). Thanks.

tsightler
VP, Product Management
Posts: 5293
Liked: 2146 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Serious Issue with 5.0.2 Restore Point Retention!!

Post by tsightler » Apr 06, 2011 12:47 pm

Well, at least I know it's not just happening in my environment. Unfortunately it looks like a very serious bug managed to creep into 5.0.2 with the addition of this new feature.

Gostev
SVP, Product Management
Posts: 24017
Liked: 3254 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Serious Issue with 5.0.2 Restore Point Retention!!

Post by Gostev » Apr 06, 2011 1:45 pm

We have completed investigations, here is the summary I have prepared for our support. I will make this topic sticky, too... we are very sorry for the inconvenience this may have caused.

SYMPTOMS
After upgrading to version 5.0.2, the next backup job run deletes all restore points older than 14 days.

CAUSE
New “Deleted VMs retention period” feature is behaving incorrectly, causing backup files older than specified amount of days to be deleted even if they contain VMs which had not been deleted from infrastructure.

WHAT IS AFFECTED?
This issue affects any backup jobs with retention policy for data on disk longer than default settings (which are: 14 restore points, daily backups). If you had previously changed default retention policy to large amount of restore points, “Deleted VMs retention period” will take priority and remove restore points older than 14 days, no matter of required restore point count settings.

WHAT IS A WORKAROUND?
Calculate the amount of days matching your restore point count retention policy. Then, open backup job settings and increase the new “Deleted VMs retention period” on “Backup Destination” step to the calculated value. For examples, for 90 restore points and daily backups, set “Deleted VMs retention period” to 90 days. For 100 restore points and backing up 2 times per day, set “Deleted VMs retention period” to 50 days.

RESOLUTION
If you implement the above workaround after upgrading to 5.0.2, but before backup your jobs run, you will avoid losing older restore points. Veeam is working on implementing correct behavior of the “Deleted VMs retention period”. We will update everyone when the new product build fixing the issue is available.

PKramer
Novice
Posts: 8
Liked: never
Joined: Mar 29, 2010 11:58 am
Full Name: Paul Kramer
Contact:

Re: Serious Issue with 5.0.2 Restore Point Retention!!

Post by PKramer » Apr 06, 2011 2:04 pm

I was just hit with this and I am extremely disappointed in Veeam QC on this, I just lost 50 days of backup on disk due to this issue.

Do you have a Powershell script that we can use to change the retention period you stated in bulk?? Or a sql script maybe?

Veeam should have sent an email notice on this immediately notifying every customer to not run any backups on 5.0.2 if they had just upgraded.

I have 40+ jobs to edit and a script would be much easier.

Gostev
SVP, Product Management
Posts: 24017
Liked: 3254 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Serious Issue with 5.0.2 Restore Point Retention!!

Post by Gostev » Apr 06, 2011 2:29 pm

Well, I can just say that this appeared to be entirely NOT a QC team fault in this case, trust me on that one.
PowerShell or SQL script should be possible, I will ask development to create something and forward this on to our support.

tsightler
VP, Product Management
Posts: 5293
Liked: 2146 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Serious Issue with 5.0.2 Restore Point Retention!!

Post by tsightler » Apr 06, 2011 5:34 pm

Yep, this was a pretty big miss. I'd love to know how QC ducks out of that. Did you ship a product without QC's blessing? Was the code posted not the code QC tested? If it was any of those reasons that's even more concerning than a QC fail.

Gostev
SVP, Product Management
Posts: 24017
Liked: 3254 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Serious Issue with 5.0.2 Restore Point Retention!!

Post by Gostev » Apr 06, 2011 6:45 pm

QC tests according to feature specification that development provides. The issue here was that specification was not correct to start with. Long story, if someone needs to be made responsible for what happen, I would say this is mostly my fault here. Always be triple checking.

Locked

Who is online

Users browsing this forum: Baidu [Spider], Karinne, M.Escobar and 16 guests