Comprehensive data protection for all workloads
Locked
tsightler
VP, Product Management
Posts: 5328
Liked: 2175 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Serious Issue with 5.0.2 Restore Point Retention!!

Post by tsightler » Apr 05, 2011 10:31 pm

UPDATED BUILD AVAILABLE FOR DOWNLOAD STARTING APRIL 11 13:00 GMT
Product binaries have been replaced on our web-site with a newer 5.0.2 build (5.0.2.230) where the issue is resolved. “Deleted VMs retention period” feature works as expected in this newer build. Only data belonging to deleted VMs is removed from backup files once the set amount of days after the corresponding VM disappears from infrastructure passes. Backup files themselves are never affected by this data retention setting, and are only handled/deleted according to the main retention policy (number of restore points to keep).

ISSUE SUMMARY
Posted on April 6, 2010 14:49 GMT

SYMPTOMS
After upgrading to version 5.0.2, the next backup job run deletes all restore points older than 14 days.

CAUSE
New “Deleted VMs retention period” feature is behaving incorrectly, causing backup files older than specified amount of days to be deleted even if they contain VMs which had not been deleted from infrastructure.

WHAT IS AFFECTED?
This issue affects any backup jobs with retention policy for data on disk longer than default settings (which are: 14 restore points, daily backups). If you had previously changed default retention policy to large amount of restore points, “Deleted VMs retention period” will take priority and remove restore points older than 14 days, no matter of required restore point count settings.

WHAT IS A WORKAROUND?
Calculate the amount of days matching your restore point count retention policy. Then, open backup job settings and increase the new “Deleted VMs retention period” on “Backup Destination” step to the calculated value. For examples, for 90 restore points and daily backups, set “Deleted VMs retention period” to 90 days. For 100 restore points and backing up 2 times per day, set “Deleted VMs retention period” to 50 days.

RESOLUTION
If you implement the above workaround after upgrading to 5.0.2, but before backup your jobs run, you will avoid losing older restore points. Veeam is working on implementing correct behavior of the “Deleted VMs retention period”. We will update everyone when the new product build fixing the issue is available.

We are very sorry for the inconvenience this may have caused.

Original post starts below this line
=============================

We are experiencing a serious issue after the upgrade of Veeam 5.0.2. Most of our backups are configured as a forward incremental, with an transformation to reverse incremental each night. This allows the backup to complete in the quickest possible time, while the reverse incremental is processed on the secondary storage after the backup is done. These jobs typically have a fairly high number of restore points (typically 90, some 45). Because they are daily backups, that typically gave us 90 or 45 days worth of recovery.

Since the upgrade to 5.0.2, Veeam has removed all of our restore points and is only keep 14 days worth. The jobs still show the correct number of restore points, although there is now a new parameter to set the number of days before purging "deleted VM".

I will open a support case with Veeam, however, I wanted to post this as a warning to others that might also retain a large number of restore points for a large number of days. I would stay away from the 5.0.2 update for a while until this issue can be explained. So far this issue has hit 100% of our jobs that use reverse incrementals or forward incremental with transform. We have some jobs that use forward incrementals only, and run periodic fulls that, at least so far, don't appear to be impacted, although we don't for sure that they can't be.

Gostev
SVP, Product Management
Posts: 24174
Liked: 3301 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Serious Issue with 5.0.2 Restore Point Retention!!

Post by Gostev » Apr 05, 2011 11:30 pm

Thanks for heads up!

tsightler
VP, Product Management
Posts: 5328
Liked: 2175 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Serious Issue with 5.0.2 Restore Point Retention!!

Post by tsightler » Apr 06, 2011 12:17 am

I should point out that it could easily be something unique to our environment, but our jobs are not that special, other than the fact that we use Linux systems as backup targets so it could possibly be something affecting on the Linux agent.

Gostev
SVP, Product Management
Posts: 24174
Liked: 3301 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Serious Issue with 5.0.2 Restore Point Retention!!

Post by Gostev » Apr 06, 2011 12:25 am

I think there is a good chance this is a bug due to new autopurge functionality for deleted/removed VMs, as this is the only big change in retention aside of bug fix. Of course, I am not sure how this could have affected "main" retention that does actual backup file deletion. But if there is issue in code, I could see how this could have slipped pass QC (daily transform coupled with very long retention policy is not a common test case I guess). Although you say this happens on regular reversed incremental jobs too, anyway...

I will update root cause and scope once I hear investigation results from development. If you have the logs from jobs where this massive old restore points deletion happened, please forward them our way, as this may save quite some time trying to reproduce.

tsightler
VP, Product Management
Posts: 5328
Liked: 2175 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Serious Issue with 5.0.2 Restore Point Retention!!

Post by tsightler » Apr 06, 2011 1:13 am

I opened a support ticket but didn't receive an automated email with a ticket number like I normally do. I'll send you the logs.

We have 5 jobs which were affected by this, 3 jobs were forward incremental with daily transform and retention for 45 or 90 restore points, 1 job was forward incremental with weekly transform and retention for 90 restore points, and 1 job was just basic reverse incremental with 90 restore points. Since all of these jobs only ran once a day effectively restore points=days so this has effectively wiped out months of on disk backups (guess we're glad we have tape!!).

mpozar
Enthusiast
Posts: 30
Liked: 1 time
Joined: Jan 01, 2006 1:01 am
Contact:

Re: Serious Issue with 5.0.2 Restore Point Retention!!

Post by mpozar » Apr 06, 2011 5:03 am

We backup the .VBK and the .VRB onto tape and after the upgrade to 5.0.2 I found that I suddenly went from two tapes to one.

My Veeam Backup jobs in Veeam 5.0.1 were setup with Reverse Incremental with a 21 Retention Period and I run one backup per day which then provides me with 21 days of rollback.

Veeam 5.0.2 has a new option in the Backup Job that I did not originally notice after the update to 5.0.2 called Deleted VMS Retention Period and it DEFAULTS to 14 Days and the Restore Points to keep on disk in my environment is set to 21.

I checked the .VBK and .VRB files and I've found that Veeam has DELETED the .VRB files that are older than 14 days. OUCH...

I have NOT deleted any VMs in my environment for MONTHS so I do not understand why .VRBs that are older than 14 days are being deleted. Is this NORMAL (hope not) or a BUG..

Have FUN!

Michael

keatho
Novice
Posts: 6
Liked: never
Joined: Mar 31, 2011 11:14 pm
Full Name: Jye Keath
Contact:

Re: Serious Issue with 5.0.2 Restore Point Retention!!

Post by keatho » Apr 06, 2011 6:20 am

I am also seeing the same problem. All my jobs were set to 30 restore points and have now been cut back to 14.

rgj
Influencer
Posts: 15
Liked: 1 time
Joined: Oct 28, 2010 6:38 am
Full Name: René Jørgensen
Contact:

Re: Serious Issue with 5.0.2 Restore Point Retention!!

Post by rgj » Apr 06, 2011 7:58 am

I'm also at 14 restore points after upgrading to 5.0.2 instead of the configured 30.

Gostev
SVP, Product Management
Posts: 24174
Liked: 3301 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Serious Issue with 5.0.2 Restore Point Retention!!

Post by Gostev » Apr 06, 2011 10:41 am

For now, I suggest you set new parameter (retention in days for larger amount from default 14). Thanks.

tsightler
VP, Product Management
Posts: 5328
Liked: 2175 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Serious Issue with 5.0.2 Restore Point Retention!!

Post by tsightler » Apr 06, 2011 12:47 pm

Well, at least I know it's not just happening in my environment. Unfortunately it looks like a very serious bug managed to creep into 5.0.2 with the addition of this new feature.

Gostev
SVP, Product Management
Posts: 24174
Liked: 3301 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Serious Issue with 5.0.2 Restore Point Retention!!

Post by Gostev » Apr 06, 2011 1:45 pm

We have completed investigations, here is the summary I have prepared for our support. I will make this topic sticky, too... we are very sorry for the inconvenience this may have caused.

SYMPTOMS
After upgrading to version 5.0.2, the next backup job run deletes all restore points older than 14 days.

CAUSE
New “Deleted VMs retention period” feature is behaving incorrectly, causing backup files older than specified amount of days to be deleted even if they contain VMs which had not been deleted from infrastructure.

WHAT IS AFFECTED?
This issue affects any backup jobs with retention policy for data on disk longer than default settings (which are: 14 restore points, daily backups). If you had previously changed default retention policy to large amount of restore points, “Deleted VMs retention period” will take priority and remove restore points older than 14 days, no matter of required restore point count settings.

WHAT IS A WORKAROUND?
Calculate the amount of days matching your restore point count retention policy. Then, open backup job settings and increase the new “Deleted VMs retention period” on “Backup Destination” step to the calculated value. For examples, for 90 restore points and daily backups, set “Deleted VMs retention period” to 90 days. For 100 restore points and backing up 2 times per day, set “Deleted VMs retention period” to 50 days.

RESOLUTION
If you implement the above workaround after upgrading to 5.0.2, but before backup your jobs run, you will avoid losing older restore points. Veeam is working on implementing correct behavior of the “Deleted VMs retention period”. We will update everyone when the new product build fixing the issue is available.

PKramer
Novice
Posts: 8
Liked: never
Joined: Mar 29, 2010 11:58 am
Full Name: Paul Kramer
Contact:

Re: Serious Issue with 5.0.2 Restore Point Retention!!

Post by PKramer » Apr 06, 2011 2:04 pm

I was just hit with this and I am extremely disappointed in Veeam QC on this, I just lost 50 days of backup on disk due to this issue.

Do you have a Powershell script that we can use to change the retention period you stated in bulk?? Or a sql script maybe?

Veeam should have sent an email notice on this immediately notifying every customer to not run any backups on 5.0.2 if they had just upgraded.

I have 40+ jobs to edit and a script would be much easier.

Gostev
SVP, Product Management
Posts: 24174
Liked: 3301 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Serious Issue with 5.0.2 Restore Point Retention!!

Post by Gostev » Apr 06, 2011 2:29 pm

Well, I can just say that this appeared to be entirely NOT a QC team fault in this case, trust me on that one.
PowerShell or SQL script should be possible, I will ask development to create something and forward this on to our support.

tsightler
VP, Product Management
Posts: 5328
Liked: 2175 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Serious Issue with 5.0.2 Restore Point Retention!!

Post by tsightler » Apr 06, 2011 5:34 pm

Yep, this was a pretty big miss. I'd love to know how QC ducks out of that. Did you ship a product without QC's blessing? Was the code posted not the code QC tested? If it was any of those reasons that's even more concerning than a QC fail.

Gostev
SVP, Product Management
Posts: 24174
Liked: 3301 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Serious Issue with 5.0.2 Restore Point Retention!!

Post by Gostev » Apr 06, 2011 6:45 pm

QC tests according to feature specification that development provides. The issue here was that specification was not correct to start with. Long story, if someone needs to be made responsible for what happen, I would say this is mostly my fault here. Always be triple checking.

PKramer
Novice
Posts: 8
Liked: never
Joined: Mar 29, 2010 11:58 am
Full Name: Paul Kramer
Contact:

Re: Serious Issue with 5.0.2 Restore Point Retention!!

Post by PKramer » Apr 08, 2011 4:43 pm

Gostev wrote:Well, I can just say that this appeared to be entirely NOT a QC team fault in this case, trust me on that one.
PowerShell or SQL script should be possible, I will ask development to create something and forward this on to our support.
Any word on that script or powershell yet??

Gostev
SVP, Product Management
Posts: 24174
Liked: 3301 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Serious Issue with 5.0.2 Restore Point Retention!!

Post by Gostev » Apr 08, 2011 6:16 pm

Our support has the SQL script in hands since yesterday, have you been in contact with them?

Gostev
SVP, Product Management
Posts: 24174
Liked: 3301 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Serious Issue with 5.0.2 Restore Point Retention!!

Post by Gostev » Apr 11, 2011 12:06 pm

Product binaries have been replaced on our web-site with a newer 5.0.2 build (5.0.2.230) where the issue above is fixed. “Deleted VMs retention period” feature works as expected in this newer build. Only data belonging to deleted VMs is removed from backup files once the set amount of days after the corresponding VM disappears from infrastructure passes. Backup files themselves are never affected by this data retention setting, and are only handled/deleted according to the main retention policy (number of restore points to keep).

This fix is the only change in the code comparing to original RTM 5.0.2 build (5.0.2.224).

digitlman
Enthusiast
Posts: 94
Liked: 3 times
Joined: Jun 10, 2010 6:32 pm
Contact:

Re: Serious Issue with 5.0.2 Restore Point Retention!!

Post by digitlman » Apr 12, 2011 12:07 pm

I'm about to upgrade, and when I run the .EXE, it says it need to remove a previous install.

I'm probably being paranoid, but is this going to wipe out all my configs and jobs?

Vitaliy S.
Product Manager
Posts: 22527
Liked: 1475 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Serious Issue with 5.0.2 Restore Point Retention!!

Post by Vitaliy S. » Apr 12, 2011 12:11 pm

Don't worry, all you job configurations/other data will stay intact after the upgrade since it is stored in configuration database on SQL Server.

digitlman
Enthusiast
Posts: 94
Liked: 3 times
Joined: Jun 10, 2010 6:32 pm
Contact:

Re: Serious Issue with 5.0.2 Restore Point Retention!!

Post by digitlman » Apr 12, 2011 12:12 pm

Thought so, but I figured I'd double check, since I knew you guys would answer this question quickly (cuz you rock!).

VER
Influencer
Posts: 23
Liked: 4 times
Joined: Jan 16, 2011 10:24 am
Full Name: Wouter
Contact:

Re: Serious Issue with 5.0.2 Restore Point Retention!!

Post by VER » Apr 14, 2011 11:31 am

Why didn't I recieve an email from Veeam with a warning for this issue? Is it so difficult to send an email to all your B&R customers?

I lost more than 2 months of restore points due to this error. I am not amused. :(

Or should I arrange a backup of our backup without using Veeam?

tsightler
VP, Product Management
Posts: 5328
Liked: 2175 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Serious Issue with 5.0.2 Restore Point Retention!!

Post by tsightler » Apr 14, 2011 1:27 pm

Well, I don't have anything to do with Veeam as a company, but the honest truth is, if your are not backing up to some offline location (tape, cloud, rsync offsite, etc) or to an appliance that offers hard protections against deleted files (snapshots, archive, etc) then there is always some risk with backups to spinning media. Even a simple NTFS issue or storage failure could wipe out your backups if you don't have an alternative copy.

Gostev
SVP, Product Management
Posts: 24174
Liked: 3301 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Serious Issue with 5.0.2 Restore Point Retention!!

Post by Gostev » Apr 14, 2011 1:50 pm

VER wrote:Why didn't I recieve an email from Veeam with a warning for this issue? Is it so difficult to send an email to all your B&R customers?
If you already downloaded and upgraded, it is too late to tell you anyway - backups are already lost, right? Plus, it takes time to prepare and launch any email campaign - this is not as easy as just sending email to a colleague. We did not have time for anything like that.

Instead, download page was updated immediately with information about this issue (huge red box with huge red font on top of the download page). So, newly downloading users were warned.

Changed topic back to Standard.

jftuga
Enthusiast
Posts: 26
Liked: 2 times
Joined: Oct 22, 2010 7:02 pm
Full Name: John Taylor
Location: Athens, GA
Contact:

Re: Serious Issue with 5.0.2 Restore Point Retention!!

Post by jftuga » Apr 14, 2011 3:00 pm

Gostev,

I would like to suggest that you go ahead and have a system in place to launch an email campaign. Serious bugs are bound to happen in the future. If you prepare beforehand, it should'nt take much time to send out an email to your customers who have logged into your website since the bad version was released. Right now, you probably still have clients unknowingly using the bad version. These people are trusting your program to safe guard their data. Veeam needs to be proactive about this and attempt to inform their customers via email. I consider anything less to be negligent.

Gostev
SVP, Product Management
Posts: 24174
Liked: 3301 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Serious Issue with 5.0.2 Restore Point Retention!!

Post by Gostev » Apr 14, 2011 4:33 pm

jftuga wrote:Right now, you probably still have clients unknowingly using the bad version.
Sure, we absolutely realize that. This is why our support has been busy reaching out to each customer who downloaded the product directly to make sure they realize the issue, and to suggest the upgrade to a newer build. Unfortunately, email does not work well anyway for situations like this, as email open rates are very low according to our statistics, but we need to make sure we reached every affected customer with this information.

I agree it would not hurt if on top of that we could quickly send email to potentially affected customers. In fact, our Director of Support has just suggested to implement this exact procedure earlier today.

MB-NS
Enthusiast
Posts: 41
Liked: never
Joined: Apr 01, 2009 10:18 am
Contact:

Re: Serious Issue with 5.0.2 Restore Point Retention!!

Post by MB-NS » Apr 15, 2011 4:32 pm

VMware, Microsoft, SAN vendors, Veeam... whatever the size of the company, not a single one is spared by these issues of youth bugs.
The only way to circumvent it is to delay the upgrade long enough to correctly assess the new code stability (when it is possible, of course...).

twilhelm
Lurker
Posts: 1
Liked: never
Joined: Mar 14, 2010 4:33 pm
Full Name: Tyler Wilhelm
Contact:

Re: Serious Issue with 5.0.2 Restore Point Retention!!

Post by twilhelm » Apr 15, 2011 10:17 pm

The loss of the prior Restore Points is a most unfortunate occurrence. It would have been nice to have received an e-mail notification that may have prevented us from running the backup jobs or to notify us that prior restore points may no longer be available. Even today, an e-mail should go out to everyone who has downloaded that version of the product.

But if your company, organization or career is dependent on YOUR ability to recover your IT systems and data, then YOU should NEVER trust a single backup solution, no matter how good it is.

gavin.kennedy
Novice
Posts: 3
Liked: never
Joined: Nov 10, 2010 12:08 pm
Full Name: Gavin Kennedy
Contact:

Re: Serious Issue with 5.0.2 Restore Point Retention!!

Post by gavin.kennedy » Jun 07, 2011 4:34 am

Gostev wrote:If you already downloaded and upgraded, it is too late to tell you anyway - backups are already lost, right? Plus, it takes time to prepare and launch any email campaign - this is not as easy as just sending email to a colleague. We did not have time for anything like that.
We have been affected by this bug also. I'm very disappointed in the lack of communication to your customers about such a serious issue.

At no point have we received communication from Veeam that there was a serious bug in the .224 build. I accept your comment about it being too late once it's downloaded. However, lets say you upgrade at the end of March to a build that has a bug in it (you don't know it has a bug in it) and your retention is set to 120 days. On the day of the upgrade you lose all but the last 14 restore points. From that point on, you only have 14 restore points. You get to the beginning of June before realizing there's a bug and you only have 14 restore points. This means no backup on disk for the period of end of March till end of May. IF Veeam had communicated to you that there was a bug, let's say a week after the release, then you would only lose a week of restore points rather than 2 months of restore points.

I take full responsibility for my backup solution, which involves Veeam's software. Luckily, we send to tape and I'm OK to restore our restore points (messily) back into the system. This however, does not make the lack of communication to the customer acceptable. It is still poor service to not alert us to the fact.

Until we realize there is a problem, we do not check the website outside of our software maintenance windows.
Gostev wrote:Instead, download page was updated immediately with information about this issue (huge red box with huge red font on top of the download page). So, newly downloading users were warned.
Newly downloading users are not the ones affected once the binaries have been updated, it's the users who downloaded prior to that, so the big red text does nothing to help those users. Why would you upgrade the version and then go back to the site the next day to check again?

In addition to that, how about an email going out saying there's a new build available on the website so that your customers don't have to check in every day in case there's a bug. Every other major software company emails product update alerts.

I wanted to express my point of view here, as the support channels do not accommodate that.

Very disappointed.

Regards,
Gavin

Gostev
SVP, Product Management
Posts: 24174
Liked: 3301 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Serious Issue with 5.0.2 Restore Point Retention!!

Post by Gostev » Jun 07, 2011 1:41 pm

Hi Gavin,
gavin.kennedy wrote:In addition to that, how about an email going out saying there's a new build available on the website so that your customers don't have to check in every day in case there's a bug. Every other major software company emails product update alerts.
Reading through my response above, there was no email because our support was reaching out to each affected customer directly with this information. I am sorry if our support was not able to reach you. I am not involved in support at all, but I will forward this information onto our Director of Support, so that they double-check the procedure they used to determine the affected customers.
gavin.kennedy wrote:I wanted to express my point of view here, as the support channels do not accommodate that.
Support channels do accommodate this. In fact, this is purely customer support issue here, so support organization is the ultimate and only destination for your feedback above. While support does not monitor this forum, they do review every single feedback form completed (the form that you are offered each time when your support case is closed with any reason whatsoever). Moreover, please note that there is an option next to that form to receive a callback from support manager. And in many cases, it will be our Director of Support who will call you ;)

Thanks.

Locked

Who is online

Users browsing this forum: Baidu [Spider], Bing [Bot] and 35 guests