Comprehensive data protection for all workloads
PKramer
Novice
Posts: 8
Liked: never
Joined: Mar 29, 2010 11:58 am
Full Name: Paul Kramer
Contact:

Re: Serious Issue with 5.0.2 Restore Point Retention!!

Post by PKramer » Apr 08, 2011 4:43 pm

Gostev wrote:Well, I can just say that this appeared to be entirely NOT a QC team fault in this case, trust me on that one.
PowerShell or SQL script should be possible, I will ask development to create something and forward this on to our support.
Any word on that script or powershell yet??

Gostev
SVP, Product Management
Posts: 24022
Liked: 3258 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Serious Issue with 5.0.2 Restore Point Retention!!

Post by Gostev » Apr 08, 2011 6:16 pm

Our support has the SQL script in hands since yesterday, have you been in contact with them?

Gostev
SVP, Product Management
Posts: 24022
Liked: 3258 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Serious Issue with 5.0.2 Restore Point Retention!!

Post by Gostev » Apr 11, 2011 12:06 pm

Product binaries have been replaced on our web-site with a newer 5.0.2 build (5.0.2.230) where the issue above is fixed. “Deleted VMs retention period” feature works as expected in this newer build. Only data belonging to deleted VMs is removed from backup files once the set amount of days after the corresponding VM disappears from infrastructure passes. Backup files themselves are never affected by this data retention setting, and are only handled/deleted according to the main retention policy (number of restore points to keep).

This fix is the only change in the code comparing to original RTM 5.0.2 build (5.0.2.224).

digitlman
Enthusiast
Posts: 94
Liked: 3 times
Joined: Jun 10, 2010 6:32 pm
Contact:

Re: Serious Issue with 5.0.2 Restore Point Retention!!

Post by digitlman » Apr 12, 2011 12:07 pm

I'm about to upgrade, and when I run the .EXE, it says it need to remove a previous install.

I'm probably being paranoid, but is this going to wipe out all my configs and jobs?

Vitaliy S.
Product Manager
Posts: 22311
Liked: 1415 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Serious Issue with 5.0.2 Restore Point Retention!!

Post by Vitaliy S. » Apr 12, 2011 12:11 pm

Don't worry, all you job configurations/other data will stay intact after the upgrade since it is stored in configuration database on SQL Server.

digitlman
Enthusiast
Posts: 94
Liked: 3 times
Joined: Jun 10, 2010 6:32 pm
Contact:

Re: Serious Issue with 5.0.2 Restore Point Retention!!

Post by digitlman » Apr 12, 2011 12:12 pm

Thought so, but I figured I'd double check, since I knew you guys would answer this question quickly (cuz you rock!).

VER
Influencer
Posts: 23
Liked: 4 times
Joined: Jan 16, 2011 10:24 am
Full Name: Wouter
Contact:

Re: Serious Issue with 5.0.2 Restore Point Retention!!

Post by VER » Apr 14, 2011 11:31 am

Why didn't I recieve an email from Veeam with a warning for this issue? Is it so difficult to send an email to all your B&R customers?

I lost more than 2 months of restore points due to this error. I am not amused. :(

Or should I arrange a backup of our backup without using Veeam?

tsightler
VP, Product Management
Posts: 5298
Liked: 2153 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Serious Issue with 5.0.2 Restore Point Retention!!

Post by tsightler » Apr 14, 2011 1:27 pm

Well, I don't have anything to do with Veeam as a company, but the honest truth is, if your are not backing up to some offline location (tape, cloud, rsync offsite, etc) or to an appliance that offers hard protections against deleted files (snapshots, archive, etc) then there is always some risk with backups to spinning media. Even a simple NTFS issue or storage failure could wipe out your backups if you don't have an alternative copy.

Gostev
SVP, Product Management
Posts: 24022
Liked: 3258 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Serious Issue with 5.0.2 Restore Point Retention!!

Post by Gostev » Apr 14, 2011 1:50 pm

VER wrote:Why didn't I recieve an email from Veeam with a warning for this issue? Is it so difficult to send an email to all your B&R customers?
If you already downloaded and upgraded, it is too late to tell you anyway - backups are already lost, right? Plus, it takes time to prepare and launch any email campaign - this is not as easy as just sending email to a colleague. We did not have time for anything like that.

Instead, download page was updated immediately with information about this issue (huge red box with huge red font on top of the download page). So, newly downloading users were warned.

Changed topic back to Standard.

jftuga
Enthusiast
Posts: 26
Liked: 2 times
Joined: Oct 22, 2010 7:02 pm
Full Name: John Taylor
Location: Athens, GA
Contact:

Re: Serious Issue with 5.0.2 Restore Point Retention!!

Post by jftuga » Apr 14, 2011 3:00 pm

Gostev,

I would like to suggest that you go ahead and have a system in place to launch an email campaign. Serious bugs are bound to happen in the future. If you prepare beforehand, it should'nt take much time to send out an email to your customers who have logged into your website since the bad version was released. Right now, you probably still have clients unknowingly using the bad version. These people are trusting your program to safe guard their data. Veeam needs to be proactive about this and attempt to inform their customers via email. I consider anything less to be negligent.

Gostev
SVP, Product Management
Posts: 24022
Liked: 3258 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Serious Issue with 5.0.2 Restore Point Retention!!

Post by Gostev » Apr 14, 2011 4:33 pm

jftuga wrote:Right now, you probably still have clients unknowingly using the bad version.
Sure, we absolutely realize that. This is why our support has been busy reaching out to each customer who downloaded the product directly to make sure they realize the issue, and to suggest the upgrade to a newer build. Unfortunately, email does not work well anyway for situations like this, as email open rates are very low according to our statistics, but we need to make sure we reached every affected customer with this information.

I agree it would not hurt if on top of that we could quickly send email to potentially affected customers. In fact, our Director of Support has just suggested to implement this exact procedure earlier today.

MB-NS
Enthusiast
Posts: 41
Liked: never
Joined: Apr 01, 2009 10:18 am
Contact:

Re: Serious Issue with 5.0.2 Restore Point Retention!!

Post by MB-NS » Apr 15, 2011 4:32 pm

VMware, Microsoft, SAN vendors, Veeam... whatever the size of the company, not a single one is spared by these issues of youth bugs.
The only way to circumvent it is to delay the upgrade long enough to correctly assess the new code stability (when it is possible, of course...).

twilhelm
Lurker
Posts: 1
Liked: never
Joined: Mar 14, 2010 4:33 pm
Full Name: Tyler Wilhelm
Contact:

Re: Serious Issue with 5.0.2 Restore Point Retention!!

Post by twilhelm » Apr 15, 2011 10:17 pm

The loss of the prior Restore Points is a most unfortunate occurrence. It would have been nice to have received an e-mail notification that may have prevented us from running the backup jobs or to notify us that prior restore points may no longer be available. Even today, an e-mail should go out to everyone who has downloaded that version of the product.

But if your company, organization or career is dependent on YOUR ability to recover your IT systems and data, then YOU should NEVER trust a single backup solution, no matter how good it is.

gavin.kennedy
Novice
Posts: 3
Liked: never
Joined: Nov 10, 2010 12:08 pm
Full Name: Gavin Kennedy
Contact:

Re: Serious Issue with 5.0.2 Restore Point Retention!!

Post by gavin.kennedy » Jun 07, 2011 4:34 am

Gostev wrote:If you already downloaded and upgraded, it is too late to tell you anyway - backups are already lost, right? Plus, it takes time to prepare and launch any email campaign - this is not as easy as just sending email to a colleague. We did not have time for anything like that.
We have been affected by this bug also. I'm very disappointed in the lack of communication to your customers about such a serious issue.

At no point have we received communication from Veeam that there was a serious bug in the .224 build. I accept your comment about it being too late once it's downloaded. However, lets say you upgrade at the end of March to a build that has a bug in it (you don't know it has a bug in it) and your retention is set to 120 days. On the day of the upgrade you lose all but the last 14 restore points. From that point on, you only have 14 restore points. You get to the beginning of June before realizing there's a bug and you only have 14 restore points. This means no backup on disk for the period of end of March till end of May. IF Veeam had communicated to you that there was a bug, let's say a week after the release, then you would only lose a week of restore points rather than 2 months of restore points.

I take full responsibility for my backup solution, which involves Veeam's software. Luckily, we send to tape and I'm OK to restore our restore points (messily) back into the system. This however, does not make the lack of communication to the customer acceptable. It is still poor service to not alert us to the fact.

Until we realize there is a problem, we do not check the website outside of our software maintenance windows.
Gostev wrote:Instead, download page was updated immediately with information about this issue (huge red box with huge red font on top of the download page). So, newly downloading users were warned.
Newly downloading users are not the ones affected once the binaries have been updated, it's the users who downloaded prior to that, so the big red text does nothing to help those users. Why would you upgrade the version and then go back to the site the next day to check again?

In addition to that, how about an email going out saying there's a new build available on the website so that your customers don't have to check in every day in case there's a bug. Every other major software company emails product update alerts.

I wanted to express my point of view here, as the support channels do not accommodate that.

Very disappointed.

Regards,
Gavin

Gostev
SVP, Product Management
Posts: 24022
Liked: 3258 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Serious Issue with 5.0.2 Restore Point Retention!!

Post by Gostev » Jun 07, 2011 1:41 pm

Hi Gavin,
gavin.kennedy wrote:In addition to that, how about an email going out saying there's a new build available on the website so that your customers don't have to check in every day in case there's a bug. Every other major software company emails product update alerts.
Reading through my response above, there was no email because our support was reaching out to each affected customer directly with this information. I am sorry if our support was not able to reach you. I am not involved in support at all, but I will forward this information onto our Director of Support, so that they double-check the procedure they used to determine the affected customers.
gavin.kennedy wrote:I wanted to express my point of view here, as the support channels do not accommodate that.
Support channels do accommodate this. In fact, this is purely customer support issue here, so support organization is the ultimate and only destination for your feedback above. While support does not monitor this forum, they do review every single feedback form completed (the form that you are offered each time when your support case is closed with any reason whatsoever). Moreover, please note that there is an option next to that form to receive a callback from support manager. And in many cases, it will be our Director of Support who will call you ;)

Thanks.

Locked

Who is online

Users browsing this forum: evilaedmin, Karinne, kubimike and 55 guests