Maintain control of your Microsoft Office 365 data
Post Reply
Polina
Veeam Software
Posts: 1726
Liked: 349 times
Joined: Oct 21, 2011 11:22 am
Full Name: Polina Vasileva
Contact:

Hotfix for the retention policy miscalculation issue (version 4c)

Post by Polina » 2 people like this post

Hi All,

Today we released a hotfix KB4099 for a recently discovered issue making Veeam Backup for Microsoft Office 365 potentially vulnerable to backup data loss.

This hotfix is applicable to Veeam Backup for Microsoft Office 365 version 4c (builds 4.0.1.531, 4.0.1.545, 4.0.1.612). You can check your current build number under the Help and Support > About in the VBO console.

You are strongly encouraged to install the patch KB4099 as soon as possible to avoid the potential for data loss. 

Mildur
Veeam Legend
Posts: 674
Liked: 306 times
Joined: May 13, 2017 4:51 pm
Full Name: Fabian Kessler
Contact:

Re: Hotfix for the retention policy miscalculation issue (version 4c)

Post by Mildur »

Thanks, will there be some scripts or automatically possibilities to check for missing data?
We have 6-7 customers on my backup service. This is not much, but I can think of service providers with many dozens of configured backup jobs.
How can they be sure about their backups :)

Gostev
SVP, Product Management
Posts: 28160
Liked: 4971 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Hotfix for the retention policy miscalculation issue (version 4c)

Post by Gostev » 1 person likes this post

I will let Polina confirm, but as far as I know it would be impossible to miss the fact that you ran into the bug, because it triggers full backup (due to no data left in the repository) which is hard to miss given how long those take due to Microsoft Office 365 throttling.

Polina
Veeam Software
Posts: 1726
Liked: 349 times
Joined: Oct 21, 2011 11:22 am
Full Name: Polina Vasileva
Contact:

Re: Hotfix for the retention policy miscalculation issue (version 4c)

Post by Polina » 1 person likes this post

Anton is right, the first thing to pay attention to is the unplanned full sync of Exchange data.
And we're working on the option (script likely) that will allow detecting it automatically. Stay tuned

Regnor
Veeam Legend
Posts: 438
Liked: 98 times
Joined: Jan 31, 2011 11:17 am
Full Name: Max
Contact:

Re: Hotfix for the retention policy miscalculation issue (version 4c)

Post by Regnor »

A full sync wouldn't cause a warning/error, or would it? If so, then you wouldn't notice the difference if you don't check the logs daily.
And in what time frame could this issue occur?

Mike Resseler
Product Manager
Posts: 6816
Liked: 900 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: Hotfix for the retention policy miscalculation issue (version 4c)

Post by Mike Resseler »

Regnor,

A full will take much longer but indeed will not give an error or warning. What do you mean by timeframe?

Regnor
Veeam Legend
Posts: 438
Liked: 98 times
Joined: Jan 31, 2011 11:17 am
Full Name: Max
Contact:

Re: Hotfix for the retention policy miscalculation issue (version 4c)

Post by Regnor »

Between v3 and v5a are almost two years; so could this issue happen anytime (and multiple times) during this two years or only recently?

Mike Resseler
Product Manager
Posts: 6816
Liked: 900 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: Hotfix for the retention policy miscalculation issue (version 4c)

Post by Mike Resseler »

Regnor,

This could have indeed happened anytime between those two years. But to be honest, if it was not noticed I doubt it happened. Surely there will have been some restores at a certain point in time

Regnor
Veeam Legend
Posts: 438
Liked: 98 times
Joined: Jan 31, 2011 11:17 am
Full Name: Max
Contact:

Re: Hotfix for the retention policy miscalculation issue (version 4c)

Post by Regnor »

That's the question. I'm sure that for smaller environments the full sync won't be noticed.
How would on see the issue when doing a restore? Would the restore point be empty or only missing deleted data before the full sync occured?

I just can't get a feeling how critical this issue is...

Gostev
SVP, Product Management
Posts: 28160
Liked: 4971 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Hotfix for the retention policy miscalculation issue (version 4c)

Post by Gostev »

If I understood it correctly, in an unlikely chance you run into it, you will lose all your mailbox backups. So please, don't take chances and install the update instead ;)

Mike Resseler
Product Manager
Posts: 6816
Liked: 900 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: Hotfix for the retention policy miscalculation issue (version 4c)

Post by Mike Resseler »

Gostev is correct. If you are unlucky to run in it, then you won't find the backups for those mailboxes. So a restore would become a problem.

karsten123
Service Provider
Posts: 54
Liked: 12 times
Joined: Apr 03, 2019 6:53 am
Full Name: Karsten Meja
Contact:

Re: Hotfix for the retention policy miscalculation issue (version 4c)

Post by karsten123 »

may you please provide proper information on that! Your answers are a joke, right?

Mildur
Veeam Legend
Posts: 674
Liked: 306 times
Joined: May 13, 2017 4:51 pm
Full Name: Fabian Kessler
Contact:

Re: Hotfix for the retention policy miscalculation issue (version 4c)

Post by Mildur » 2 people like this post

As bad the answer may seem, what else can Anton say. It‘s the direct answer for what to expect if the bug has hit your backup server. There is no „nicer“ way to say the obvious.

They have found the error and have provided a solution for ongoing backups as fast as possible.
Bugs exists everywhere. Some are small, some are more dangerous like this one.

We are doing the 3-2-1 rule with Office 365 backups too. If something is happening to my first copy, i have several other copies of the backup data at a second location. That‘s why I'm not very worried now.

The only bad thing I‘m concerned here is, if someone is using object storage as a target. There is no 3-2-1 rule possible with Veeam at the moment in this scenario.

We will go to object storage as a o365 target this year. Hopefully, there is some 3-2-1 rule (snapshot like) with object storage to have a copy of the bucket at a second site with multiple versions.

Gostev
SVP, Product Management
Posts: 28160
Liked: 4971 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Hotfix for the retention policy miscalculation issue (version 4c)

Post by Gostev » 1 person likes this post

Agree, periodic dump to cold object storage tiers like Glacier Deep Archive would be really nice to have.

Mike Resseler
Product Manager
Posts: 6816
Liked: 900 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: Hotfix for the retention policy miscalculation issue (version 4c)

Post by Mike Resseler » 1 person likes this post

We are indeed looking into something for object storage

A.Rogers
Lurker
Posts: 1
Liked: never
Joined: Jan 08, 2021 5:05 pm
Full Name: Alan Rogers
Contact:

Re: Hotfix for the retention policy miscalculation issue (version 4c)

Post by A.Rogers »

Polina wrote: Feb 17, 2021 7:42 am Anton is right, the first thing to pay attention to is the unplanned full sync of Exchange data.
And we're working on the option (script likely) that will allow detecting it automatically. Stay tuned
Hi, Has there been a script released to check if a tenant has run into this error?

Mike Resseler
Product Manager
Posts: 6816
Liked: 900 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: Hotfix for the retention policy miscalculation issue (version 4c)

Post by Mike Resseler »

Hi Alan,

First: Welcome to the internal forums
Second: No, we don't have that specific script, the only people that can check this would be our support engineers, or by seeing the unplanned full sync of Exchange data.

Did you already upgrade to 4c + hotfix or even later?

Post Reply

Who is online

Users browsing this forum: b.gahr and 2 guests