Hotfix for the retention policy miscalculation issue (version 4c)

[Polina]

Hi All,

Today we released a hotfix KB4099 for a recently discovered issue making Veeam Backup for Microsoft Office 365 potentially vulnerable to backup data loss.

This hotfix is applicable to Veeam Backup for Microsoft Office 365 version 4c (builds 4.0.1.531, 4.0.1.545, 4.0.1.612). You can check your current build number under the Help and Support > About in the VBO console.

You are strongly encouraged to install the patch KB4099 as soon as possible to avoid the potential for data loss. 

[Mildur]

Thanks, will there be some scripts or automatically possibilities to check for missing data?
We have 6-7 customers on my backup service. This is not much, but I can think of service providers with many dozens of configured backup jobs.
How can they be sure about their backups

[Gostev]

I will let Polina confirm, but as far as I know it would be impossible to miss the fact that you ran into the bug, because it triggers full backup (due to no data left in the repository) which is hard to miss given how long those take due to Microsoft Office 365 throttling.

[Polina]

Anton is right, the first thing to pay attention to is the unplanned full sync of Exchange data.
And we're working on the option (script likely) that will allow detecting it automatically. Stay tuned

[Regnor]

A full sync wouldn't cause a warning/error, or would it? If so, then you wouldn't notice the difference if you don't check the logs daily.
And in what time frame could this issue occur?

[Mike Resseler]

Regnor,

A full will take much longer but indeed will not give an error or warning. What do you mean by timeframe?

[Regnor]

Between v3 and v5a are almost two years; so could this issue happen anytime (and multiple times) during this two years or only recently?

[Mike Resseler]

Regnor,

This could have indeed happened anytime between those two years. But to be honest, if it was not noticed I doubt it happened. Surely there will have been some restores at a certain point in time

[Regnor]

That's the question. I'm sure that for smaller environments the full sync won't be noticed.
How would on see the issue when doing a restore? Would the restore point be empty or only missing deleted data before the full sync occured?

I just can't get a feeling how critical this issue is...

[Gostev]

If I understood it correctly, in an unlikely chance you run into it, you will lose all your mailbox backups. So please, don't take chances and install the update instead

[Mike Resseler]

Gostev is correct. If you are unlucky to run in it, then you won't find the backups for those mailboxes. So a restore would become a problem.

[karsten123]

may you please provide proper information on that! Your answers are a joke, right?

[Mildur]

As bad the answer may seem, what else can Anton say. It‘s the direct answer for what to expect if the bug has hit your backup server. There is no „nicer“ way to say the obvious.

They have found the error and have provided a solution for ongoing backups as fast as possible.
Bugs exists everywhere. Some are small, some are more dangerous like this one.

We are doing the 3-2-1 rule with Office 365 backups too. If something is happening to my first copy, i have several other copies of the backup data at a second location. That‘s why I'm not very worried now.

The only bad thing I‘m concerned here is, if someone is using object storage as a target. There is no 3-2-1 rule possible with Veeam at the moment in this scenario.

We will go to object storage as a o365 target this year. Hopefully, there is some 3-2-1 rule (snapshot like) with object storage to have a copy of the bucket at a second site with multiple versions.

[Gostev]

Agree, periodic dump to cold object storage tiers like Glacier Deep Archive would be really nice to have.

[Mike Resseler]

We are indeed looking into something for object storage

[A.Rogers]

Anton is right, the first thing to pay attention to is the unplanned full sync of Exchange data.
And we're working on the option (script likely) that will allow detecting it automatically. Stay tuned

Hi, Has there been a script released to check if a tenant has run into this error?

[Mike Resseler]

Hi Alan,

First: Welcome to the internal forums
Second: No, we don't have that specific script, the only people that can check this would be our support engineers, or by seeing the unplanned full sync of Exchange data.

Did you already upgrade to 4c + hotfix or even later?