Veeam O365 upgrade to v7 and Wasabi Repo issue.

[zxzxzxzxzx]

We use Veeam backup for Office 365 and recently upgraded to v7 (From version 5b). We extend our repo to Wasabi and use local disk for meta data/ persistent cache.

Every once in a while we would visit the wasabi web console and poke around our bucket. Before we upgraded Veeam we noticed a few days where we couldn’t explore the bucket, then it would be fine. The error said something like "internal error please try again." Backups always worked so we never thought much about it.

We had some issues after the Veeam upgrade and got Veeam to help. We decided to remove the repo connection and try to re add the repo. Now we can’t make the connection because we are getting the Wasabi internal error.

I had a call with Wasabi this morning and they noticed over 51million markers for deletion. We tried to calculate the exact amount with AWS CLI and a python script to help delete the markers. We also get the timeout error while trying to do this. Wondered if anyone else had issues browsing a bucket? opened a ticket with wasabi and his initial thought was it was too large of a bucket and the LIST command was timing out. We are around 25tb in this bucket.

Veeam is saying that they don't mark anything for deletion and Wasabi wants to know if its ok to Delete these markers and will it mess up the chain.

I am not sure of the exact issue from the Veeam upgrade but now we are stuck. We thought it might have to do with how the original connection to the bucket was made. Orignally there was no Wasabi storage radio button during Object storage setup, and now there it.

We noticed that retention was never kicking in and out 8tb O365 environment turned into 30tb in Wasabi because nothing was being deleted.

We also have a VMware snapshot on the VBO server, but hesitant to go back to it, incase the wasabi repo actually did update.

We are approaching close to a week on this problem and usually only get 1 update a day from either side.

Veeam Support - Case # 06036846
Wasabi case # 87193

[chrisWasabi]

Hello,

I looked at your case here, and I am unsure I have an answer aside from deleting the versions and seeing what happens. But this is my best guess at what is going on now.

In previous versions of VBO365(<7), neither versioning nor object lock were supported configurations. This particular bucket had versioning enabled on it. The way that versioning works is that it keeps every version of an object unless those versions are explicitly deleted. Since VBO365 didn't support versioning, it did not know it was enabled and just kept backing up and deleting items while the versions were stored. This led to nothing being deleted and DeleteMarkers of objects being added, racking up to 51mil DeleteMarkers + the object count of non-deleted objects.

I am unsure whether this bucket will continue to work after upgrading to V7 since it originally used an unsupported configuration, and I think Veeam now checks for this unsupported configuration and will not let you add it. In either case, you can probably eliminate all versions with DeleteMarkers, and the bucket would continue to work again on V5 or V6.

I believe that is now in progress.

[zxzxzxzxzx]

Thank you! What are your thoughts on rolling back our veeam upgrade to what is was last week? To see if we can still access the data and then worry about the possible upgrade to 7?

[chrisWasabi]

It seems like a viable option, I will let Veeam chime on it, though.