Immutable backups – but one deleted file makes everything useless?

[SaschaRTG152]

Hello

Support Case: 07675579.

We came across a critical limitation while testing Veeam Backup for Microsoft 365 with immutable object storage.
We have not yet tested whether this also affects buckets used by Veeam Backup & Replication, but we assume it does.

In our test setup, we manually deleted a bucket used by Veeam.
The goal was to simulate a situation where an attacker gains administrative rights and then tries to delete or encrypt files, like in a ransomware attack.

What happened:
- The backup data was still present (protected by versioning and immutability)
- Veeam Software marked the repository as Invalid
- It is no longer possible to restore data via the Explorer
- Re-attaching the repository fails also

What Veeam support told us:
- The configuration/metadata files from Veeam are not protected by immutability
- If they are lost, the repository becomes unusable
- There is no way to restore access without help from Support or R&D
- This behavior is “by design” and considered unsupported

Why this is a major concern:
Even though the backup data is technically still there, it becomes inaccessible to the customer.
This creates a single point of failure — not in the data, but in a few unprotected files.

In a real-world scenario, this would mean:
- Delays in recovery during a ransomware attack (because we have to wait for a fix from the R&D)
- Dependency on support for disaster recovery (Veeam Support also mentioned the Storage Provider has to help first!?)
- Immutability that does not actually guarantee accessibility

We believe Veeam should urgently improve this behavior!
What is your view on this?

Best regards,
Sascha

[Mildur]

Hi Sascha

Welcome to the forum.

Let me first say, your data is not lost. Metadata is also immutable.
Your immutable backup data (+metadata) is not lost after such an attack.

You can reconnect to the bucket by removing two specific deletion markers in the bucket (tested in my lab). By removing deletion markers, the immutable object will be „active“ again and our product can read it.

Your system engineer at Veeam has reported this situation to us already and I‘m in contact with QA about the official procedure to get back access to the data on the bucket.

I will contact your support engineer after I have an official answer from QA. And I will also update this topic.

Best,
Fabian

[vmtech123]

Great support and happy ending to a ticket! Nice work Fabian!

[SaschaRTG152]

Hi Fabian,

thanks a lot – that actually sounds really good!
If recovery is possible just by removing two deletion markers, then three support engineers unfortunately gave me the wrong information.
I also tried to remove all deletion markers during my test, but I still couldn’t re-add the repository.
It would be great if this could be fixed automatically in the future – maybe with a built-in repair option.
Looking forward to the official solution!

Best regards,
Sascha

[Mildur]

Hi Sascha,

Good news, I got an update from our QA Team. I also sent a notification to the support engineer in your case.

First, QA confirmed to me that configuration/metadata files are always immutable and protected.

Secondly, we can safely remove the delete markers from the RepositoryConfig.*******. After doing so, you will be able to reconnect to your repository. We have logged a bug for this issue—our goal is to allow Veeam Backup for Microsoft 365 to read such "deleted repository configuration files" out of the box in the future.

Can you please let me know what kind of object storage you are using and if you observe any unexpected behavior after reconnecting the bucket?

You can find the location of the configuration files under this path in your bucket:

Code: Select all

<bucket>/Veeam/Backup365/<bucketfolder>/CriticalDataBackup/CommonInfo/RepositoryConfig.*******

Best,
Fabian