Storage consumption of Veeam O365 on Object storage

[sumeet]

Hello all,

Our setup is VBO v5d using NetApp Storage Grid S3.

We are observing discrepancy in storage usage reported from Veeam object storage repository to what is reported on Storage grid.
If I add up the used space for all the object storage repository as listed in Veeam and match that storage with what is reported as used in Storage grid for the specific tenant used size, it is observed that Veeam is reporting approximately 10% less.
We have checked three of our tenancies that are using Object storage for this usage miss-match and all of them match.

Example for our own tenancy, Veeam reports Used space for object storage repo as 2 TB, and if I check in storage grid, the space used is 2.23 TB
Some of our clients have 100+ TB M365 storage, and so in this case this difference is of approximately 10 TB

I'm using VBO API ObjectStorageRepositories to loop through all the repo while adding up usedSpaceBytes / (1024 * 1024 * 1024)

Has anyone seen a mismatch when using Object storage?
Is this known or expected?

Thanks,
-Sumeet.

[HannesK]

Hello,
what did Netapp say about that? I remember that object storage has some internal overhead.

Best regards,
Hannes

[sumeet]

Hello Hannes,

I have not yet checked with NetApp, to be honest I do not expect much help from them. I'm assuming that they will say that this usage size at tenant level is what the application has stored.
Having said this, I will go ahead and open a case with NetApp and hope that I'm wrong.

From what I know storage grid maintains the meta data separately and is not part of the tenant.

It will be good to know from R&D on how Veeam calculates the used in case of object storage.
Since the usage is reported at repository level, which can be a folder in a bucket at SG level, I’m assuming Veeam does not get the usage from SG. And I have created multiple repository in a single bucket, so each repo is on its own folder. And I don't think Storage Grid reports used size at the folder level.
So is Veeam adding up each batch of small-files-total-size before sending it to SG and using this to calculate the used size?

Thanks,
-Sumeet.

[Mike Resseler]

Sumeet,

We are getting the information by requesting data from the storage... So if there is a discrepancy, I fear it is not on our side.

[sumeet]

Hello Mike,

Thanks for the confirmation.
Will you be able to get the API or CLI that is used for getting the used space from the storage?

I used S3 explorer to look at few of the folders (folders on which Veeam Object repository is created) properties, it takes sometime but then S3 explorer eventually reports the total size of the folder.
What I have observed is that the size reported in S3 explorer is two decimal places, whereas Veeam reports only one decimal place.
I checked for two repositories and Veeam in the UI reports used space as 1.3 and 3.8 TB. But in the S3 explorer the size is reported as 1.35 and 3.84 TB

But the total size I calculate is by using the API /v5/ObjectStorageRepositories and then dividing the usedspace by (1024 * 1024 * 1024).
So this calculation should avoid any decimal point inconsistency.

I have informed this to NetApp and waiting for them to check.

Thanks,
-Sumeet.

[sumeet]

Hello all,

NetApp has informed that there is a known issue in the version we have installed, which causes the used space to be incorrect. There is a HF provided, and hopefully after that this gets resolved.
I will update this thread, on how this goes.

Thanks,
-Sumeet.

[Mike Resseler]

Great stuff Sumeet. Thanks for this, looking forward to the update

[sumeet]

Hello all,

Working with NetApp, the HF was applied and then they ran few scripts to update the usage capacity.
But none of this made a change.

Finally they have informed that it has to be KibiByte and KiloBytes conversion that is causing the confusion.

According to NetApp, S3 explorer and Veeam are using Mibibyte, where as StorageGrid is using Megabyte.

1 Mibibyte = 1024 (bytes) * 1024 (KibiByte) i.e., 1,048,576 bytes.

1 Megabyte = 1000 (bytes) * 1000 (KiloBytes) i.e., 1,000,000 bytes.

If I use this conversion, then the capacity values match.

Would someone in Veeam be able to confirm that Veeam uses Kibibyte ?

Thanks,
-Sumeet.

[Mike Resseler]

Sumeet, working on it. I believe we do but I asked for explicit confirmation just to be sure.