Is massive storage growth normal when upgrading to v12?

[FR8dawg]

We recently completed an upgrade to v12. Shortly after we saw rather significant increases in AWS storage and the associated cost. In March we experienced a 3x increase. It was followed by a futher 2.8x increase in April. That is a combined 8.45x increase in just two months when the underlying source data has not changed significantly.

Is this normal? Could it be transitory due to the upgrade -- especially writing to immutable storage? Have others experienced something similar? Did we miss something obvious in the upgrade process? Was there a small black control labeled in black on a black background with a little black light that lights up black when you flip it saying... "Please don't flip this switch again. Your storage may explode."

Eagar to see what others have experienced.

[Gostev]

No, it's not normal or expected. I recommend you have our support investigate your situation. Please share a support case ID when you open one. Thanks!

[FR8dawg]

Thank you. I will have our Veeam guy open a support case when he returns to the office on Monday.

[Drew Schlussel]

Please share the resolution in this thread. Very curious to see what's causing this issue for you... thanks!

[FR8dawg]

--- UPDATE ---
We do have an active Case #06051226 - Massive increase in AWS Storage.

No resolution yet, but Veeam is working diligently to try and track this down. Short version is that Veeam is attempting to delete from AWS and the delete is failing. That results in the massive storage growth. Even though we use immutable storage, the items being deleted are past the retention date.

We have confirmed that the problem began with the upgrade to v12.

[FR8dawg]

--- UPDATE ---
Still unresolved. Troubleshooting continues. Calls to delete certain blocks or files are failing. Doesn't seem to be a problem with AWS. Only started with the upgrade to v12.

Paying an EXTRA $6K/month. Need to get this fixed ASAP.

[Gostev]

Honestly, AWS S3 API call failing to work on certain objects does seem like a problem with AWS to me. Especially knowing there isn't really any wiggle room as it comes to issuing the S3 DELETE call.

Anyway, it should be easy to get to the bottom of this by asking AWS support to see what exactly has prevented the particular block from being deleted? Our support should be able to provide you specific object names for which delete operation has failed and time stamps of when the S3 DELETE commands were issued against those.

[pirx]

--- UPDATE ---
Still unresolved. Troubleshooting continues. Calls to delete certain blocks or files are failing. Doesn't seem to be a problem with AWS. Only started with the upgrade to v12.

Paying an EXTRA $6K/month. Need to get this fixed ASAP.

Do you see any errors?

We had issues with outdated data in capacity tier since beginning of using SOBR. 2 years ago we already deleted 50+TB that were long out of retention and object lock was also removed. Turned out that was a locking issue. We receive hotfixes but we still see the same result. I just had a case where 1,5 year old data cause index errors and found out that again a lot of backup chains were never cleaned up. Received again a hotfix for v11 to exclude chains with a regkey. In addition we have errors like "the XML you provided is not valid" during cleanup/deletion that - according to AWS - result in failing delete operations. After 4-5 months without solution I closed the case. We'll migrate all of our capacity tiers to copy-to-S3 jobs. Hoping that this will work better with more transparency.

As the indexes in V12 will be removed/migrated I can imagine that this might lead to strange results. That is also my biggest fear for upcoming v12 migration, that offloads/index-migration will fail because the data in S3 is kind of inconsistent.