-
- Novice
- Posts: 7
- Liked: 2 times
- Joined: Apr 07, 2021 7:08 pm
- Full Name: Dave Purscell
- Contact:
Is massive storage growth normal when upgrading to v12?
We recently completed an upgrade to v12. Shortly after we saw rather significant increases in AWS storage and the associated cost. In March we experienced a 3x increase. It was followed by a futher 2.8x increase in April. That is a combined 8.45x increase in just two months when the underlying source data has not changed significantly.
Is this normal? Could it be transitory due to the upgrade -- especially writing to immutable storage? Have others experienced something similar? Did we miss something obvious in the upgrade process? Was there a small black control labeled in black on a black background with a little black light that lights up black when you flip it saying... "Please don't flip this switch again. Your storage may explode."
Eagar to see what others have experienced.
Is this normal? Could it be transitory due to the upgrade -- especially writing to immutable storage? Have others experienced something similar? Did we miss something obvious in the upgrade process? Was there a small black control labeled in black on a black background with a little black light that lights up black when you flip it saying... "Please don't flip this switch again. Your storage may explode."
Eagar to see what others have experienced.
Dave Purscell
IT Director
Steele County (MN)
IT Director
Steele County (MN)
-
- Chief Product Officer
- Posts: 31621
- Liked: 7110 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: Is massive storage growth normal when upgrading to v12?
No, it's not normal or expected. I recommend you have our support investigate your situation. Please share a support case ID when you open one. Thanks!
-
- Novice
- Posts: 7
- Liked: 2 times
- Joined: Apr 07, 2021 7:08 pm
- Full Name: Dave Purscell
- Contact:
Re: Is massive storage growth normal when upgrading to v12?
Thank you. I will have our Veeam guy open a support case when he returns to the office on Monday.
Dave Purscell
IT Director
Steele County (MN)
IT Director
Steele County (MN)
-
- Novice
- Posts: 3
- Liked: 2 times
- Joined: May 05, 2021 12:52 am
- Full Name: Drew Schlussel
- Contact:
Re: Is massive storage growth normal when upgrading to v12?
Please share the resolution in this thread. Very curious to see what's causing this issue for you... thanks!
-
- Novice
- Posts: 7
- Liked: 2 times
- Joined: Apr 07, 2021 7:08 pm
- Full Name: Dave Purscell
- Contact:
Re: Is massive storage growth normal when upgrading to v12?
--- UPDATE ---
We do have an active Case #06051226 - Massive increase in AWS Storage.
No resolution yet, but Veeam is working diligently to try and track this down. Short version is that Veeam is attempting to delete from AWS and the delete is failing. That results in the massive storage growth. Even though we use immutable storage, the items being deleted are past the retention date.
We have confirmed that the problem began with the upgrade to v12.
We do have an active Case #06051226 - Massive increase in AWS Storage.
No resolution yet, but Veeam is working diligently to try and track this down. Short version is that Veeam is attempting to delete from AWS and the delete is failing. That results in the massive storage growth. Even though we use immutable storage, the items being deleted are past the retention date.
We have confirmed that the problem began with the upgrade to v12.
Dave Purscell
IT Director
Steele County (MN)
IT Director
Steele County (MN)
-
- Novice
- Posts: 7
- Liked: 2 times
- Joined: Apr 07, 2021 7:08 pm
- Full Name: Dave Purscell
- Contact:
Re: Is massive storage growth normal when upgrading to v12?
--- UPDATE ---
Still unresolved. Troubleshooting continues. Calls to delete certain blocks or files are failing. Doesn't seem to be a problem with AWS. Only started with the upgrade to v12.
Paying an EXTRA $6K/month. Need to get this fixed ASAP.
Still unresolved. Troubleshooting continues. Calls to delete certain blocks or files are failing. Doesn't seem to be a problem with AWS. Only started with the upgrade to v12.
Paying an EXTRA $6K/month. Need to get this fixed ASAP.
Dave Purscell
IT Director
Steele County (MN)
IT Director
Steele County (MN)
-
- Chief Product Officer
- Posts: 31621
- Liked: 7110 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: Is massive storage growth normal when upgrading to v12?
Honestly, AWS S3 API call failing to work on certain objects does seem like a problem with AWS to me. Especially knowing there isn't really any wiggle room as it comes to issuing the S3 DELETE call.
Anyway, it should be easy to get to the bottom of this by asking AWS support to see what exactly has prevented the particular block from being deleted? Our support should be able to provide you specific object names for which delete operation has failed and time stamps of when the S3 DELETE commands were issued against those.
Anyway, it should be easy to get to the bottom of this by asking AWS support to see what exactly has prevented the particular block from being deleted? Our support should be able to provide you specific object names for which delete operation has failed and time stamps of when the S3 DELETE commands were issued against those.
-
- Veteran
- Posts: 598
- Liked: 87 times
- Joined: Dec 20, 2015 6:24 pm
- Contact:
Re: Is massive storage growth normal when upgrading to v12?
Do you see any errors?
We had issues with outdated data in capacity tier since beginning of using SOBR. 2 years ago we already deleted 50+TB that were long out of retention and object lock was also removed. Turned out that was a locking issue. We receive hotfixes but we still see the same result. I just had a case where 1,5 year old data cause index errors and found out that again a lot of backup chains were never cleaned up. Received again a hotfix for v11 to exclude chains with a regkey. In addition we have errors like "the XML you provided is not valid" during cleanup/deletion that - according to AWS - result in failing delete operations. After 4-5 months without solution I closed the case. We'll migrate all of our capacity tiers to copy-to-S3 jobs. Hoping that this will work better with more transparency.
As the indexes in V12 will be removed/migrated I can imagine that this might lead to strange results. That is also my biggest fear for upcoming v12 migration, that offloads/index-migration will fail because the data in S3 is kind of inconsistent.
Who is online
Users browsing this forum: Baidu [Spider], Bing [Bot] and 69 guests