Error: S3 error: No space left on device.

[HansMeiser]

Hello,

i do some tests with an S3 Repo on a NetApp.
In Veeam i have a scale-out repo which is based on a 10T performance and a 20T capacity backup repo.

I created a job which stores some VMs daily to this S3 Sobr. this was running fore some weeks without problems.
Since a few days i get an errormessage, but only 3 of the VMs are affected. May job fails with: Error: S3 error: No space left on device.
This is suprising, I can see that performance repo is used 3,5T and capacity 325G, so there should be enough space left. And why should the repo have no empty space only for 3 VMs, others are stored the normal way.
In between the standard job "scale-out S3 offload" also fails, but this was fixed by a rescan of the Sobr.
I deleted these 3 VMs from backup, this also fails with message "no space left on device", but its not visiblke any more.
My backupjob still fails and iam running out of ideas now.

Please give me a hint, what should i check next?

Thanks,
Hans

[Mildur]

Hi Hans,

Unfortunately, this isn’t something we can investigate through a forum post.
Please comply with our forum rules and discuss technical issues, especially environmental ones with Customer Support first before bringing them to the forum.
After you’ve created the case, please share the case number with us here.

S3 error: No space left on device.

I assume it might be related to free disk space on volume with the temp folder on the gateway server — the one handling offloads from the Performance Tier to the Capacity Tier.
Our Support team can confirm the exact root cause by reviewing the log files.

Thank you,
Fabian

[HansMeiser]

Hello,

thanks for your help. Creating a ticket is much effort in this case. We have to contact our service contractor who is trying to solve it or is creating a case on your page. for testing purposes to much effort at this point.
I tried to find a hint in the logs, but yet unsuccesful.
If i do not find a solution within next 2 weeks, i will go official way.

Thanks,
Hans

[HansMeiser]

Hello,

we did some tests and i think we figured out a little bit more.
The problem is related to the 2 buckets building our SOBR and "automatic bucket creation enabled". On NetApp we saw the buckets we created and additional some buckets with generic names. all of them in size of 800G and half of them were written full to 100%.
We extended all of them to 1 T and restartet our job which was running succesful now.
So by "automatic bucket creation enabled" Veeam is allowed to create additional buckets, default number is 10. but these buckets are not cleaned in any way and prevent a successful backup even if the main buckets have much free space.
Hmm, what do you think? Is this a configuration problem on our side?

Thanks,
Hans

[HansMeiser]

Hello,

i ask again for answer in this case.
Currently i dont understand this feature of automatic bucket creation.
Veeam is creating succesful a bunch of buckets (ends with -aux) and writes data. now it happend again, some of the autocreated buckets are full and my backup job stopped workinhg with "no space left on device". this happens even if my mainbucket for actually data is almost empty and other autocreated buckets are available. I dont understand what Veeam is doing or expect to do by S3 Server.

Thanks,
Hans

[Mildur]

Hi Hans,

Have you opened a case with our customer support team as requested?
Since this is not a support forum, we can’t troubleshoot this issue here.
This is well explained in our forum policy.

Thanks,
Fabian

[HansMeiser]

Hello,

yes, we did. our Service Contractor opened a ticket and resonded a few times already, but no solution by now. no detalied look, only basic answers.
I just wanted to know what veeam is expecting from S3 side. B default the *-aux buckets not autogrow, number is not limited.

Thanks,
Hans

[Mildur]

Hi Hans,

I still don’t see your case number. If you prefer not to share it publicly, please send it to me via a private message here in the forum.

Currently i dont understand this feature of automatic bucket creation.

The Multi-Bucket feature is explained here.
The first bucket you connect to acts as the master bucket and contains metadata about the child buckets that store the backups.
The number of child buckets created depends on the setting “Maximum number of workloads per bucket.”

E.g. --> With the setting "10", we will create a new child bucket after each 10 protected machines (virtual, physical) or other support workloads.


There’s no “autogrow” functionality in Veeam — we simply use what the object storage provides.
Maybe you’ve limited the usable bucket size in the repository options?


Best regards,
Fabian

[HansMeiser]

Hello Fabian,

yesterday after reading your hint about limiting object storage consumption i had a good feeling and thought we have a hit. but unfortunately no change. may be i should describe alle the steps to create buckets and jobs.

- create a performance bucket in netapp with 10T
- create a capacity bucket in netapp with 20T
- added the to buckets as backup repos in veeam. "automatic bucket creation" is on with default value of 10 for number of workloads and also we set the soft limit by 10T respectively 20 T for the buckets.
- created a scale out backup repo consisting of the two backup repos.
- created a testjob which stores a bunch of VMs, target is the s§ SOBR.

The job ran for a few days without problems. at first only a few VMs showed the errormessage "no space left...", after a few runs all of the VMs could not backed any more.
At this point we saw that new -aux buckets in size of 800G were created and some of them were completely filled.

Yesterday i removed the limit for object storage consumption and deleted all accrued backups. I can see the main buckets performance and capacity are empty. currently 2 -aux buckets are existing and one of them is filled up. so my testjob fails for 7 of 10 VMs, only 3 are working and i expect to see them fail in a few days.
It seems Veeam maneuvered itself in a insoluble situation.

So the main problem are filled up -aux buckets and i dont know to avoid it. should not happen...

Thanks,
Hans

[Mildur]

Hi Hans,

After reading the comments in your case and doing some research myself, I believe the issue may lie on the object storage side rather than in Veeam.

From the NetApp screenshot in the case, it looks like the -aux bucket is limited to 800 GB or 1 TB.
Veeam can only use the capacity provided by the object storage, which explains why your first -aux bucket fills up so quickly to 100%. I assume you have protected VMs over 1TB?

The master bucket for the Performance Tier is configured with 10 TB limitation on the NetApp, but we only store metadata in these — not backups.
The backups themselves are stored in the -aux buckets, which currently have the very low bucket size (800GB/1TB).

Is there perhaps a default policy on the NetApp that creates buckets with a size limitation?
If so, please disable that policy or ensure it allows creating child buckets large enough to accommodate your backups.
The backup server won't set any size limitation during bucket creation on the Target Object Storage.

For the existing buckets, you can use NetApp CLI (or UI) to increase the bucket size of the -aux buckets:
https://docs.netapp.com/us-en/ontap/s3- ... -task.html

Code: Select all

vserver object-store-server bucket modify -vserver <SVM_name> -bucket <bucket_name> -size {<integer>[KB|MB|GB|TB|PB]}

Best regards,
Fabian

[HansMeiser]

Hello,

we had already a session with our responsible person for netapp. he could not find a policy or base configuration for S3 bucket size. Also there is no autogrow just like volumes have. the idea of this feature seems to be to have a more broad S3 directory structure. but iam surprised that this is a recommend fearure. I think there is a whole bunch of negative circumstances, which may result in failures. In my case i even cant repair the situation without manual manipulating the bucket size.
I conclude to remove this SOBR and its basic S3 repos and recreate them without auto-bucket-creation. In this case the data should be stored directly in S3 repos performance and capacity repos and we should have no self inflicted space problems.
In my opinion, the functionality of this feature is hanging by a thread. i can not recommend using it.

Thanks,
Hans

[Mildur]

Hi Hans,

The “multiple child buckets” feature is recommended for some S3 object storage appliances, as it can improve the performance of S3 operations on these buckets. It has nothing todo with the "No space left on device" error you are facing in your environment.

I recommend checking with NetApp for their own recommendations and best practices when using their storage with Veeam products — most vendors provide such documentation to their customers.
NetApp has one for their StorageGRID product, but I couldn’t find one for NetApp ONTAP.

I conclude to remove this SOBR and its basic S3 repos and recreate them without auto-bucket-creation. In this case the data should be stored directly in S3 repos performance and capacity repos and we should have no self inflicted space problems.

Feel free to disable this feature when creating your repository. But I suggest to first reach out to NetApp to discuss Best Practices in regards to their and our product.

Best regards,
Fabian

[MatzeB]

Hey Hans, i just saw this topic.
In general i didn't use multi bucket creation in the past, with Ontap S3 if you don't plan to go to 100-200TB i think one bucket is enough. The reason behind this is because each bucket is "backed" by a Flexgroup.
I blogged something about how to create S3 buckets in Ontap for Veeam, the only difference is maybe that you don't need Object Lock.
https://tecblog.au.de/ontap-9-14-1p1-ve ... ject-lock/

Regarding your error and because each Bucket is a FlexGroup i think here is maybe your problem.
You can see the FlexGroup volume and all it's constituent volumes with this commands:

Code: Select all

volume show -vserver vs1 -volume-style-extended flexgroup
volume show -vserver vs1 -volume-style-extended flexgroup-constituent

https://docs.netapp.com/us-en/ontap/fle ... -task.html

My assumption is that maybe one or more of your constituent volumes are full. This effect is typical if you see "no space errors" on some files, but not on all. So please review the commands above.

Matze