Error: S3 error: No space left on device.

[HansMeiser]

Hello,

i do some tests with an S3 Repo on a NetApp.
In Veeam i have a scale-out repo which is based on a 10T performance and a 20T capacity backup repo.

I created a job which stores some VMs daily to this S3 Sobr. this was running fore some weeks without problems.
Since a few days i get an errormessage, but only 3 of the VMs are affected. May job fails with: Error: S3 error: No space left on device.
This is suprising, I can see that performance repo is used 3,5T and capacity 325G, so there should be enough space left. And why should the repo have no empty space only for 3 VMs, others are stored the normal way.
In between the standard job "scale-out S3 offload" also fails, but this was fixed by a rescan of the Sobr.
I deleted these 3 VMs from backup, this also fails with message "no space left on device", but its not visiblke any more.
My backupjob still fails and iam running out of ideas now.

Please give me a hint, what should i check next?

Thanks,
Hans

[Mildur]

Hi Hans,

Unfortunately, this isn’t something we can investigate through a forum post.
Please comply with our forum rules and discuss technical issues, especially environmental ones with Customer Support first before bringing them to the forum.
After you’ve created the case, please share the case number with us here.

S3 error: No space left on device.

I assume it might be related to free disk space on volume with the temp folder on the gateway server — the one handling offloads from the Performance Tier to the Capacity Tier.
Our Support team can confirm the exact root cause by reviewing the log files.

Thank you,
Fabian

[HansMeiser]

Hello,

thanks for your help. Creating a ticket is much effort in this case. We have to contact our service contractor who is trying to solve it or is creating a case on your page. for testing purposes to much effort at this point.
I tried to find a hint in the logs, but yet unsuccesful.
If i do not find a solution within next 2 weeks, i will go official way.

Thanks,
Hans

[HansMeiser]

Hello,

we did some tests and i think we figured out a little bit more.
The problem is related to the 2 buckets building our SOBR and "automatic bucket creation enabled". On NetApp we saw the buckets we created and additional some buckets with generic names. all of them in size of 800G and half of them were written full to 100%.
We extended all of them to 1 T and restartet our job which was running succesful now.
So by "automatic bucket creation enabled" Veeam is allowed to create additional buckets, default number is 10. but these buckets are not cleaned in any way and prevent a successful backup even if the main buckets have much free space.
Hmm, what do you think? Is this a configuration problem on our side?

Thanks,
Hans

[HansMeiser]

Hello,

i ask again for answer in this case.
Currently i dont understand this feature of automatic bucket creation.
Veeam is creating succesful a bunch of buckets (ends with -aux) and writes data. now it happend again, some of the autocreated buckets are full and my backup job stopped workinhg with "no space left on device". this happens even if my mainbucket for actually data is almost empty and other autocreated buckets are available. I dont understand what Veeam is doing or expect to do by S3 Server.

Thanks,
Hans

[Mildur]

Hi Hans,

Have you opened a case with our customer support team as requested?
Since this is not a support forum, we can’t troubleshoot this issue here.
This is well explained in our forum policy.

Thanks,
Fabian

[HansMeiser]

Hello,

yes, we did. our Service Contractor opened a ticket and resonded a few times already, but no solution by now. no detalied look, only basic answers.
I just wanted to know what veeam is expecting from S3 side. B default the *-aux buckets not autogrow, number is not limited.

Thanks,
Hans

[Mildur]

Hi Hans,

I still don’t see your case number. If you prefer not to share it publicly, please send it to me via a private message here in the forum.

Currently i dont understand this feature of automatic bucket creation.

The Multi-Bucket feature is explained here.
The first bucket you connect to acts as the master bucket and contains metadata about the child buckets that store the backups.
The number of child buckets created depends on the setting “Maximum number of workloads per bucket.”

E.g. --> With the setting "10", we will create a new child bucket after each 10 protected machines (virtual, physical) or other support workloads.


There’s no “autogrow” functionality in Veeam — we simply use what the object storage provides.
Maybe you’ve limited the usable bucket size in the repository options?


Best regards,
Fabian