Dell ECS not longer Working after 11a build 11.0.1.1261 P20211211

[bdunington]

Good Afternoon Veeam Team,

We did the upgraded to P20211211 from P20210525 and we now receive "bucket name contains invalid characters" when modifying the bucket. The name of bucket is "veeam_q9" The ECS supports "_" in the name and your https://www.veeam.com/kb3095 only says no to "." . My question is did the bucket logic change in the latest build of Veeam?

We would like to get to V11a but this my be a show stopper for us


Veeam Ticket #05250433

[HannesK]

Hello,
That was a bugfix on Veeam side: accepting _ (underscore) character was wrong according to the S3 API specification for bucket naming and may cause unforeseen consequences with other S3 API functions or in device-specific S3 Server implementations, including potential data loss even. We'd rather not have our customers take these risks and follow the specification from the vendor behind S3 API closely as far as accepted values go.

Bucket names can consist only of lowercase letters, numbers, dots (.), and hyphens (-).

Best regards,
Hannes

[tkeith]

When using Scale Out Backup Repositories (SOBR) you have the options to add your typical "buckets" or "Blobs" from AWS, Google, Azure, etc... we use the "Other" where its an "S3 compatible" bucket.... on our local Dell ECS.

Some new logic in v11a puts naming checks on bucket names and after upgrading to 11a we can no longer manage/edit/configure our SOBR due to "bucket contains invalid characters".
Ended up rolling back to v11 where the issue doesn't exist until this can be solved.

After further investigation it seems our bucket name has an "_". Not discussing how or why an "_" was used but this character is an acceptable character according to Dell on ECS buckets. We do know these are not compatible on AWS S3, Azure Blobs, etc but on the ECS its an acceptable character.

Renaming buckets is not a possibility and with a PB of data in this bucket, migrating the data to a new bucket/name is also an issue. We have opened a ticket to see why this logic was added to 11a but more specifically why this was added to the "S3 Compatible" selection when clearly there are solutions where these characters are accepted. I believe these checks are valid when selecting Amazon, Google, and Azure based on their naming standards. Also realized that each of these even have their own different naming standards that do differ from each other. So curious as to where did these naming validation checks in veeam come from and are they specific for which bucket type you select or did they set this generically to all these categories.

Is it possible to remove these specific validation checks when setting up under the generic "S3 compatible" group, or be a "warning" instead of a hard stop failure, as they do have products where the naming is not "invalid" when certain characters are used.

This can be found under case # 05271472

Thank you,
Keith

[Gostev]

Hello, I merged your post into the existing discussion about this. Please see the answer above.
I'll investigated if we can provide a registry value to configure these restrictions. Thanks!

[HannesK]

Hello,
in case that there is no way to disable the check, there is a tool called S3P built for a scenario like yours. For 1PB it would take 48h if the speed is the same as the use-case they had with Amazon.

Best regards,
Hannes

[tkeith]

Hopefully there is a way to hotfix/modify as Dell has options for migrating buckets but there is no "fastcopy" type scenario and not using AWS S3. But still feel we shouldn't be forced to rename our buckets when they have supported ECS naming just veeam's validation doesn't like it....

[Gostev]

Yes, we decided to make a hotfix that will add a reg key to allow unsupported symbols. But in reviewing this situation I also found that this change was not "random" but caused by support cases where restricted symbols in bucket names caused actual issues. Thanks!

[tkeith]

Still seems random... as I don't understand why is it Veeam's problem if customers used invalid bucket names in their systems? They should be following bucket naming rules and in our case we did and our name is supported by our product so this should not be Veeam's problem what I named my bucket... no? Unless these restricted symbols were supported by their products and veeam was the one having issues with them....

[Gostev]

Because as soon as any such RFC non-compliance causes unforeseen reliability issues, it suddenly becomes Veeam's problem. And when it happens, affected customers are NEVER as reasonable as you are in your post above.

[tkeith]

Fair and I get it.... hopefully we can get a hot fix soon then so we can move to 11a.
Thanks for your assistance in moving this forward.

[tkeith]

We received the fix today. We will attempt the 11a upgrade and test patch/fix tomorrow and report its behavior. Thank you all for your assistance in this issue.

[tkeith]

Fix works! Appreciate everyone's assistance on this speedy fix! I assume it will be added to any future updates.
Thank you again!
Keith

[GregorS]

I am on VeeamBackup&Replication_11.0.1.1261_20230227. Did the fix make it into this version, or do I need to open a support case to get it separately? What is the registry key?

Regards
Gregor

[Gostev]

Gregor, 20230227 was an emergency patch with a single security fix, so we did not have a chance to include ANY private fixes created since the previous cumulative patch (which is something we normally try to do when a patch release is not rushed).

[GregorS]

I opened case #06016065 and got the fix (thanks Bogdan!). I could not install the EXE contained in the ZIP since it is only intended for 11.0.1.1261 P20220302. However, the patch obviously made it into 11.0.1.1261 P20230227 since applying only the registry value took effect. According to Bogdan the patch is also part of v12.

Code: Select all

HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication
S3AllowUnsupportedSymbols [REG_DWORD] = 1

Best regards
Gregor

[Gostev]

Please be aware that it is technically impossible for a fix that was not already included in 11.0.1.1261 P20220302 to make it into P20230227. Any private fix that was created following the P20220302 release must now be rebuilt for P20230227.

[Gostev]

Although reading this thread again, the timeline would indicate that a private fix in question was created in February for P20211211, meaning it should have been automatically included into P20220302 release that came a few week later, as it normally happens with all "safe" fixes.