This case is now open for nearly a year. It deals with errors of type 400 and 404 during offload to AWS S3. One issue (404) could be solved after 10 months, there was orphaned data in a S3 bucket.... For the 400 error I opened multiple cases at AWS too and forwarded the information I received from Veeam support. Please see latest feedback from AWS support below. As we are seeing this problem at different locations in different regions (not always so persistent as in the current case) I get the feeling that indeed something might be wrong in Veeam code/logic. My problem is, that I have to be the proxy between Veeam support AWS support. As fas as I understand this topic has already reached the highest escalation level within Veeam. Why is it not possible that Veeam directly talks to AWS? I think we are far beyond a standard support case as development is involved for some time now.
I am reaching out to you with the findings from our team in reference to this issue and after performing a deep-dive investigation on the logs shared, we have found that the error “The XML you provided was not well-formed or did not validate against our published schema” is related to the “keynames” which Veeam is trying to remove not being valid XML.
Unfortunately, this is a known limitation of delete-objects (multiple) requests that some characters which can be used for these object keys are not permitted/allowed in XML document, hence, API would return this error 400.
Prior to deleting them, the “DeleteObjects” compiles a list of objects to delete in an XML document, and if an object which is requested, does not validate against the allowed XML characters, those objects would fail to delete with the aforementioned response.
The ranges of valid XML characters below are taken from here shared with a document for your reference [1]:
-----------------
#x9 | #xA | #xD | [#x20-#xD7FF] | [#xE000-#xFFFD] | [#x10000-#x10FFFF]
If any of your keys which are being requested in the DeleteObjects API call, do not validate against the valid XML characters, then we would anticipate this error.
To remediate this error, there are two options since I don’t believe Veeam will support single DeleteObject API calls:
-----------------
1.) Manually delete these objects using the CLI:
$ aws s3api delete-object --bucket --key
2.) Rename the objects using a Copy command.
$ aws s3 cp s3:/// s3:///
-----------------
Because we don’t yet know which keys are being requested in the DeleteObjects request, can you please provide me with a list of objects you believe are involved in the DeleteObjects request?
Having this information may help us identify which object key names are returning this error. Unfortunately, since the keynames are not valid, we do not see the keys requested in the S3 logs you provided.
If you’re unsure of the keynames being requested in the S3 bucket, are you able to check to see if Veeam has more verbose logging available for this job so that we may check to see which keynames are being acted on in the DeleteObjects request? I am also sharing the XML related user guide for object key in this link [2].
I will look forward to your response, in the meantime if you have any questions or concerns please feel free to respond to the case and I will be more than happy to assist.
Have a great rest of the day, stay safe and take care!
References:
[1] Extensible Markup Language (XML) 1.0 (Fifth Edition)
https://www.w3.org/TR/2008/REC-xml-20081126/#charsets
[2] XML related object key constraints
https://docs.aws.amazon.com/AmazonS3/la ... l-related- constraints