Initial Backup Copy failing with Key related errors

[mrowell]

We have created a Backup Copy job and are seeing the job fail to complete on the initial and subsequent runs.

Errors in the console are:

Code: Select all

The given key was not present in the dictionary.
An item with the same key has already been added.

Version:
6.0.0.379
Case ID #:
02580909

We have WCF OperationTimeout set to "2400".

Example of error:

Code: Select all

04.06.2022 09:31:37]  117 (8436) Data decompressed (object key: Veeam/Backup365/veeam/Organizations/52d4818d9a3f4db08237c730973dde85/Webs/909c70c87b874aa29a63889a5ecff1f1/Files/Data/73a490512efe4d9ba0e00f4ce2e9eb30.00000001, compressed size: 1156969, decompressed size: 4201854)
[04.06.2022 09:31:37]  117 (8436)  Saving file chunk (ID: 840f22cc-1ed0-4c1b-b61f-29d81d3264e0.1.1.512.0, length: 364)...
[04.06.2022 09:31:37]  117 (8436)  Saving file chunk (ID: 30345587-7465-4a71-9e92-7a13510f6bc9.1.1.512.0, length: 680)...
[04.06.2022 09:31:37]  117 (8436)  Saving file chunk (ID: 24e2f7d4-87e2-4e3b-b050-b823a838708a.1.1.512.0, length: 7980)...
[04.06.2022 09:31:37]   63 (5540) Error: The given key was not present in the dictionary.
[04.06.2022 09:31:37]   63 (5540) Type: System.Collections.Generic.KeyNotFoundException
[04.06.2022 09:31:37]   63 (5540) Stack:
[04.06.2022 09:31:37]   63 (5540)    at System.ThrowHelper.ThrowKeyNotFoundException()
[04.06.2022 09:31:37]   63 (5540)    at System.Collections.Generic.Dictionary`2.get_Item(TKey key)

This appears to be an issue reading from the Primary Object Repository. We've tried changing the destination of the Copy Job to different folder in the Same S3 bucket, but still get the same errors.

[Mildur]

Hi Marcus

Welcome to the forums.

Please update to the newest Build 6.0.0.385 (P20220524). You are running on GA. There is a cumulative update which resolves some issues with copy jobs.

https://www.veeam.com/kb4285

Copying the backup of a OneDrive account may fail with the error:
The given key was not present in the dictionary

Thanks
Fabian

[Mike Resseler]

And if that doesn't work, please create a support call and let us know the support case ID.

What storage are you using for the backup copy? Archive tier / Glacier or Deep Archive?

[mrowell]

Fabian and Mike, thank you very much for the feedback.

Case ID #: 02580909

I've upgraded to the latest version as suggested.

The source repository is Wasabi. The destination is an AWS Glacier Deep Archive.

I created another new backup repository (same S3 bucket, different folder) and started the copy job again on Saturday afternoon.

The update appears to fix the “The given key was not present in the dictionary “ errors and increased the data transfer rate.
I am now only seeing Time Out errors and a single “An item with the same key has already been added”.

Code: Select all

[04.06.2022 19:18:12]   65 (10376) Error: An item with the same key has already been added.

In regards to the Time Out errors, I am writing to an AWS region on another continent, so latency/roundtrip might be an issue. The odd thing is that I haven’t seen a single retry in the Exchange Copy job, which is copying to the same AWS Region and has been over ran 6 times. Maybe the larger objects in a SharePoint/Teams backup trigger the Time Out.

I do have some hardware constraints on the VM running VBM365. Whilst the VM has 8 virtual cores and 24GB of RAM, the host only has 4 physical cores but it otherwise lightly loaded. Hence I am only running 12 threads on the Proxy.

Any further suggestions would be greatly appreciated.
Thanks, Marcus.

[Mike Resseler]

Marcus,

Pass this information in the support ticket please. The one with the same key but also the time out errors. They might be able to help you. I am not sure if latency would be the issue in this case.

[mrowell]

This issue was eventually resolved with some custom updated dlls for VM365 and a few tweaks to the Config.xml and Proxy.xml.

My understanding is that limited 100Mbps bandwidth and the increased latency of a cross continent copy triggered the errors.

Changes to VBM365, which I believe have been included in 6a, were required to accomodate these slower, high latency network conditions.

We sincerely thank Veeam for resolving this issue, I think many other companies would have said "get better internet" or "you are doing it wrong".

It was a lot work for both Veeam and Us, but a good solution in the end that hopefully make VM365 more robust.