S3 Connection lost at 5 GB

[BFAndrew]

I was attempting to make a large initial backup 75+ GB to an S3 compatible bucket.

When the upload hit 5 GB, I got the message: Connection was lost the task will be canceled.

Is there a limit to file size when using Veeam with S3. 5GB is the limit of S3's PUT Object command. The multipart upload function needs to be used with larger files.

From Amazon:
Q: How much data can I store in Amazon S3?

The total volume of data and number of objects you can store in Amazon S3 are unlimited. Individual Amazon S3 objects can range in size from a minimum of 0 bytes to a maximum of 5 TB. The largest object that can be uploaded in a single PUT is 5 GB. For objects larger than 100 MB, customers should consider using the multipart upload capability.

https://aws.amazon.com/s3/faqs/

[BFAndrew]

This is a File Copy Job. There are files that are over 12 GB that will be in the backup.

Full Error Message:

Backing up 3051 files and 61 folders (5.1 GB) transferred at 296 MB/s
Connection was lost, task will be cancelled
Failed to perform file backup Error: Agent: Failed to process method {NasMaster.ExecuteBackupProcessor}: Connection was lost, task will be cancelled
Error: Agent: Failed to process method {NasMaster.ExecuteBackupProcessor}: Connection was lost, task will be cancelled

[Gostev]

Veeam integrates with object storage natively both for image-based and file-based backups. Object storage based backup repositories are supported both for primary backups as well as for backup copies.

You shouldn't be using File Copy jobs for object storage under any circumstances. Object storage has no file system so there's no notion of "files" in the first place. And you will face various issues if you try to put each source file into individual object (you've already faced one such issue due to maximum object size limit but there are other).

[BFAndrew]

I'm sorry. The job is actually a File Backup Job, not a File Copy job.

Per the documentation, this is supported.
https://helpcenter.veeam.com/docs/backu ... ml?ver=120

The max object size is 5 TB not 5 GB. The 5 GB limit is only on the PUT method. AWS recommends any file over 100 MB use the multipart method.

It seems the File Backup Job might be using the PUT method. Or, the multipart method is being used but there are "too many parts". The AWS CLI documentation mentions the "too many parts" issue and how to work around it:

Example 12: Uploading a local file stream that is larger than 50GB to S3

The following cp command uploads a 51GB local file stream from standard input to a specified bucket and key. The --expected-size option must be provided, or the upload may fail when it reaches the default part limit of 10,000:

aws s3 cp - s3://mybucket/stream.txt --expected-size 54760833024


https://docs.aws.amazon.com/cli/latest/ ... s3/cp.html

[Gostev]

File Backup jobs don't use objects larger than 64MB, so you should open a support case as I've never seen even remotely similar issues reported before with AWS S3.

[BFAndrew]

Sure will do. Thanks.

[Gostev]

S3 compatible bucket

Wait, you've been talking about AWS S3 all the time but I've just noticed you may not be actually using AWS S3 but rather some S3-compatible storage? In that case, the issue is most likely on your object storage side, perhaps its S3 protocol implementation is not actually fully "S3-compatible" as the vendor claims. Because as I've said, I never saw similar issues reported with actual AWS S3.

What object storage do you use?

[BFAndrew]

It's one on the list pinned to this topic.

I did mention "S3 compatible" in the first post.

I'm not authorized to mention their name, but I will open a ticket tomorrow morning through them.

I ran another test last night. It made it further, but it was an updated data set.
Looking through the logs I can see where 5 1 GB blocks were sent and bam...connection closed. It doesn't look like a timeout. The timing looks like a forced close from the server. Maybe a half-closed socket. Data was moving at 500 MB/sec ( megabyte)

[sfirmes]

I work with many of our s3 compatible object storage partners. If you post the Veeam Support case# or pm me with it, I will take a look at it.

[BFAndrew]

Just wanted to follow up: We found that the system was running out of RAM.
The File Backup job is using a lot of resources. I did not expect that. I thought it would just copy the files. They are already RAR files so the job has compression turned off.

[Gostev]

Veeam offers both job types. The File Copy job type is the one that "just copies the files". The File Backup job type is different, this one is built to scale protecting billions of files on many years of retention and considering all the file versions, this renders simple file copy unusable due to the sheer number of objects this would produce in a backup.