Comprehensive data protection for all workloads
Post Reply
Didi7
Veteran
Posts: 617
Liked: 105 times
Joined: Oct 17, 2014 8:09 am
Location: Hypervisor
Contact:

Backup Copy Jobs missing Data Transfer option ...

Post by Didi7 »

Hi everybody,

currently we are struggling with a BackupCopy job that should copy data from a central OnPrem backup repository to an Azure StorageContainer. No SOBR is involved currently. We copy VM backups from different site locations to a central backup repository and can use pre-configured WAN accelerators in the Data Transfer window to minimize data amount and to limit data transfer speed.

Why is this option (Data Transfer) missing, when an Azure StorageContainer is configured as a target in a BackupCopy job? We have a target gateway server involved and we have a source (OnPrem) and a target WAN accelerator (Azure) available for use, but it looks like the two WAN accelerators cannot be used in this scenario. How is limiting network traffic supposed to be done in this case, for which we configured Network Traffic Rules?

Any reply is very much appreciated.

Thanks
Didi7
Using the most recent Veeam B&R in many different environments now and counting!

*** Nominated for being the earliest early adopter of VSA ***
david.domask
Product Manager
Posts: 3645
Liked: 886 times
Joined: Jun 28, 2016 12:12 pm
Contact:

Re: Backup Copy Jobs missing Data Transfer option ...

Post by david.domask »

Hi Didi7,

I see the same behavior in my lab but not immediately aware of such a limitation. Let me check internally and will update the thread when I have more information.
David Domask | Product Management: Principal Analyst
Didi7
Veteran
Posts: 617
Liked: 105 times
Joined: Oct 17, 2014 8:09 am
Location: Hypervisor
Contact:

Re: Backup Copy Jobs missing Data Transfer option ...

Post by Didi7 »

Hi David,

thanks for letting me know. Would you say this is common behavior when using an Azure StorageContainer or might this be a bug?

Btw, I am talking about Veeam B&R 13.0.1.2067 (Windows) in case anybody wants to know :wink:

Regards,
Didi7
Using the most recent Veeam B&R in many different environments now and counting!

*** Nominated for being the earliest early adopter of VSA ***
david.domask
Product Manager
Posts: 3645
Liked: 886 times
Joined: Jun 28, 2016 12:12 pm
Contact:

Re: Backup Copy Jobs missing Data Transfer option ...

Post by david.domask »

Just confirmed, with object storage repositories from Hyperscalers (AWS, Azure, Google), this is by design; WAN Accelerators have specific interactions with the target storage that require reading blocks fairly consistently from the target storage which can be quite costly (many GET requests) and introduces other performance considerations. So not a bug, by design.

If main goal is to limit the traffic, doesn't the built-in Internet rule assist here?
David Domask | Product Management: Principal Analyst
Didi7
Veteran
Posts: 617
Liked: 105 times
Joined: Oct 17, 2014 8:09 am
Location: Hypervisor
Contact:

Re: Backup Copy Jobs missing Data Transfer option ...

Post by Didi7 »

Hi David, good to know.

So the conclusion therefore is, deployment of WAN-accelerators do not make sense, as they cannot be used with Hyperscaler target repositories (this is by design).

When configuring backup repositories based on Azure StorageContainers you have the option to use Direct "Connection Mode' or configure one or more Gateway servers.

In order to configure the right 'Network Traffic Rules', the question is ... who is speaking with whom?

If you configure a BackupCopy job for data to be copied from one OnPrem location to another OnPrem location and use a source and target WAN accelerator, the traffic on the WAN-connection runs through the source and target WAN accelerators, therefore traffic rules can be limited between the IP addresses of both WAN accelerators.

Now, if configuring a BackupCopy job for data to be copied from one OnPrem location to an Azure StorageContainer the question is who is responsible for the traffic between both backup repositories?

Is it ...

* (Source) the main backup server (as no Proxy and WAN accelerator is involved and the backup repository server only holds the data)
* (Target) the gateway server(s) that can be configured in 'Connection Mode'

or who is in the end speaking with whom?

As far as we understand, you can configure more than one Gateway server in 'Connection Mode' for redundancy purposes?

Regards,
Didi7
Using the most recent Veeam B&R in many different environments now and counting!

*** Nominated for being the earliest early adopter of VSA ***
david.domask
Product Manager
Posts: 3645
Liked: 886 times
Joined: Jun 28, 2016 12:12 pm
Contact:

Re: Backup Copy Jobs missing Data Transfer option ...

Post by david.domask »

Correct, in this case use the Internet Rule if you need to throttle the connection.
Now, if configuring a BackupCopy job for data to be copied from one OnPrem location to an Azure StorageContainer the question is who is responsible for the traffic between both backup repositories?
This step controls how you connect to the target object storage repository, either Direct Connection or through a gateway, and it works as you suppose in your post. You can configure multiple gateways for redundancy purposes naturally.

For Internet rule however, it works by detecting if the target connection is NOT in the private IP classes; if we're targeting a public IP, then Internet rule applies.
David Domask | Product Management: Principal Analyst
Didi7
Veteran
Posts: 617
Liked: 105 times
Joined: Oct 17, 2014 8:09 am
Location: Hypervisor
Contact:

Re: Backup Copy Jobs missing Data Transfer option ...

Post by Didi7 »

Many thanks for replying my questions David, appreciated.
For Internet rule however, it works by detecting if the target connection is NOT in the private IP classes; if we're targeting a public IP, then Internet rule applies.
That is something though that really makes us a little bit skeptical. As we only use none public IPs to interact with the corresponding Azure StorageContainers, we wonder how we can throttle traffic in this case?

Our datacenter infrastructure is connected through an Express Route to Azure but why can't we throttle traffic by adding a Veeam Network Traffic Rule with the Veeam Gateway Server IP addresses in Azure and the main Veeam Software Appliance in our datacenter?

Thank you.
Using the most recent Veeam B&R in many different environments now and counting!

*** Nominated for being the earliest early adopter of VSA ***
david.domask
Product Manager
Posts: 3645
Liked: 886 times
Joined: Jun 28, 2016 12:12 pm
Contact:

Re: Backup Copy Jobs missing Data Transfer option ...

Post by david.domask »

Got it, with PrivateLink it is a bit more complex, and regrettably you'll need to create rules for the IPs used by your Private Endpoints, and if I remember right these may rotate or be a rather large range, so it may be a few rules.
David Domask | Product Management: Principal Analyst
Didi7
Veteran
Posts: 617
Liked: 105 times
Joined: Oct 17, 2014 8:09 am
Location: Hypervisor
Contact:

Re: Backup Copy Jobs missing Data Transfer option ...

Post by Didi7 »

Hello David, in this case the OnPrem backup server is one IP address and the Gateway server in the cloud is another IP address, so the rule is pretty simple to build, as we discussed these are the components that talk to each other to transfer the backup data from an OnPrem backup repository to the Cloud storage container.
Using the most recent Veeam B&R in many different environments now and counting!

*** Nominated for being the earliest early adopter of VSA ***
Didi7
Veteran
Posts: 617
Liked: 105 times
Joined: Oct 17, 2014 8:09 am
Location: Hypervisor
Contact:

Re: Backup Copy Jobs missing Data Transfer option ...

Post by Didi7 »

Using the storage container in Cool-mode worked pretty fast, using another storage container in Hot-mode didn't work flawlessly and we had significant transfer problems with the following error ...

Hard disk 1 (50 GB)

Error: Unstable connection: unable to transmit data. Failed to upload disk 'TESTSERVER-flat.vmdk'
Exception from server: End of file [asio.misc:2] Unable to retrieve next block transmission command.
Number of already processed blocks: [3908]. Failed to download disk 'TESTSERVER-flat.vmdk'.
(Unstable connection: unable to transmit data.) (Failed to upload disk 'TESTSERVER-flat.vmdk')

(Exception from server: End of file [asio.misc:2]) (Unable to retrieve next block transmission command. Number of already processed blocks: [3908].) (Failed to download disk 'TESTSERVER-flat.vmdk'.)
Processing finished with errors at 5/8/2026 8:12:06 PM

We wonder where's the problem now? What does the error tell us? Problems downloading the 'TESTSERVER-flat.vmdk' from the source or problems uploading the 'TESTSERVER-flat.vmdk' to the target?

We are not sure. Maybe someone else knows how to interpret this error?

In the meantime, we will transfer the relevant VIA-based Hardened Backup Repository (a VM) from one to another VMware cluster, where the source is a Nutanix cluster based on NFS storage and the target is an HPE cluster with dedicated FC-storage.

Thank you.
Using the most recent Veeam B&R in many different environments now and counting!

*** Nominated for being the earliest early adopter of VSA ***
david.domask
Product Manager
Posts: 3645
Liked: 886 times
Joined: Jun 28, 2016 12:12 pm
Contact:

Re: Backup Copy Jobs missing Data Transfer option ...

Post by david.domask »

Error: Unstable connection : unable to transmit data. Failed to upload disk 'TESTSERVER-flat.vmdk'
https://www.veeam.com/kb2140

From experience, "unstable connection" means exactly like the KB article tells -- this error has some validation leading up to it to try it, meaning it's not triggered just by a flaky connection but instead the behavior experienced by the data mover agents suspects it is being blocked.
David Domask | Product Management: Principal Analyst
Post Reply

Who is online

Users browsing this forum: ruddj, Semrush [Bot] and 88 guests