Feature Request: Modular Processing of Veeam Agent Job

[micoolpaul]

Hi,

Full Disclaimer I have a case that has frustrated me and makes me think Veeam could handle this in a better way. Case #04955644

Summary of case:
Customer is backing up to a Veeam Cloud Connect repository. It's a Windows Server approximately 2TB of data, takes a few days to create an active full. Job successfully processes all disks, then complains about a session being closed when uploading the metadata. As a result there's a full VBK of all the relevant data, but no metadata uploaded, meaning Veeam wishes to carry out another FULL backup. However I've had this issue occur multiple times for the customer.

Feature Request 1:
Request: Veeam to track its metadata locally on the machine in the event of connection interruptions.
Benefit: Veeam will prevent frustration when what should be a recoverable error instead results in another full backup needing to take place.

Feature Request 2:
Request: Veeam to track each disk individually within a full system backup job. Being able to process disks that have been uploaded successfully as incremental whilst processing disks that haven't been uploaded completely as full backups.
Benefits:
1. We have more hybrid and remote workers than ever before and this won't be going away. Businesses wish to protect data beyond their perimeter, however if a user has a huge amount of data that needs protecting, and likely a poor upload speed this will take a while and worker machines are far more likely to experience interruptions in connectivity and uptime.
2. I fully understand that physical machines can't just retain a snapshot like VMs however if we at least could process disks individually, in the event of a connection interruption if we assume a customer has a C & D drive. If the job failed midway through processing the D drive but the C drive had been uploaded. That backup would still help the customer should they need to restore their boot volume.
3. If users have data caps, we're more efficient with their bandwidth.
4. Retries take less time as we don't need to process as much data, we can move the uploaded disks to being processed with CBT tracking and focus the available uptime and bandwidth on the required uploads.

I'd love to hear the thoughts from the community and how they currently tackle these restrictions.

[HannesK]

Hello,
if you are also fine to hear thoughts from Veeam: as you talk about C and D drives, I assume that you are using backup for "entire computer" or "volume level backup", right?

Are you using backup cache and is it the very first full backup of that machine?

https://helpcenter.veeam.com/docs/agent ... tml?ver=50
https://helpcenter.veeam.com/docs/agent ... tml?ver=50


Thanks,
Hannes

[micoolpaul]

Full computer backup.

Yep building a new chain, carried out a full backup, it processed all disks successfully then decided the connection had been closed and so it didn’t send the metadata.

I haven’t seen anything that would imply backup cache would mitigate the issues I’m facing here unfortunately whereby Veeam will retry the entire job when all disks were successfully processed but the metadata just wasn’t sent.

[HannesK]

I'm asking because I remember that backup cache helps will all kind of connection issues which is the reason why I always recommend it. So I would give it a try as it might solve the issue without having to wait for new features getting implemented.

[micoolpaul]

Was being managed by Veeam B&R so it wasn't an option. I've just had to make some changes so I've changed the deployment to use backup cache. Though I do still think this would be a good feature to complement backup cache.

[Dima P.]

Hello Michael,

Mind me asking what Veeam B&R & Agent version you are running? With latest Veeam B&R v11 and VAW v5 we've added resume backup functionality (for entire computer and volume level backup modes) to continue the interrupted backup and reuse already uploaded data blocks. Resume should be performed with or without backup cache functionality as it's been integrated to the backup job engine. Thanks!

[micoolpaul]

Hi Dima!,

Latest Veeam B&R & VAW v5 releases, I've definitely not seen that resume backup functionality working in the wild, every retry just reprocesses the full job again, even in this scenario whereby the disks were fully uploaded and only the metadata file needed to be sent, it started the whole job again.

[Dima P.]

Possibly you have a confirmation in a lab or at your customer side to start the investigation? If so please raise a case and share it with me.

[micoolpaul]

Hi Dima,

Case is already in progress, reference Case #04955644

[Dima P.]

Seems that I've missed the id from your first post, sorry for that. We will check your logs with the team, stay tuned for updates from support team.

[kevlahau]

We also have been getting this disconnect error and were forced to assist the customer to set up a local primary backup and then use a copy job. This is fine a work around but really need to get the primary backup working. Seems V11 & VSP 5 is very latency sensitive.

We also endorse this feature request.

[Dima P.]

Hello Kevin,

Do you have a case ID to share? Thank you in advance!

[kevlahau]

Hi Dima, Case #04961966

[Dima P.]

Hello folks,

Looks like in both cases the problem was related to the connectivity issues and not resume functionality.

Case #04955644

The problem was related to one of gateways.

Case #04961966

This one was related to the network latency.

Please let us know if you need any further assistance!

[micoolpaul]

Hi Dima,

I see what you're saying but wouldn't you agree that resume functionality should resume from connection interruptions? As that's one of the most likely scenarios that a resume would be required?

Thanks!

[Dima P.]

Michael,

Sure, fully agree. However if the connection between agent and repository cannot be established due to the component issues, resume wont help. Thanks!

[micoolpaul]

Hi Dima,

Of course in such a scenario it would eventually have to fail. One example I can think of as to something I'd expect to resume would be as follows:

Backup upload completed for the .vbk file, this goes to one Veeam Cloud Connect gateway, the Cloud Connect gateway then crashes, all that's remaining is an upload of a metadata file, it seems strange that it wouldn't attempt to reconnect and send this via another Cloud Connect gateway and instead fails. The main reason I find this so peculiar is because the metadata file is only kilobytes in size, but instead we have to resend potentially an entire multiple TB file because of a random connection drop, no different to a single node in a load balancer failing.

I appreciate the engagement on this and I hope this is being seen as constructive feedback for the product only!

[kevlahau]

I concur Michael. This was exactly the issue we were seeing. Backup in question was about 2T and on almost every occasion the 2T would be written but the backup would be failed when completing the job. So I suspect that there is some thread open used for the meta data that is dropping or timing out whilst the backup is running .

[Dima P.]

Hello folks,

Thank you for the feedback! I've asked RnD folks to review the case details once again to better understand why resume functionality did not work. I assume the original job cycle has failed because of the infrastructure issues so no retry was performed.