Move-VBOEntityData takes a long time to process each user

[bg.ranken]

We currently have an on-going case regarding this: 07013392

I believe we may have discovered a bug of some kind in recent versions, but not really sure at the moment. We are in the process of migrating from an on-prem local repository to an Azure storage account and are seeing really strange behavior using the Move-VBOEntityData command. The command completes successfully and creates the job, however the job itself takes an incredible amount of time. On average it's taken around 5 hours regardless of how much data is in the user being moved. We've seen it take the same amount of time for a user with less than 100mb as it does for a user with several GB.

When monitoring the job, we've noticed that it uploads nearly all the data for the user within the first few minutes, and then basically sets there and does nothing for the next few hours. Looking through the Veeam logs confirm this behavior; we see a multi-hour gap between the last "Moving file chunk batch" and the "Cleaning up data..." entry.

The repository we are migrating data from only contains site information, so we get expected warnings regarding Veeam not finding mailboxes, archive mailboxes, and OneDrive data (we specify all 4 flags to confirm no data is missed). We previously moved a different repository that contained only Teams data, and did not see this same kind of behavior. It almost seems like there's some sort of timeout that has to happen before the job decides to finalize the move and complete things.

Here's all the things we have tried so far:

Disabled antivirus
Opened up all firewall access
Increased resources for all proxy servers involved
Ensured we are on the latest version: 7.0.0.4388
Multiple reboots of all proxy servers
Reduced and increased threads for all proxy servers
Changed the number of concurrent move jobs were running, all the way down to 6 and up to 32 (this one does seems to have some effect on job duration, as when we were doing 6 we'd see averages of 5 hours per user but at 32 we're seeing averages around 8 hours)

At this point we are not sure what's going on. Currently support is asking questions regarding throttling or possibly something on Azure side, but seeing as there's a big gap in lobs on the Veeam side, I'm more inclined to believe it's something within Veeam.

The other issue is because of this move, the backup job has been disabled for more than a week. I know you're not supposed to re-enable the job before finishing the data moves, however at this point with the speed things are going we're looking at around another 2-3 weeks to move the remaining 600GB of data.

[Mildur]

Hello Randall

Thanks for the case number. Please continue to work with our customer support team on this issue.
There is not much we can do at this stage. Initial troubleshooting is still going on. I'll keep an eye on the case.
Please consider to use the "Talk to a Manager" option if you feel the case goes nowhere:
https://www.veeam.com/kb2320

The other issue is because of this move, the backup job has been disabled for more than a week. I know you're not supposed to re-enable the job before finishing the data moves, however at this point with the speed things are going we're looking at around another 2-3 weeks to move the remaining 600GB of data.

We are working on optimizing our Move-VBOEntityData command for our next version of VB365. The goal is to allow moving backups while the backup jobs still can run.

Best,
Fabian

[bg.ranken]

Well so far we have been working with support but haven't gotten anywhere. The tech assigned is denying support due to kb3067 mentioning that the script is provided as a courtesy and is unsupported.

Can you ask someone to update this article to make it more clear that what isn't being supported is the example script that was provided? Maybe additional verbiage in this line "Veeam Technical Support will not assist in the usage or troubleshooting of this tool." to make it clear that it is only regarding the example script and not the actual migration process. Normally I wouldn't ask this, but if the support reps are getting confused by this then I think it should be re-worded to be more clear.

I used the "Talk to a Manager" function to clarify the issue is with the running migration jobs, not the example script or even the Move-VBOEntityData command. So the case should proceed soon, but would still like the KB updated to be more clear to prevent any other support resources from misunderstanding.

[Mildur]

Hello Randall

I'm sorry to hear about the first response with the script. But I can see that the case is now back on track.

Can you ask someone to update this article to make it more clear that what isn't being supported is the example script that was provided? Maybe additional verbiage in this line "Veeam Technical Support will not assist in the usage or troubleshooting of this tool." to make it clear that it is only regarding the example script and not the actual migration process. Normally I wouldn't ask this, but if the support reps are getting confused by this then I think it should be re-worded to be more clear.

Thanks for your feedback, I will discuss it with our KB team.

Thanks,
Fabian