Move-VBOEntityData takes a long time to process each user

[bg.ranken]

We currently have an on-going case regarding this: 07013392

I believe we may have discovered a bug of some kind in recent versions, but not really sure at the moment. We are in the process of migrating from an on-prem local repository to an Azure storage account and are seeing really strange behavior using the Move-VBOEntityData command. The command completes successfully and creates the job, however the job itself takes an incredible amount of time. On average it's taken around 5 hours regardless of how much data is in the user being moved. We've seen it take the same amount of time for a user with less than 100mb as it does for a user with several GB.

When monitoring the job, we've noticed that it uploads nearly all the data for the user within the first few minutes, and then basically sets there and does nothing for the next few hours. Looking through the Veeam logs confirm this behavior; we see a multi-hour gap between the last "Moving file chunk batch" and the "Cleaning up data..." entry.

The repository we are migrating data from only contains site information, so we get expected warnings regarding Veeam not finding mailboxes, archive mailboxes, and OneDrive data (we specify all 4 flags to confirm no data is missed). We previously moved a different repository that contained only Teams data, and did not see this same kind of behavior. It almost seems like there's some sort of timeout that has to happen before the job decides to finalize the move and complete things.

Here's all the things we have tried so far:

Disabled antivirus
Opened up all firewall access
Increased resources for all proxy servers involved
Ensured we are on the latest version: 7.0.0.4388
Multiple reboots of all proxy servers
Reduced and increased threads for all proxy servers
Changed the number of concurrent move jobs were running, all the way down to 6 and up to 32 (this one does seems to have some effect on job duration, as when we were doing 6 we'd see averages of 5 hours per user but at 32 we're seeing averages around 8 hours)

At this point we are not sure what's going on. Currently support is asking questions regarding throttling or possibly something on Azure side, but seeing as there's a big gap in lobs on the Veeam side, I'm more inclined to believe it's something within Veeam.

The other issue is because of this move, the backup job has been disabled for more than a week. I know you're not supposed to re-enable the job before finishing the data moves, however at this point with the speed things are going we're looking at around another 2-3 weeks to move the remaining 600GB of data.

[Mildur]

Hello Randall

Thanks for the case number. Please continue to work with our customer support team on this issue.
There is not much we can do at this stage. Initial troubleshooting is still going on. I'll keep an eye on the case.
Please consider to use the "Talk to a Manager" option if you feel the case goes nowhere:
https://www.veeam.com/kb2320

The other issue is because of this move, the backup job has been disabled for more than a week. I know you're not supposed to re-enable the job before finishing the data moves, however at this point with the speed things are going we're looking at around another 2-3 weeks to move the remaining 600GB of data.

We are working on optimizing our Move-VBOEntityData command for our next version of VB365. The goal is to allow moving backups while the backup jobs still can run.

Best,
Fabian

[bg.ranken]

Well so far we have been working with support but haven't gotten anywhere. The tech assigned is denying support due to kb3067 mentioning that the script is provided as a courtesy and is unsupported.

Can you ask someone to update this article to make it more clear that what isn't being supported is the example script that was provided? Maybe additional verbiage in this line "Veeam Technical Support will not assist in the usage or troubleshooting of this tool." to make it clear that it is only regarding the example script and not the actual migration process. Normally I wouldn't ask this, but if the support reps are getting confused by this then I think it should be re-worded to be more clear.

I used the "Talk to a Manager" function to clarify the issue is with the running migration jobs, not the example script or even the Move-VBOEntityData command. So the case should proceed soon, but would still like the KB updated to be more clear to prevent any other support resources from misunderstanding.

[Mildur]

Hello Randall

I'm sorry to hear about the first response with the script. But I can see that the case is now back on track.

Can you ask someone to update this article to make it more clear that what isn't being supported is the example script that was provided? Maybe additional verbiage in this line "Veeam Technical Support will not assist in the usage or troubleshooting of this tool." to make it clear that it is only regarding the example script and not the actual migration process. Normally I wouldn't ask this, but if the support reps are getting confused by this then I think it should be re-worded to be more clear.

Thanks for your feedback, I will discuss it with our KB team.

Thanks,
Fabian

[sraza]

Hello,
We are facing exact same issue, is there any update on this ?

[Mildur]

Hi Sraza

I'm sorry to hear that you are facing migration performance issues.
Please open a case with our customer support team. Randall's case is still in progress, and his migration may be affected by a different cause than yours. Don't forget to provide me with your case number.

Best,
Fabian

[sraza]

Hi Fabian, here you go.
Veeam Support - Case # 07037212 - Migration of local repositories to Wasabi is extreamly slow
Case was oppenned on 5th Dec 2023, since then comms are going back and forth with no solution so far, pretty frustrating !
After looking the MoveJob logs, found exactly the same behaviour as below.
When monitoring the job, we've noticed that it uploads nearly all the data for the user within the first few minutes, and then basically sets there and does nothing for the next few hours. Looking through the Veeam logs confirm this behavior; we see a multi-hour gap between the last "Moving file chunk batch" and the "Cleaning up data..." entry.

[bg.ranken]

So our case is still ongoing unfortunately with no resolution.

I can note that the log gap appears to be due to the way that Veeam uploads the data. At least that's what I think based on logs, it moves the data into the cache and then upload from there without adding any logs to the Move.web logs for some reason. For us our source local repository is on a proxy server that's not the "master", and the target object repository is on the "master" server, which causes the gaps in the logs to appear. We found this out by moving our target object repository to the same proxy that has the local repository, and then all the Move.web logs started showing Blob upload entries (when advanced logging is enabled). Apparently the blob upload entries do show on the server that has the object storage attached under the Veeam.Archiver.Proxy logs instead of one of the Move.web logs.

Unfortunately moving the object repository had a negative effect. In reviewing the history, all of our jobs were completing anywhere from 1:45 to 1:55 when running 4 concurrently. One thing we noticed was how consistent this time frame was, and it didn't seem to be affected by site size or object count; we would have almost 10 jobs in a row finish with a runtime with a minute variance of each other. The only thing that changes the runtime of the jobs is how many are going concurrently, reducing concurrent jobs will make them go quicker, but all jobs will still take roughly the same amount of time as other jobs regardless of size or object count. After moving the object repository from the "master" server to the same proxy server hosting the local archive, it made all jobs take a consistent 2:40 runtime with 4 concurrently.

I really think there's something going on. The fact a site migration with 25mb and 50 items takes nearly the exact same time to upload as a 900mb and 5000 item site really says there's something going on. Opening a case with Microsoft they confirmed there's no throttling on their end and we are well under our limits. To date we haven't been able to see any bottlenecks in our chain, nothing on Azure site, nothing in the pipes to azure (10G direct internet or 10G over an express route); and Veeam just states the bottleneck is the target.

One thing that was noted by the tech though is that just running the Get-VBOEntityData command is extremely slow. For me it has always taken anywhere from 10 to 20 minutes to run the command against a repository; apparently this isn't expected and the tech said it should be instant.

So at this point we are planning on moving everything to the master server, both the target object repository as well as the source local repository (detaching the VMDK from the original proxy and attaching it to the master) in hopes it helps, but honestly I don't think so at this point. There is constant traffic going to the object repositories, but it's almost like there's some hardcoded throttle somewhere within Veeam that keeps uploads at a very consistent rate.

[Mildur]

@sraza
I apologize for the late answer.

We now have escalated the case from our side (product management) within our support organization.
Currently I don't have much more information. We need to see what support and RnD together can find out.

We know that the current migration process is not perfect. That's why we already have invested time to research and development to optimize the feature in our upcoming version. But I understand, it won't help you in your current migration project.

Best,
Fabian

[bg.ranken]

@mildur

Is there any way that this can be tested internally at Veeam to see if there is some issue with the current build with Azure and might be a bug? So far for me it seems limited to 1Mb/s somewhere, and while I accept it might still be on our end somewhere, it would be nice to know it's been confirmed it should be able to go faster than that. In testing with other tools on the same server (Azure Data Explorer) we can easily get 10-50MB/s upload to the same storage account, even when using tons of small files. I know you guys do QC for each new build but perhaps actual throughput specifically to Azure is not tested.

And yes, I know the new version probably wouldn't help us at this point. It would be nice to still be able to have the backup job run while migrating, but if it's still just as slow as it is now then there's potentially a problem somewhere.

[admcomputing]

We too are experiencing a similar behavior as @sraza.
Migrations from local repositories to Wasabi storage were running find until around the time we deployed XML document hotfix.
If logs are needed from us, I can raise a support case today

[bg.ranken]

For us, we are not on that hotfix. We are still on 7.0.0.4388 P20231015, but were on the version prior and can confirm that issue had the same problem.

@adamcomputing Are you seeing any throughput numbers that appear to be throttling? For us we were going to Azure, and I'm somewhat convinced there's a 1Mb/s throttling limit hardcoded somewhere, but I was under the impression that might only exist to Azure storage accounts. I'd be curious if you are seeing the same thing to Wasabi.

[admcomputing]

We are on 7.0.0.4388 P20231015 but have deployed the manual EWS hotfix.

I kicked off a 32 user move on the 01/01/24 and those jobs are still running now. We are seeing a network cap of 6.6Mbps according to task manager, it's flat-lined.
I submitted a 1 user move last night and that completed in just over 6 hours and the status shows it transferred 17MB.
All jobs are showing the target as the bottleneck so I'll see if I can raise a support ticket with Wasabi to see if they have implemented any throttling on our account.
This is my 4th local repository I am trying to move and is about 3TB in size, the 3 previous ranged from 2-8TB and they migrated fine over a couple of weeks.
Good luck

[admcomputing]

Hi @bg.ranken, just wondering how you were getting on with this issue?

I setup a new VBM365 server using a slightly older version but have the same issues - have now upgraded to 7.1 and still the same.
When I start a new batch of users migrating, I see alot of upload bandwidth in use, 100-500Mbps, but almost around the 90minute mark, it drops to 7Mbps and flat lines until it completes, which can be up to 6-8 hours for a 4 user batch.

Ive just logged a ticket with support as this is taking too long for us: Case #07083010

[bg.ranken]

We hadn't made any progress really. I too upgraded to 7.1.0.1401 P20231218 in hopes it would fix it, but haven't started a new move yet to see if it helps or not.

However I did just get an update from my case yesterday regarding a fix from the development side. Only issue is they mentioned the fix was for the older version we were using (7.0.0.4388). I'm still waiting to hear back if the fix applies to the newer version or not, but they do specifically say it addresses an issue regarding the exact slowness we are seeing.

My case is higher up if you want to ask your case manager if they can provide you the same update.

[bg.ranken]

So this past Friday we were provided the private hotfix for 7.1.0.1401 P20231218. The migration jobs for our one-drive repository went from taking an hour and 10 minutes consistently to 30 seconds on average. So this patch definitely resolves whatever was getting stuck.

So far the longest job I've seen took 8 minutes for 10GB, which I feel is acceptable. But most of our users have less than 100mb and those jobs just fly by now compared to pre-patch. I'm really curious what the issue was though, but so far I haven't been provided that data yet.

[bg.ranken]

Here's what I got from the support rep, unofficial response of course:

The issue is caused by how we were attempting to update actual restore point data (as opposed to backup data) in object storage during that end process. My colleagues adjusted things to push the updates in batches instead of sequentially and this resolved the issue. There was also a memory adjacent portion to it but the explanation presented that as "Oh yeah, we also noticed this but it was a symptom not a cause".

And I can confirm this behavior. I have a monitor script that was repeated querying the repository with Get-VBOEntityData to find things like used storage, user count, etc., that I was using to track progress. Previously it showed a decrease in storage space and users a lot quicker, but now I after the patch that I was able to process almost 200 users before the storage space showed as being decreased.

I'm just glad it's fixed, and this seems like it was a long term problem based on other people who had previously reported the issue. Hopefully an actual KB with a real patch gets released soon.

[DaStivi]

hi there, my experience with move-vboentityData, onPremis from one Jet to another also was very bad... the time it takes is one thing... another was that some objects couldn't be moved at all... and next one is, that the Destination JET DB is "corrupted" what the size of the objects in there is regarding... ah Repository that took like 17TB before now shows around 30 or 50TB and also grow unexpectedly faster then all other... strangest thing about this issue is, that the JetDB Files still only consume 17TB on disk...

another customer with around 60TB on Disk, according to the Powershell Commands from VB365 consumes 6600TB of repos...

you can imagine what ah hassle this is to get the customers ah correct bill for the used space (vb365-as-a-service)... obvisously we can't charge for the 6600TB eventhough i would like to and get ah big bonus of it each month..

my learnings are, don't use the move-vboentitiydata command!

i hope the V8 move will get better....

right now we start with fresh S3 Buckets for our customers and we just gift the customers the space we need for this double retentionholding

[bg.ranken]

Well I just wanted to report back to confirm that the issue is indeed fixed with the private hotfix. So far no official word on when it will be implemented into an official patch or version.

Before the patch we had it took nearly 2 months to move 1TB of data, after the patch we completed a full migration of a 2.5TB repository to Azure in about 48 hours. And it probably could have been faster, but we still had things limited to only 4 concurrent jobs at a time, and by the time someone reviewed the migration process things were already completed, so we couldn't test more concurrent jobs. It most likely could have gone even faster.

[DaStivi]

hi there, my experience with move-vboentityData, onPremis from one Jet to another also was very bad... the time it takes is one thing... another was that some objects couldn't be moved at all... and next one is, that the Destination JET DB is "corrupted" what the size of the objects in there is regarding... ah Repository that took like 17TB before now shows around 30 or 50TB and also grow unexpectedly faster then all other... strangest thing about this issue is, that the JetDB Files still only consume 17TB on disk...

another customer with around 60TB on Disk, according to the Powershell Commands from VB365 consumes 6600TB of repos...

you can imagine what ah hassle this is to get the customers ah correct bill for the used space (vb365-as-a-service)... obvisously we can't charge for the 6600TB eventhough i would like to and get ah big bonus of it each month..

my learnings are, don't use the move-vboentitiydata command!
...

my issues, that the destination Repos got "blown-up" has also been fixed with the latest 7a release update... pretty interesting never thought this could happen... another side-effect i still have to investigate is that now there are ah lot less licenses counted!

[bertdhont]

We are experiencing exactly the same problems while migrating to object storage. Jobs are started, and logging shows a huge gap between the last "Moving file chunk batch" and the "Cleaning up data..." entry.
We are running Veeam Backup for Microsoft 365 7a P20240123
Opened a case #07133822

[nickfranzen]

I'm dealing with this same issue now. All the jobs are taking between 12 - 15 hours per user. It seems to have completed moving the data within the first couple hours but then just sits. I have a case open now.

[sknetsch]

Doing an on-Premise from one Jet to another Jet and it is ridiculously slow. Can confirm. This tool is next to useless, but I feel like I am committed now.

I can probably restart my backups sometime in June it looks like.

Trying to decide what's worse, letting this process play out, or just giving up and backing everything up to a new target and eating the time to do a full backup of my 365 environment all over again.

Repo management is so poorly handled, I will be investigating alternatives to Veeam. Moving my target to new storage shouldn't be this hard, this unsupported, and this confusing.

[bg.ranken]

I'm dealing with this same issue now. All the jobs are taking between 12 - 15 hours per user. It seems to have completed moving the data within the first couple hours but then just sits. I have a case open now.

Have you requested the same hotfix I was supplied. It completely resolved my issue, but so far I haven't gotten any word on if it has been included in the latest version or not.

[sknetsch]

I'm on version 7.1.0.1501 P20240123 I had assumed the hotfix would be incorporated by now, given how long ago the hotfix was available. I have opened Case #07218054 and requested it.

[sknetsch]

Case #07218054
Hotfix is incorporated into 7.1.0.1501 P20240123

Apparently my issue is "Bottleneck: Source" trying to read from the JET DBs

[nickfranzen]

I did and they supplied me the hotfix. It did lower the memory that the jobs were using but didn't really speed up the jobs too much. With 64 GB, prior to the fix, I was only able to process around 6 jobs at a time. Now, I'm processing 24 jobs at a time. Most jobs are taking 8 - 12 hours to complete. I'm nearly through the migration but this has taken me 3 weeks!