Migrate backup data to Object Repository

[squebel]

It was a mix, yes. No Teams data though. We're on v6.

[squebel]

I found the issue as to why the job was enabled before all of the move operations completed: Once the last move operation is kicked off and the active tasks drop below the max thread count allowed, it moves onto the next line of code. It should not do that. At that point, it should start to watch for any Move operations and then only move on if it returns no active Move operations. I've added that code to my version of the script. It looks like this:

while((Get-VBODataManagementSession | ? {$_.Status -eq "Running" -and $_.jobname -like "Move*"}).Count -gt 0){
write-host "Moves still active..." -foregroundcolor Yellow
sleep 30}

As far as the slowness and some move tasks just sitting there and moving no data... no clue.

[Mike Resseler]

For the slowness. We are performing some tests at this point in time to see what is going on. However, those tests will take quite some time as we wanted to test with a big set of data.

[Polina]

@squebel

As Mike said above, we're doing some tests on our side, but seeing your case may provide us additional food for thought. Since you're in the middle of a migration and the logs are available, can I ask you to open a support case and share with engineers your configurations and logs? Send me to please your case ID (or just post it here), so that I could follow it up together with support and RnD and investigate the tasks that are "just sitting there and moving no data".

Thanks!

[squebel]

I'll get that case opened and let you guys know. I have a couple more questions on this process:
1) How much RAM should each thread use during the move? I have a vm with 14 cpus and 48GB RAM and I'm limiting the moves to 28 threads (2x the cpu count) and seeing active memory on the server basically pegged the whole time
2) Is this truly a MOVE of the data or a copy? - I ask this because what if I start up a move of 5TB of data, it's 2 days in and not going fast and I want to stop? Can I stop the move tasks and start the job back up using the original local disk repo?

[Polina]

1) This consumption is not specified, threads are taking as much as they need
2) Data is first copied to the target location, and deleted from the source afterward. If you stop the moving task, technically, nothing critical should happen. Note though, that when you resume it again, it'll take additional time to check which items already exist in the target repository (there's no delta or change tracking).

[squebel]

@Polina

Sorry, I wasn't clear about my ask on #2. If I stop the Move operations and not all of the data is moved to the new repo and I reenable the M365 backup job going to the original repository, will the backup job basically do a check to compare the repo data vs M365 data and then copy down (from M365) the data that is not in the repo but should be based off the retention set on the job?

[Polina]

If a backup job identifies that some data fitting the retention is missing in the repository, it'll download it again from the source.

[jaredk]

Im getting similar stuff with a move to S3 Object, started off running fast for a day, now basically stopped, no disk or network traffic but 50% use over 12 CPU threads. Still 14 of 115 user moves to do. Every once and a while it decides to start up again and it speeds up and finished one user then stops.

Had a odd thing to, had moved to a new server and was trying to move the data to object storage, the script would not even start, tried powershell, still would not start. Set backup the jobs to the disk repositorys again to get some backsups. After a couple days tried the move again and they started!

[jaredk]

One more thing i found while moving a 10TB repository, a lot of the time there is no/very little network traffic but still high CPU. The move jobs also look like they have moved very little data, sometimes for days. Then all of a sudden you look and find the move job finished and moved 100GB.

[Polina]

Jared,

Thanks for sharing. Our RnD is now closely investigating the data move process with the goal to improve the process, and real-life examples can significantly help us. If you could provide us with the information on your move results via a support case (logs, details on the infrastructure, network, and hardware), I'd greatly appreciate it. And I'll need your support case ID to escalate it to RnD.

Thanks!

[squebel]

@Polina

I've opened case 05611753 regarding the move operation slowness.

I'll echo what Jared has said about the little not no network traffic but high cpu/memory usage because I'm seeing the same thing at times. I too have noticed that it seems to go in bursts. For the past hour, it's been doing something against one final user but no data movement at all and then all of a sudden it starts pushing data out to the S3 bucket.

[Polina]

Are you moving the data to a new repository, or the move task has been stopped and then resumed?

[squebel]

I was running a Move operation using the Move-VBOEntityData cmdlet for a tenant that had about 250 users and under 4TB of total data. I never interrupted any of the move operations. I took 29 hours to move one user's OneDrive data that was less than 20GB.

[Polina]

OK, thanks for these details. I've asked to escalate your case to RnD and will watch it closely.

[squebel]

Support has asked me to upgrade to 6a and install an additional dll file that will help with some advanced logging or something to that matter. I will try to do this tomorrow.

[Polina]

Correct, that's the plan. v6a has quite a lot of optimizations, and we want to ensure if they change the game in this case. If not, the extended logging will provide us insights on bottlenecks to optimize further.

Appreciate your cooperation!

[squebel]

v6a with the provided dll has been installed and the proxy servers upgraded as well. I have another tenant I plan to migrate to an object repo tomorrow. Guess we'll see how this performs.

I want to make sure that when the moves are all completed to the object storage repository, the old local repository disk can be safely moved and deleted, right? I'm assuming that the move operation has moved all the data, kept the retention on the object, and that there's no reason to keep the old local disk repo hanging around.

[DanielJ]

I am planning to migrate to object storage, and this thread does not fill me with confidence. Has anyone tried this for multi-TB repositories after upgrading to 6a? I could live with the migration taking weeks or months if there is some way of continuing to do incrementals while at the same time migrating old data in the background. But it sounds like there isn't.

[crun]

Are there some improvements planned (e.g. mentioned by Daniel concurrent backup & migration) in V7? Looking at the horror stories, migration does not seem viable option yet

[chris2737mcdo]

I've been doing some testing and here is what I have found so far.

Using:
v7.0.0.3604 P20230512
Script from https://www.veeam.com/kb3067
59 users (Mail, OneDrive, Sharepoint, Teams)
Local Repository - 2TB VMDK attached to VM with 16GB RAM, 2 vCPUs (1.8TB backup data to move)
Target Object repository - on prem MinIO S3 bucket - No versioning / Immutability set

1st attempt:
Backed up VM using VBR (just in case )
Took snapshot
Disabled job and ran script
After 5 mins all RAM / CPU consumed. VM became unresponsive and had to be powered off
Snapshot reverted
New bucket created as some data (very little) had been copied.

2nd attempt:

Reading posts here, I decided to increase VM RAM to 100GB
Took snapshot
Disabled job and ran script
Move User tasked started to increase. About 10 tasks after a few minutes.
Tasks keep increasing as does RAM and CPU usage.
RAM finally hits 100% used as more move tasks are created.
VM becomes unresponsive but this time I leave it as I can see data / files increasing in the bucket
After a few hours, the VM recovers and the RAM usage seems to settle around 60% with 32 tasks running.
I can see that data is being sent from the VM at around 2-3 Mbps
After 24 hours, ~285GB data / ~500k files copied.
Running tasks have now dropped to 25
RAM usage at 20%, CPU 40%
I can see that some user objects have completed ok, whilst most are still moving and some have errors:

The errors are appearing on mailbox and sharepoint objects
E.g.
"Failed to move mailbox: <user>. Client disconnected before response was ready. The remote server returned an error:(499) status code 499."

Case # 06272983 opened

Recommendation is to run Move-VBOEntityData PowerShell cmdlet on a single user

I will let the script run for now to see what happens next. Although it looks like it will need many more days at the current rate.

I'm not sure why new move user tasks are not starting now that the running number is 25.

I hope someone finds this information useful.

I have to conclude - like others here - that this not a viable process at the moment.

[Polina]

Hi Chris,

Thank you for sharing this. The results are indeed useful to know and would be interesting for Veeam RnD.
At the moment we're working on an alternative solution that hopefully will allow for more efficient data migrations.

[andre.simard]

Hi Polina,

We have start a move from a jet-database to object storage 3 weeks ago and it's still transferring. depo is about 12TB we have used the Script provided in the kb3067.

The transfer is extremely slow: 57min for a SharePoint site of 286MB (184 Items). at that rate we still have for about another month before everything was transfer.

Do you have any recommendation to increase the speed?

the source proxy as 8 vcpu and 128GB of RAM. I found that the sweets spot is 4 transfer at the time because more that that, transfer as small as 286MB can took 4-5hours to complete and ram increases to 100% and move job failing with out of memory error.

since the job as to be keep disable during the transfer we will be like 2 months without backup and I don't feel safe with that.

What will happend if i decide to stop the move and start the jobs whit on the new object repo?

Case 06296596