Migrate backup data to Object Repository

[toellbe]

Why is there no way to migrate to object storage that doesn't end in frustration?

I used the script from https://www.veeam.com/kb3067. But not one move is successfully completed.

Either the job stops or the database reports errors like "outofsessions"... It is not even possible to restart the jobs ...

Can it be expected that there will be a way that works in the future?

700 User 3,8TB...
8 CPUs, 32GB RAM, 40 Threads.

[nielsengelen]

Hi, could you contact support for insight on this and see what is causing this? We’ve had several reports from customers who used the script without issues so it may be something else is going on.

Did u by any chance tried to move just 1 entity as a test to see if just one works?

[toellbe]

Today, two 2.2 GB mailboxes were completed after 27 hours of operation.

In these 27 hours, the Veeam Archiver Service and the migration script had to be restarted three times. In all three cases, the error "JET_errOutOfSessions-1101 "reported.

With 20 simultaneous migrations I have a memory usage of 28 / 32GB. An average performance of 150 Mbit/s for the first 6 hours. Now max 30 Mbit/s - average 3 Mbit/s. Azure VM HDD Repository -> Blob Cold.

The migration is so incredibly slow and needs to be actively monitored because there can always be any error.

But no problem. I just schedule 24/7 shifts that look at it every hour. Can only be a few days. Or weeks. /irony off

[nielsengelen]

As advised, please open a support case for insight to see what is causing this and let us know the support case ID.

[toellbe]

Case#: 03928665

Veeam Support says the script is for demonstration only and not supported...

As suggested by the support, the max cursors were set to 6000. After about 2 hours we get the jet error outofsessions again:
<Storage> <Params MaxCursors="6000"/> </Storage>
in :\ProgramData\Veeam\Backup365\Proxy.xml

It remains unclear what triggers the outofsession error or what the maximum load for the jetdb is.

[Polina]

Hi toellbe,

It's a common practice to provide PowerShell scripts as a reference/demonstration resource only because all the customers' environments are very different and it's hardly possible to deliver a swiss-army knife that fits any configuration.

Your case will be escalated to RnD for further investigation, so please keep in touch with our support team.

Thanks!

[gbrunner]

Hallo,
I have the same problems (outofsessions Error, decreased performance after some hours).
Are any insights to get a stable and performing migration process?

Best Regards
Georg

[nielsengelen]

Georg, please contact support for insight and help. If possible, please share the support case ID for future reference.

[toellbe]

...
I have the same problems (outofsessions Error, decreased performance after some hours).
Are any insights to get a stable and performing migration process?
...

After a few weeks for 3.8 TB and several dozen problems, the migration is now complete.

After every error I restarted the veeam services and than the powershell script.

I canceled the support case after not feeling that things were going in the right direction.


Good luck to you.

[Polina]

Hi toellbe,

I'm disappointed to know that you're not satisfied with the support provided for your case. Unfortunately, you requested to close the case the day it has been planned to escalate for further investigation.

4.0 is a relatively new product version, and to isolate and troubleshoot the new issues it may take time and more efforts from the Support and RnD teams.

[mattythefatty]

I've been having the same issue, attempting to migrate about 6.7 terabytes. First I was using half the number of threads available on the proxy. I kept walking this number back but continue to have the out of sessions error. My backend S3 repository is Wasabi, which looking at TCP connections appeared to be limited to 5 sessions. Putting the value to 5 also didn't resolve the issue. Because the customer has beefy ESXi hosts, I've increased the RAM on the VM to 100GB's and restarted the process with 32 sessions. Memory utilization seems to have flattened out at 38GBs. I'll update this post if this change results in positive results.

[mattythefatty]

So far so good. The migration is still progressing without throwing the OutofSessions error. It is currently consuming 67GB of RAM.

[mattythefatty]

RAM utilization may be the answer to this error. Since upgrading from 20 to 100GB of ram, this has not failed yet. I'll be monitoring over the weekend, and I'll continue to post the results. Here is the script I'm using to move the data

$org = Get-VBOOrganization
$users = Get-VBOOrganizationUser -Organization $org
$source = Get-VBORepository -Name "Local---E_Drive"
$target = Get-VBORepository -Name "S3 Backed Storage Repository"
$proxies=Get-VBOProxy
$jobs_for_move=32

foreach
($user in $users)
{
$data = Get-VBOEntityData -type user -repository $source -name $user.displayname
Move-VBOEntityData -From $source -To $target -User $data -Mailbox -ArchiveMailbox -runasync -Verbose -Confirm:$false -ErrorAction SilentlyContinue
while((Get-VBODataManagementSession | ? {$_.Status -eq "Running"}).Count -ge $jobs_for_move){sleep 30}

}

[Polina]

Matthew,

Our RnD team has investigated the OutOfSessions issue recently, and I believe Support can now provide a fix. Please open a support call and share its ID in this thread for the followup.

Thanks!

[mattythefatty]

Polina,

I did submit a case over the weekend, since the problem continues. Case # 03994002. I've provided the Veeam logs, windows event logs, and system info. No response yet.

Matt

[Polina]

Matt,

Your case is now on the radar; engineers need some time to confirm the root cause and will get back to you shortly.

Thanks!

[mattythefatty]

Here are the steps requested by Veeam support. I've made the changes and restarted the migration script.

1. Ensure that no Backup Job is running, then stop "Veeam Backup Proxy for Microsoft Office 365" service.
2. Navigate to the C:\ProgramData\Veeam\Backup365 folder on the VBO Controller/Remote Proxy server that works with the database in question.
3. Open the Proxy.xml file and add a new node and the following parameter:
(Node <Storage> should be placed inside node <Archiver> of the Proxy.xml document. The default value is 2000. Start from increasing the number of cursors by two, making it 4000.)
<Storage>
<Params MaxCursors="4000"/>
</Storage>
4. Start "Veeam Backup Proxy for Microsoft Office 365" service.
5. Try migrating again

This may need to be repeated a few times. Go ahead and try increasing the cursors by 2000 each time until you reach 8000 if the migration doesn't succeed. Let me know how things go or if you have questions.

[Polina]

Matt,

It should be some miscommunication on our end; checking with the support team on why the fix wasn't provided to you straight away.

[mattythefatty]

Polina,

Support sent me another hotfix option, which i won't try if/when the current process fails. Likely, this is what you were referring to?

[Polina]

Right, this is the one I was thinking of initially. And, whatever helps you to resolve the issue, please take a minute to share the news here.

Thanks!

[mattythefatty]

Update: After the hotfix above, the process would fail in about 15 minutes rather than 15 hours. I escalated the issue again. This time, support provided me another hotfix https://www.veeam.com/kb3065, which brings me from version 4.0.0.2516. (dont install this if your also using the same VM for Backup and Replication) I also utilized the provided powershell script, which I had found on some 3rd party website previously and where I got the commands I was using today. https://www.veeam.com/kb3067, there is a link at the bottom to download the script. Very easy to run. So far my migration is running smooth and using only 20GB of RAM instead of 60GB.

I'll check it once in the evening and again in the morning and hopefully update this thread.

Matt

[mattythefatty]

Bad news.

Enable-VBOJob : JetError -1101, JET_errOutOfSessions, Out of sessions
At C:\Users\mhunter\Desktop\VeeamKB3067.ps1:65 char:5
+ Enable-VBOJob -Job $job
+ ~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Enable-VBOJob], FaultException
+ FullyQualifiedErrorId : System.ServiceModel.FaultException,Veeam.Archiver.PowerShell.Cmdlets.Jobs.EnableVBOJob

The saga continues.........

[mattythefatty]

Good news!

Working with support, they provided me another patch for the proxy, bringing it to version 4.0.0.2526. The process has been running since Saturday without failure.
Matt

[Polina]

Hi Matt,

First, thank you for collaborating with our engineers. I was in touch with Support on your case and it seems that hotfixes are delivered by RnD faster than a word spreads between different teams
If possible, I'd be interested to know how long it took you to migrate your 6.7TB of data (after applying the hotfix).

[mattythefatty]

Well, i thought things were going well, but after checking again, it appears the hotfixes just remove the errors. It appears to be running, but the disk i/o and network activity is so low, im not making any progress. I'm hoping to get my support ticket re-opened. I've got a single user that has been running since the 15th (142 hours). I've now not had a backup since 1/31.

My opinion on the process? It sucks, and If i had to do it over again, I wouldn't. Unless your customer has a very small amount of data, just build yourself a blank S3 backed repository and start backing up there. Then, delete the local data after you feel warm and fuzzy about it..

Matt

[mattythefatty]

I figured I'd give another update on this miserable process. I've continued to work with support and continue to break things worse than they already were. Each change or upgrade results in a worse situation. My customer hasn't had a backup since the end of January and I'm stuck in the middle of having about half the data locally and half in S3 storage.

DO NOT attempt this migration unless you are a glutton for punishment.

Matt

[rmehta]

Just wanted to check if anyone has recently tried this process and is there a better way to send data to object storage. at the moment, the script reads items at a time and one mailbox with lots of items (1242104 items processed) taking over 26 hours just to transfer 40GB of data. The bottleneck showing source however there are no logs and I am not sure sure if that is expected with these many objects. If this process takes this long it would take month just to migrate a small customer (2TB).

The approach we are planning to take is
- if we stop the backup job; take a backup of the disk repository
- copy it to a USB or to an onsite server with attached disk and hand it over to the customer
- Create a new repo with Object storag and and ask customer to do new full backup.

Or if there is any step to copy this data directly to object storage bucket (using object explorer) and import this location while defining the new repo so that the full does not transfer all the data again.

[Mike Resseler]

There is work being done on this process. But I have no ETA or more information at this point in time. We will also verify if certain operations "outside" of VB365 (like copying object storage data with an external tool to another object storage location) can be supported (after the cache has been rebuild).

[squebel]

We are starting a project to move our tenant VB365 repo data over to object and so far have done two small customers. One tenant was about 800GB of data and 125 users and it took nearly 24 hours. The other was much smaller and seemed to run faster. However, each job resulted in the script not actually stopping itself for some reason. And on the 2nd tenant, the job reenabled itself automatically somewhere in the middle of the migration. I have no idea how that happened.

Is this migration script and commands really that un-polished at this point? We have some large customers we want to do this with and need to know if we should continue or not. The last post in this thread definitely gives me pause.

[Polina]

Hi,

The script from the KB was created almost 2 years ago and is provided as an example. Re-enabling might be some retry in case of connection issues, but I'll check it to be sure.

May I ask if the 800GB that you were moving included a mix of Exchange, SharePoint, and Teams data? And what product version are you currently on?