Maintain control of your Microsoft 365 data
Post Reply
ScottyP89
Service Provider
Posts: 19
Liked: 4 times
Joined: Jan 02, 2019 4:13 pm
Full Name: Scott Prudence
Contact:

Changing datastores from item-level retention to snapshot-based retention

Post by ScottyP89 » 3 people like this post

Hi All,

I've put the script below together and Polina asked me to post it here. This will migrate all data from a V2 item-level retention datastore to a V3 snapshot-based retention datastore. We sell our backup packages with x months retention so this was a big thing for the company I work for.

The script will disable any jobs going to the existing repository, migrate user data (mailboxes, OneDrive, sites and archives), then it will migrate site data (SharePoint sites) and finally any group data (mailboxes, archives, OneDrive, sites, group mailboxes and group sites) to the new repository that you've created and specify in the script. Once the data has migrated, the backup jobs will be updated to go to the new repository and then the jobs will be re-enabled. You can then manually remove the old repository and the actual data location where the repository exists. I didn't add this to the script because I wanted to run a backup job to the new repository before deleting the old one, just being cautious :)

From my experience, there are a few small caveats:
  • The migration is a copy, so ensure you have plenty of free disk space before doing this
  • You will likely see a lot of warnings, but all of mine were just where user's didn't have archive mailboxes and the warning stated it couldn't move the archive because it didn't exist

Code: Select all

param($OldRepository,$NewRepository)

# Migrate data to new repository

if ($OldRepository -eq $NULL) {
	$from = Read-Host "Please type the full name of the old repository"
} else {
	$from = $OldRepository
}
if ($NewRepository -eq $NULL) {
	$to = Read-Host "Please type the full name of the new repository"
} else {
	$to = $NewRepository
}

# Disabling backup jobs

Write-Host "Disabling any jobs going to the repository $from" -ForegroundColor Yellow
$jobs = Get-VBOJob | Where-Object {$_.Repository -like "$from"}
$jobs | Disable-VBOJob

# Migrating data

$repository = Get-VBORepository -Name $from
$destination = Get-VBORepository -Name $to
Write-Host "Starting to migrate user data to the repository $to" -ForegroundColor Yellow
foreach ($user in (Get-VBOEntityData -Type User -Repository $repository)) {
	Move-VBOEntityData -From $repository -To $destination -User $user -Mailbox -ArchiveMailbox -OneDrive -Confirm:$false
}
Write-Host "Starting to move SharePoint data to the repository $to" -ForegroundColor Yellow
foreach ($site in (Get-VBOEntityData -Type Site -Repository $repository)) {
	Move-VBOEntityData -From $repository -To $destination -Site $site -Confirm:$false
}
Write-Host "Starting to migrate groups data to the repository $to" -ForegroundColor Yellow
foreach ($group in (Get-VBOEntityData -Type Group -Repository $repository)) {
	Move-VBOEntityData -From $repository -To $destination -Group $group -Mailbox -ArchiveMailbox -OneDrive -Sites -GroupMailbox -GroupSite -Confirm:$false
}

# Changing jobs to the new repository

Write-Host "Updating the jobs to backup to the repository $to" -ForegroundColor Yellow
foreach ($job in $jobs) {
	Set-VBOJob -Job $job -Repository (Get-VBORepository -Name $to)
}

# Enabling jobs

Write-Host "Re-enabling backup jobs" -ForegroundColor Yellow
$jobs | Enable-VBOJob
Write-Host "Migration completed" -ForegroundColor Green
There are two ways you can use this script:
  1. Save the script to a PS1 file, for example MigrateDatastore.ps1, then from within PowerShell run

    Code: Select all

    C:\somepath\MigrateDatastore.ps1 -OldRepository Old Repo -NewRepository New Repo
    changing Old Repo for the name of the existing datastore, then change New Repo for the name of your new datastore that's already configured for snapshot-based retention.
  2. Run the script itself, it will then ask you type in the name of the repository you are migrating data from, and then it will ask you for the name of the repository you are migrating data to.
Hopefully some of you will find this useful!
Mike Resseler
Product Manager
Posts: 8191
Liked: 1322 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: Changing datastores from item-level retention to snapshot-based retention

Post by Mike Resseler »

This is great stuff ScottyP89!

Thanks for this, many people her are going to be very happy with it!

cheers
Mike
jandrewartha
Enthusiast
Posts: 34
Liked: 6 times
Joined: Feb 13, 2017 1:49 am
Contact:

Re: Changing datastores from item-level retention to snapshot-based retention

Post by jandrewartha »

So I've used this script, it took several months for about 9TB of data (at least it gracefully restarted after monthly patch reboots). I have a few questions though:
  1. The documentation for Move-VBOEntityData says "This cmdlet does not support migration of organization entities data between repositories with different types of retention policy." but it worked so ?
  2. The old repository is still taking up all the space, although when I run the command again it completes basically instantly. Does it not shrink the database after removing the data?
  3. Relatedly, when I go to remove the repository from the manager, it says I may lose access to old restore points. Is there a way I can check if all the old restore points have been moved across? Or can I remove it, check the restore points, and then re-add it if not?
Steve-nIP
Service Provider
Posts: 129
Liked: 59 times
Joined: Feb 06, 2018 10:08 am
Full Name: Steve
Contact:

Re: Changing datastores from item-level retention to snapshot-based retention

Post by Steve-nIP »

"several months"..

Why on earth is it so slow? I've just started it myself, and I can see that it's going to take ages..
Steve-nIP
Service Provider
Posts: 129
Liked: 59 times
Joined: Feb 06, 2018 10:08 am
Full Name: Steve
Contact:

Re: Changing datastores from item-level retention to snapshot-based retention

Post by Steve-nIP »

I would like to add, however, that I am certainly grateful for such a script - it's a shame it wasn't an in-built function from day #1 in v3
Polina
Veeam Software
Posts: 3195
Liked: 774 times
Joined: Oct 21, 2011 11:22 am
Full Name: Polina Vasileva
Contact:

Re: Changing datastores from item-level retention to snapshot-based retention

Post by Polina » 1 person likes this post

Following up on the questions from @jandrewartha:

1) The documentation is misleading and I greatly appreciate that you brought this to our attention, it'll be fixed ASAP.
This cmdlet is designed to help with moving the backup data between different repositories regardless of their retention settings. There are some specifics for repositories configured with different retention policies: Data is moved to the target repository preserving the source repository retention policy settings. If the retention period on a source is longer than retention on a target, the irrelevant data will be cleared up on the next retention policy job run. But if you want to save this data, you can change the target repository settings after the move.

2) The move operation doesn't shrink the database size on disk. It frees up space inside the database so that it can be reused.

3) Restore points are preserved after the move. They are accessible from the organization's node.


And also, by default, data move is performed in a single thread, which is why you may see issues with the speed. To make it faster with multithreading, add the RunAsync parameter (note that it'll start a separate job per each user/group/site, so try not to overload your proxy).
Mohammed Sohail
Service Provider
Posts: 36
Liked: 3 times
Joined: Nov 25, 2017 2:43 am
Full Name: Mohammed Sohail
Contact:

Re: Changing datastores from item-level retention to snapshot-based retention

Post by Mohammed Sohail »

ScottyP89 wrote: Apr 18, 2019 2:52 pm Hi All,

I've put the script below together and Polina asked me to post it here. This will migrate all data from a V2 item-level retention datastore to a V3 snapshot-based retention datastore. We sell our backup packages with x months retention so this was a big thing for the company I work for.

The script will disable any jobs going to the existing repository, migrate user data (mailboxes, OneDrive, sites and archives), then it will migrate site data (SharePoint sites) and finally any group data (mailboxes, archives, OneDrive, sites, group mailboxes and group sites) to the new repository that you've created and specify in the script. Once the data has migrated, the backup jobs will be updated to go to the new repository and then the jobs will be re-enabled. You can then manually remove the old repository and the actual data location where the repository exists. I didn't add this to the script because I wanted to run a backup job to the new repository before deleting the old one, just being cautious :)

From my experience, there are a few small caveats:
  • The migration is a copy, so ensure you have plenty of free disk space before doing this
  • You will likely see a lot of warnings, but all of mine were just where user's didn't have archive mailboxes and the warning stated it couldn't move the archive because it didn't exist

Code: Select all

param($OldRepository,$NewRepository)

# Migrate data to new repository

if ($OldRepository -eq $NULL) {
	$from = Read-Host "Please type the full name of the old repository"
} else {
	$from = $OldRepository
}
if ($NewRepository -eq $NULL) {
	$to = Read-Host "Please type the full name of the new repository"
} else {
	$to = $NewRepository
}

# Disabling backup jobs

Write-Host "Disabling any jobs going to the repository $from" -ForegroundColor Yellow
$jobs = Get-VBOJob | Where-Object {$_.Repository -like "$from"}
$jobs | Disable-VBOJob

# Migrating data

$repository = Get-VBORepository -Name $from
$destination = Get-VBORepository -Name $to
Write-Host "Starting to migrate user data to the repository $to" -ForegroundColor Yellow
foreach ($user in (Get-VBOEntityData -Type User -Repository $repository)) {
	Move-VBOEntityData -From $repository -To $destination -User $user -Mailbox -ArchiveMailbox -OneDrive -Confirm:$false
}
Write-Host "Starting to move SharePoint data to the repository $to" -ForegroundColor Yellow
foreach ($site in (Get-VBOEntityData -Type Site -Repository $repository)) {
	Move-VBOEntityData -From $repository -To $destination -Site $site -Confirm:$false
}
Write-Host "Starting to migrate groups data to the repository $to" -ForegroundColor Yellow
foreach ($group in (Get-VBOEntityData -Type Group -Repository $repository)) {
	Move-VBOEntityData -From $repository -To $destination -Group $group -Mailbox -ArchiveMailbox -OneDrive -Sites -GroupMailbox -GroupSite -Confirm:$false
}

# Changing jobs to the new repository

Write-Host "Updating the jobs to backup to the repository $to" -ForegroundColor Yellow
foreach ($job in $jobs) {
	Set-VBOJob -Job $job -Repository (Get-VBORepository -Name $to)
}

# Enabling jobs

Write-Host "Re-enabling backup jobs" -ForegroundColor Yellow
$jobs | Enable-VBOJob
Write-Host "Migration completed" -ForegroundColor Green
There are two ways you can use this script:
  1. Save the script to a PS1 file, for example MigrateDatastore.ps1, then from within PowerShell run

    Code: Select all

    C:\somepath\MigrateDatastore.ps1 -OldRepository Old Repo -NewRepository New Repo
    changing Old Repo for the name of the existing datastore, then change New Repo for the name of your new datastore that's already configured for snapshot-based retention.
  2. Run the script itself, it will then ask you type in the name of the repository you are migrating data from, and then it will ask you for the name of the repository you are migrating data to.
Hopefully some of you will find this useful!
Hi Scott, Great Job. i was half thru writing this code myself to migrate data to the new Object Repository until i saw your post. you saved me the troubleshooting efforts :).
I agree with Polina to add RunAsync to speed the performance on Move-VBOEntityData Command.

Thank you.
DaStivi
Service Provider
Posts: 295
Liked: 46 times
Joined: Jun 30, 2015 9:13 am
Full Name: Stephan Lang
Location: Austria
Contact:

Re: Changing datastores from item-level retention to snapshot-based retention

Post by DaStivi »

Does this still work on V5 ?
Polina
Veeam Software
Posts: 3195
Liked: 774 times
Joined: Oct 21, 2011 11:22 am
Full Name: Polina Vasileva
Contact:

Re: Changing datastores from item-level retention to snapshot-based retention

Post by Polina »

Hi Stephan,

I didn't have a chance to test it with v5, but since there haven't been any changes to the cmdlets I believe it should work out of the box.
JasonLCL
Lurker
Posts: 1
Liked: never
Joined: Jun 13, 2021 8:38 pm
Contact:

Re: Changing datastores from item-level retention to snapshot-based retention

Post by JasonLCL »

Hi

I am looking to convert our original datastore to snapshots and have found this script, currently around 70 users and 2.4Tb in the repository. In the replies above there is mention of using the runasysc parameter to speed this process up. To do this I just modify the script at each move-vboentitydata command and add it to the end, like this?
Move-VBOEntityData -From $repository -To $destination -User $user -Mailbox -ArchiveMailbox -OneDrive -Confirm:$false -RunAsync

there was mention of not overloading the proxy, I guess by having too many threads running, is there a way to limit how many threads this command will start? Looking at the powershell reference page for this command I couldn't see a way to limit it.

Also can this script be stopped at will and then started? will it continue on from where it left off?

Can the existing backups be set to use the new repository and continue to run while this script is in operation, or will we have to run with no backups while the script is doing its thing?

Sorry for all the questions, but I don't want to learn the hard way that the only stupid question is the one not asked ;-)
Polina
Veeam Software
Posts: 3195
Liked: 774 times
Joined: Oct 21, 2011 11:22 am
Full Name: Polina Vasileva
Contact:

Re: Changing datastores from item-level retention to snapshot-based retention

Post by Polina »

Hi,

I'd suggest using the official script from the KB3067. The KB is focused on migrating to an object storage repository, but you can safely use it for moving backup data between local repositories as well. It already includes the RunAsync parameter and limits threads allocation.

During data migration, backup jobs must be stopped to avoid any potential issues.

Hope it helps!
rura_at_tivoli
Influencer
Posts: 11
Liked: 1 time
Joined: Oct 09, 2018 11:55 am
Contact:

Re: Changing datastores from item-level retention to snapshot-based retention

Post by rura_at_tivoli »

Hi

I'm in this situation now having to switch from item-level to snapshot-based for our mailboxes.
Current item-level repository is set to "Keep forever" retention and the new snapshot-based will be set to 5 year retention.

What happens to mailboxes that are not actively backed up (because they were deleted in the cloud) after moving to the new repository?
Does the metadata contain a mailbox deletion date to "simulate" a last snapshot?
Does it just keep the entire mailbox for 5 years?
or will each item in the mailbox be evaluated individually based on creation/modification date?
admd
Service Provider
Posts: 10
Liked: 1 time
Joined: Feb 22, 2024 1:37 pm
Contact:

Re: Changing datastores from item-level retention to snapshot-based retention

Post by admd »

I am also interested in an answer for the questions that rura_at_tivoli asked.
Post Reply

Who is online

Users browsing this forum: No registered users and 6 guests