Maintain control of your Microsoft 365 data
Post Reply
pr_osit
Service Provider
Posts: 75
Liked: 12 times
Joined: Jan 02, 2024 9:13 am
Full Name: Pat
Contact:

Veeam 365 migrate from object storage to different object storage

Post by pr_osit »

Hey folks,

I've been given a task to migrate a significant amount of data (300TB+) from Azure Blob to Wasabi and have been looking into the best way to go about this.

This should be pretty straight forward, but the issue is that there are several clients/tenants/jobs backing up to a single Azure Blob repository, and I need to separate this into separate Wasabi repositories.
I feel like this is going to be pretty manual, however my plan is to try and identify unique each client/job and then migrate it to an a new unique bucket and recreate the job.
First question, do you think this will work?

I know the object to object migration feature has been requested for a long time, and is in the works, but I don't think it's going to be ready any time soon and I need to do this now.

So far the issue I have is that there are a large number of clients/jobs in the single repository, and each client/job is separated by a unique ID in the repo, but I have been struggling to match some of these with the client jobs in the Veeam 365 console. I have been able to identify a chunk of these as they are currently active jobs and the IDs appear in the Veeam logs, some are inactive jobs/archive jobs no longer running, so aren't in current logs, and some of it may be data for old jobs that no longer exist and doesn't need to be migrated. Worst case scenario I would need a way to work out which data belongs to active jobs, and which does not, so I can delete the old data and move the lot of it to a single wasabi repo and try and sort it out from there in the future.

I haven't found a way to do this in bulk, or obtain these specific IDs for jobs/clients via powershell. Does anyone have any ideas on how to do this?

Example:
<bucket>.blob.core.windows.net/veeam365backuprepo/Veeam/Backup365/Clients/Organizations/12345678123456781234567812345678/Mailboxes/

I need to try and find an easier way of matching these IDs to clients and jobs so I can work out what to move. I have a test tenant I'm going to start testing with, but just trying to plan it out currently.

Second question, we have split clients into multiple jobs for mailbox/onedrive/sharepoint and so I will need to migrate each type of data separately, I believe it's just separated by folders so moving the right folders should match the right job type. I'm hoping there isn't too much more to it than this. Is it going to be as simple as putting the right data in the right buckets and recreating the job on the other end? I don't think this is really supported by Veeam so I'm not expecting a migration guide, but wondering if anyone has had to do this before or has any suggestions?

Obviously need to do a lot of testing, but trying to avoid too much in the way of egress fees from Azure while testing the process so trying to plan it out as much as I can first. Hopefully I've explained what I'm trying to achieve clearly enough, if not I can try to elaborate further. Looking forward to any advice or suggestions the community or Veeam team may have, thanks.
pr_osit
Service Provider
Posts: 75
Liked: 12 times
Joined: Jan 02, 2024 9:13 am
Full Name: Pat
Contact:

Re: Veeam 365 migrate from object storage to different object storage

Post by pr_osit »

well I spent some time today working out how to match the jobs in veeam up to the folder structure in the object storage, came up with a script that goes through all of the log files and pulls out the data
i'm not a developer so this isn't pretty, and might have some errors but it seems to work for me.

unfortunately has left me with more questions than answers about what needs to migrate :D

after iterating through the logs for each client, and exporting any paths to the object storage, different clients seem to store their data in the same folders.
I was kind of assuming that each folder would be unique to each client, or at least each job, but this doesn't seem to be the case, so I don't really understand how the data is being stored
still open to any suggestions on how to do this as it's not as simple as I was thinking

anyone able to elaborate or point me in the right direction for further research/testing?

Code: Select all

function Search-LogFileForBlobPath {
    [CmdletBinding()]
    param (
        [string]$LogDirectoryPath,
        [string]$BlobPathPattern
    )

    # Initialize a set to store unique matches
    $uniqueMatches = New-Object System.Collections.Generic.HashSet[string]

    # Construct the regex pattern dynamically using the blobPathPattern
    $regexPattern = [regex]::Escape($BlobPathPattern) + "(?:\/[^\/,]+){0,3}"
    $regex = [regex]::new($regexPattern)

    # Excluded pattern
    $excludedPattern = "$BlobPathPattern/RepositoryLock/"

    # Get all log files in the specified directory
    $logFiles = Get-ChildItem -Path $LogDirectoryPath -Filter "*.log" -File

    # Iterate through each log file
    foreach ($logFile in $logFiles) {
        # Read the file line-by-line
        Get-Content $logFile.FullName | ForEach-Object {
            $line = $_

            # Check if the line matches the search pattern
            if ($regex.IsMatch($line)) {
                $match = $regex.Match($line).Value

                # Check if the match starts with the excluded pattern
                if (-not $match.StartsWith($excludedPattern)) {
                    $uniqueMatches.Add($match) | Out-Null
                }
            }
        }
    }

    # Sort the unique matches alphabetically
    $sortedMatches = $uniqueMatches | Sort-Object

    # Output sorted unique matches
    $sortedMatches
}

# Define the log directory path
$logDirectoryPath = "C:\ProgramData\Veeam\Backup365\Logs\"

# Define the blob path pattern (this is the path to your veeam365 data in your object storage)
$blobPathPattern = "Veeam/Backup365/Clients"

# Initialize an array to store the results
$results = @()

# Get all subdirectories (orgs) in the log directory
$orgDirectories = Get-ChildItem -Path $logDirectoryPath -Directory

# Iterate through each org directory
foreach ($orgDirectory in $orgDirectories) {
    # Output org name
    $orgName = $orgDirectory.Name
    Write-Host "Org Name: $orgName"

    # Get all subdirectories (job names) in the org directory
    $jobDirectories = Get-ChildItem -Path $orgDirectory.FullName -Directory

    # Iterate through each job directory
    foreach ($jobDirectory in $jobDirectories) {
        # Output job name
        $jobName = $jobDirectory.Name
        Write-Host "Job Name: $jobName"

        # Search for unique paths within the job directory
        $uniquePaths = Search-LogFileForBlobPath -LogDirectoryPath $jobDirectory.FullName -BlobPathPattern $blobPathPattern

        # Check if any paths were found
        if ($uniquePaths.Count -eq 0) {
            Write-Host "No matching paths found in $jobDirectory."
        } else {
            # Output unique paths found
            Write-Host "Paths Found:"
            $uniquePaths | ForEach-Object {
                Write-Host $_
                # Add the result to the results array
                $results += [PSCustomObject]@{
                    "OrgName"   = $orgName
                    "JobName"   = $jobName
                    "BlobPath"  = $_
                }
            }
        }
    }
}

# Output the results
$results | Format-Table -AutoSize

# Define the output CSV file path
$outputDirectory = "C:\Users\<username>\Desktop\" #where you want to save the logs to
$hostname = $env:COMPUTERNAME
$outputCSVPath = Join-Path -Path $outputDirectory -ChildPath "$hostname-object-storage-list-v2.csv"

# Export the results to a CSV file
$results | Export-Csv -Path $outputCSVPath -NoTypeInformation
pr_osit
Service Provider
Posts: 75
Liked: 12 times
Joined: Jan 02, 2024 9:13 am
Full Name: Pat
Contact:

Re: Veeam 365 migrate from object storage to different object storage

Post by pr_osit »

thinking out loud here but would a 365 copy job work as a workaround method of migrating the data with less manual intervention?
pr_osit
Service Provider
Posts: 75
Liked: 12 times
Joined: Jan 02, 2024 9:13 am
Full Name: Pat
Contact:

Re: Veeam 365 migrate from object storage to different object storage

Post by pr_osit »

Just updating this, I have figured out a way to do this and appears to work successfully, however more testing is required before I move any clients data. I've managed to migrate an org with 4 years of retention out of the shared repository to a new repository, and tested new backup jobs run, and restores work and all retention appears to be available. I can't test every file in every restore point but I feel pretty confident so far. The problem is it's a slow and manual process, and to be worthwhile doing I probably need to find a way to speed things up.
Mildur
Product Manager
Posts: 10110
Liked: 2696 times
Joined: May 13, 2017 4:51 pm
Full Name: Fabian K.
Location: Switzerland
Contact:

Re: Veeam 365 migrate from object storage to different object storage

Post by Mildur » 1 person likes this post

Hi Pat
I know the object to object migration feature has been requested for a long time, and is in the works, but I don't think it's going to be ready any time soon and I need to do this now.
Correct, we are working on options to migrate backup from object storage to object storage repositories.
But it will come in an version after v8. No ETA yet.
thinking out loud here but would a 365 copy job work as a workaround method of migrating the data with less manual intervention?
The Data format between backup and backup copy jobs are not compatible to each other. You would not be able to map a backup job to the backup copy backups.
Something else to be aware off. A copy job will not copy "everything" after it's creation. It will start copying backups from the most recent restore point. Historical data before the copy job was created will not be copied.
so I don't really understand how the data is being stored
still open to any suggestions on how to do this as it's not as simple as I was thinking
Our data structure is documented in our user guide: https://helpcenter.veeam.com/docs/vbo36 ... tml?ver=70

Maybe that can help you to determine which folders are required.
Please note, your approach is completely unsupported. Our customer support team won't be able to help if this leads to corruption of the backups or metadata.

Best,
Fabian

Best,
Fabian
Product Management Analyst @ Veeam Software
pr_osit
Service Provider
Posts: 75
Liked: 12 times
Joined: Jan 02, 2024 9:13 am
Full Name: Pat
Contact:

Re: Veeam 365 migrate from object storage to different object storage

Post by pr_osit »

Hi Fabian,

Thanks for your reply. I have already tested and ruled out the copy job, it's fine to copy a single restore point, but we need to copy several years of retention.

One of the things that has been confusing me is that I find many references in the logs for a single job referencing many different 'organization ID' type folders, outside of what seems to be it's own organization ID.
The only way I've been able to correlate this data is by reading the Veeam 365 log files. Is there any other way I can find this info?

A lot of the time it looks something like this, it seems when the jobs run this is removing resources from other locations (other clients). I haven't been able to work out what's going on here, any ideas? My first guess was some kinda dedup but really I have no idea.

104 (16436) Removing resource (memory cache: BlobCache, cache size: 2147473288, key: f2323e2b-5e65-4903-a005-7fd150c93b88Veeam/Backup365/Data/Organizations/dbd24518d5b24bdf87779813b2df04c9/WebData/9f6a7174b95d4106a4b9b4a4dc9756c2/32b5dd300434415c86ee354bc0d0360c/7614d51aa54b453f806550fe762e8599.00000045, resource size: 23664)...

Thanks
aeph
Enthusiast
Posts: 62
Liked: 4 times
Joined: Sep 26, 2024 11:02 am
Contact:

Re: Veeam 365 migrate from object storage to different object storage

Post by aeph »

@pr_osit i am very interested here.

How did it go from here? Were you successful in migrating the data from Azure to Wasabi? We are planning to migrate a customer from Azure Blob Storage to Wasabi as well. Can we load the restore points if we migrate the data from Azure to Wasabi (is it safe to just move the "folders" from A to B or is it necessary to adapt something on the structure as well?).
pr_osit
Service Provider
Posts: 75
Liked: 12 times
Joined: Jan 02, 2024 9:13 am
Full Name: Pat
Contact:

Re: Veeam 365 migrate from object storage to different object storage

Post by pr_osit »

@aeph

if you just have 1 client in a single repository, it should be fairly straight forward to just move the data. this may give you a start:
https://benyoung.blog/blog/migrate-veea ... r-cluster/

for us, the complication was having hundreds of clients in a single azure repository/blob and wanting to move them to individual wasabi repos. I did find a way to do it but was quite tedious so I ended up just reseeding a lot of them to wasabi and archiving the azure blob until retention expires.
Mildur
Product Manager
Posts: 10110
Liked: 2696 times
Joined: May 13, 2017 4:51 pm
Full Name: Fabian K.
Location: Switzerland
Contact:

Re: Veeam 365 migrate from object storage to different object storage

Post by Mildur »

Hi Pat,

Thanks for sharing.
Supported Object Storage to Object Storage migration is still on our roadmap, and we haven't forgotten about it. First, we plan to introduce the new "migration method" to migrate from "Jet DB-based repositories" to "object storage" in our next minor update. This method will allow for migration while running backup jobs.

The migration from Object Storage to Object Storage will be based on that functionality.

Best,
Fabian
Product Management Analyst @ Veeam Software
aeph
Enthusiast
Posts: 62
Liked: 4 times
Joined: Sep 26, 2024 11:02 am
Contact:

Re: Veeam 365 migrate from object storage to different object storage

Post by aeph »

Hi all, thank you.

I will try it out in our test environment with an Azure Blob Storage.

But what I'm wondering: How is the repository and the organization connected? Typically: The repo is not linked directly to the organization, but the repo is linked to the organization via the job. How does the repo then know which organization it belongs to (especially when using a restore portal)? Does it store the respective tenant ID in the repo?
Polina
Veeam Software
Posts: 3308
Liked: 798 times
Joined: Oct 21, 2011 11:22 am
Full Name: Polina Vasileva
Contact:

Re: Veeam 365 migrate from object storage to different object storage

Post by Polina »

Yes, the data ownership is identified by the organization ID. Each repository can store data from multiple organizations.
pr_osit
Service Provider
Posts: 75
Liked: 12 times
Joined: Jan 02, 2024 9:13 am
Full Name: Pat
Contact:

Re: Veeam 365 migrate from object storage to different object storage

Post by pr_osit »

just wanted to update this now that the topic is active again and state that most of the info in the first few posts is inaccurate and I found much simpler and easier ways to do everything and was able to successfully extract single orgs out of the big shared repo and migrate them to new repos (object to object migration for individual orgs in a shared repo) but it was quite a manual process and did not do it in bulk for all of the orgs.
aeph wrote: Oct 18, 2024 5:04 pm But what I'm wondering: How is the repository and the organization connected? Typically: The repo is not linked directly to the organization, but the repo is linked to the organization via the job. How does the repo then know which organization it belongs to (especially when using a restore portal)? Does it store the respective tenant ID in the repo?
there's kind of some high level metadata in the repo then a folder per org with all of the object data within each, if you only have 1 org you can basically just copy paste it all to a new repo and then re-sync it and veeam will rebuild it's cache of that repo (not tested in v8) and this only gets more complicated if there's many orgs in one repo
aeph
Enthusiast
Posts: 62
Liked: 4 times
Joined: Sep 26, 2024 11:02 am
Contact:

Re: Veeam 365 migrate from object storage to different object storage

Post by aeph »

Thank you !

If we migrate the data from one object storage to another, do we have to pay attention to anything when naming, e.g. jobs, orga, repo?
Do they have to be named the same as before?
aeph
Enthusiast
Posts: 62
Liked: 4 times
Joined: Sep 26, 2024 11:02 am
Contact:

Re: Veeam 365 migrate from object storage to different object storage

Post by aeph »

Update:

I have successfully transferred the data from an Azure Blob Storage to another S3 provider using rclone.
The data was not encrypted and did not have Immutability enabled. I was able to load the restore points and everything was fine.

However, what did not work:

I could not set encryption or immutability on this repo afterwards. I can understand encryption, as this would require the existing structure to be encrypted. However, it is not clear to me why I cannot set immutability afterwards.

In my second attempt, I also tried to create a bucket with versioning and object lock and transferred the files there. However, I cannot then integrate the repository because this message appears:
"The selected bucket supports blob versioning.
Select a bucket with disabled blob versioning."

I know this KB (https://www.veeam.com/kb4446).
However, the bucket has both versioning and object lock activated.

Can I subsequently activate immutability for an existing repository without having to create a full backup in a new repo?
Mildur
Product Manager
Posts: 10110
Liked: 2696 times
Joined: May 13, 2017 4:51 pm
Full Name: Fabian K.
Location: Switzerland
Contact:

Re: Veeam 365 migrate from object storage to different object storage

Post by Mildur »

No, immutability can only be enabled when creating the S3 bucket. It's not possible to enable immutability options on the bucket afterwards.
Even if your object storage provider allowed that change somehow, we only support "new" immutable repositories. Repositories with existing non-immutable backups cannot be converted to immutable repositories.

Best regards,
Fabian
Product Management Analyst @ Veeam Software
andre.simard
Service Provider
Posts: 278
Liked: 32 times
Joined: Feb 05, 2016 8:07 pm
Contact:

Re: Veeam 365 migrate from object storage to different object storage

Post by andre.simard »

Hi,

Is there any update on a working or supported way to move all our repo from Azure to Wasabi? we have one repo/bucket per organization.
Data is encrypted but not immutable

thank you
Polina
Veeam Software
Posts: 3308
Liked: 798 times
Joined: Oct 21, 2011 11:22 am
Full Name: Polina Vasileva
Contact:

Re: Veeam 365 migrate from object storage to different object storage

Post by Polina »

Hi Andre,

If my memory serves me well, there were a couple of posts here on forums sharing examples of successful migrations, but officially we do not support any of those. Should something go wrong, our support won't be able to fix it.

At the same time, we are planning to deliver some built-in tool/mechanism but it's not there yet and won't be ready soon.

Thanks!
Post Reply

Who is online

Users browsing this forum: No registered users and 48 guests