Best way to setup local repository and copy job

[sogapex]

Hi,

I am currently changing our old Veeam B&R server. It is also running VBO365.
I installed VBO365 on the new server, create a new repository on the same volume as the one used by B&R (reFS) and let run the first "full" backup. (it took 2 days)
Then, I set up a "copy job" in B&R (in fact, a "files copy job") to make a copy of the VBO files to another server (linux hardened repository)

Problem = after a few days, the copy job is not working = (this is the same for all files)
process can't access file because this file is used by another process. Failed to open file [D:\VBO\2011\adb.log] in readonly mode. File is locked by 1 processes: [6412:Service] Veeam Backup Proxy for Microsoft 365 (Veeam.Archiver.Proxy.exe)

I try to find more information about best practice for VBO and find this "guide" :
https://bp.veeam.com/vb365/guide/buildconfig/

So, basically, "reFS" is not good, I should go with ntfs. (ok, in such case, I can't share the same disk as B&R, I will buy a new disk especially for VBO, not a problem in our case since VBO data is about 600GB now)
But, the guide also tell us to go "object storage" instead of "jet base file storage" => the only problem= there are no explanation how to achieve this with local storage (direct attached disk)

1. what is the best way to setup a VBO backup job to make a "direct attached storage" backup job ?
2. what is the best way to make a backup copy on another "on premise" server ?

Thanks

[Mildur]

Hi Sogapex

With jet db based repositories, you should take a image level backup with Veeam Agent or a VM Backup Job if the VB365 server is a VM. With image level backups, only changed blocks would be copied to the backup repository.
If you do File based backups/copies, your file copy job will always copy all files again. Because they changing all the time.
Disadvantage: You need a lot of backup storage and Copy process will always have the same long duration.

For files based backup jobs, you must stop the VB365 services. Or the database files will be still accessed by VB365.
You can use a Veeam Agent Job with Job Script.

But, the guide also tell us to go "object storage" instead of "jet base file storage" => the only problem= there are no explanation how to achieve this with local storage (direct attached disk)

Object Storage is a new storage technology. You cannot just simply use object storage with normal disks.
There are software based solution. For example you can set up MinIO on a Linux machine which allows you to use your disk as an Object Storage repository. Or just use one of the known cloud provider like AWS, Azure or less expensive ones like Wasabi. There are a lot of them to choose from:
https://www.veeam.com/alliance-partner- ... vbo&page=1

Best,
Fabian

[sogapex]

Hi Fabian,

I am really not fond of using script to "kill" the 365 proxy while making the file copy. And since you said all the files are changing with the "jet engine", I decide to go the "object storage" way.
So, I someway "fought" to set up a new linux VM with minio yesterday (ubuntu 22.04 + minio 2023-03-24) and then I created a new job in Veeam VB365 to make use of this S3 compatible storage object. (Once the minio was running in https, it was a piece of cake on the veeam side)
I let the job run all the night (and stop it this morning) : about 40% of the first (full) "backup" is done.

Once all the mailboxes are saved, I will set up a backup job in Veeam B&R for this VM

Thanks again,
I suppose this is a better way to handle this task locally than going the "jet engine" way (I already got some problems with the jet engine on the old veeam server = memory usage and cpu usage)
Now, I am more confident I can rely on this system for several years (see you in 2030 when changing the Veeam Server again )

[Mildur]

Hi Sogapex

Thank you for the feedback.
Object Storage is our recommended backup repository for Veeam Backup for Microsoft 365. Instead of creating a backup of the VM, you could also use a backup copy job to a second MinIO server.
https://helpcenter.veeam.com/docs/vbo36 ... tml?ver=70

If you can, please share your experience with performance and handling after a few weeks of running your backups.
I also suggest to run a restore test. You could use instant recovery for the MinIO server and VB365 server to test the restore in an isolated network.

Best,
Fabian

[sogapex]

backup copy job to a second MinIO server => I think this is better to push a backup on the "Hardened" repository rather than setting up another "minio" server.
I am even wondering if this is necessary to add the "minio" VM to the "rotated drive" backup job or not...

I pratically never used the old system (I mean : from memory, maybe I tried to restore 2 emails in 3 years)
But I remember it was not great "performance wise" when searching for an email.

So, yes, i will try to restore an email in a few weeks to see how it goes.
My VB365 server = the same server as Veeam B&R = physical machine.
I don't backup this one because it seems "so easy" to install it again when needed.
1. install windows 2019
2. install veeam B&R
3. restore veeam config
(maybe I should install a veeam agent on it and backup its system drive onto the hardened repository)

I guess I can install VB365 (community version) on my test PC and try to restore from the isolated network and the restored "minio"

I will get back to this topic when I get more data to share

[Mildur]

3. restore veeam config
(maybe I should install a veeam agent on it and backup its system drive onto the hardened repository)

Yes, for VBR it's easy to set up a new machine. VB365 doesn't currently have a configuration backup. But it's on our roadmap.

I guess I can install VB365 (community version) on my test PC and try to restore from the isolated network and the restored "minio"

Yes, that will work as long you connect to the "restored MinIO VM". Connecting to the original bucket will overwrite ownership of the backup repository to the new VB365 server. And you need to add the same organization to the VB365 server to be able to read the backup data from the MinIO bucket.

Best,
Fabian

[sogapex]

Hi,
today, I repaired my veeam 365 config :
for testing purpose (see veeam-backup-for-microsoft-365-f47/wind ... 86251.html), after doing a "config" reset and rollback => the backup job was not longer processing our "licenced" mailboxes
I created the object storage
Then, the object repository
And then, I added again my organization
Finally, I created the job task => the only "warning" message I got = there is already a proxy attached to the repository.
I launched the "first" backup and ... eveything worked as if there had been no change at all. The job only backupped the new data between "exchange online" and my S3 minio.
So, yes, the only thing missing to save some time = being able to save/restore the VB365 config.

I also had to restore some deleted emails last week, but it was before putting into place the S3 minio repository (february = previous veeam server), and there was no search involved = the user exactely knew where I could find the missing folder on the backup => it was "nearly" a straightforward job for me : nearly because when I launched the Veeam365 on the old server, it wouldn't want to open the backup. Veeam B&R had been updated to V12 on this server, but Veeam365 was still version 6 => the "exchange component" is shared between these 2 products = too recent for Veeam365. I had to update Veeam365 to v7 before being able to access the backup

[Mildur]

Hello Sogapex

Thanks for sharing your experience.

the backup job was not longer processing our "licenced" mailboxes

Did the hostname change?

Veeam B&R had been updated to V12 on this server, but Veeam365 was still version 6 => the "exchange component" is shared between these 2 products = too recent for Veeam365. I had to update Veeam365 to v7 before being able to access the backup

Yes, Explorers are shared between both products. You find a compatibility matrix in our KB:
https://www.veeam.com/kb4325

Best,
Fabian

[sogapex]

Hi Fabian,

Did the hostname change?

for our test, I just renamed the 2 folders (ConfigDb and ProxyDb) with ".old" to backup them. But the server name remained the same. After our test, I removed the .old to get back my config files => but after that, I got the problem ("no licence found for XX object" if I remember well).
The hostname was changed a week before (and everything was fine as far as the backup job was concerned)

[rasmusan]

Hi Sogapex,

I am currently looking in to somewhat same setup as you - with a VBM repo (Jet) around 75TB.

I am thinking of putting up a MinIO VM appliance, and doing VBM backup to this - and then doing VBR backup of the VM periodically.

Can I ask how your performance is on backup going to MinIO appliance? and if this is stable in regards to VBM backup/restore and also VBR backup of MinIO appliance?

[sogapex]

Hi rasmusan,

75TB of jet database seems pretty huge to me.
My minio Virtual Machine = 700GB right now (659K objects) from daily backup.
But there is something unusual happening since a few weeks = (Microsoft messing up with the mailboxes ?)
everyday (even sunday), it seems some inboxes (or inplace archive boxes) are fully "modified" and then, backupped.
Our "usual" daily backup size has always been about 2GB since the begining.
Yesterday (sunday), it was 20GB ! => 22617 items in 2hours12min (start time = 22h00)
saturday it was 8GB => 10634 items in 53min
friday it was 12GB => 19561 items in 1hour20min

MinIO performance = I think it can get everything Exchange can deliver (limiting factor = exchange servers)
MinIO VM backup time by VBR = the same as any other VM.
Last one is a full backup, 712GB read, 697GB transferred, 59min (10Gbit link between VM and repository, NVME SSD for the VM, 4x20TB HDD for the repo)
and previous one (one backup per week, everyday sunday) = 214GB read, 84GB transferred, 7min43s

Restore test =
about 2min to launch VBO, select a mailbox, search for a word in the inbox folder and get the result (nothing to compare with the Jet database performance)

Stability = nothing to complain (0 problem encounter so far)

Regards