Large amounts of AVHDX, MRT and RCT Files

[danielalpha]

Hello,
I am using B&R now a couple of weeks in my lab and i am very appreciated.

but i got a Problem: My VHD Disk is running out of space.

I have no Snapshots or similar in Hyper V, that files have to be created by veeam.

All VMs have .vhdx.mrt and rct files, but only one of the pair, One VM of mine ( The Filer VM ) has now about 80 GB of that files.

What did i misconfigure ? I configured the VM in my mind same as the others.

Greetings

[Mike Resseler]

Hi Daniel,

First: Welcome to the forums
Second: Can you tell me exactly what that Filer VM has. the .VHDX file is a disk (so if that VM has multiple disks, you should see multiple). Per disk, you should see MRT and RCT files which is the change block tracking information. It remains attached to the disk.
When Veeam takes a backup, you should (during the backup) see a .AVHDX file which is the temporary snapshot. What files are you all seeing?

[danielalpha]

Hello,
Thank you very much.
The Filer VM has 1 4 TB VHDx mounted which is stored at the Host. Additionally it has a VHDx for the System which is about 100 or 30 GB great.

I have about 80 GB of these MRT and RCT files. I have just 1 or 2 Restore Points configured. The Space is now such low, that i cant backup Any VM on that SSD anymore.

Greetings

[danielalpha]

Here a Screenshot: https://ibb.co/HNggyfh

The Problem is slowly getting serios for me. The whole System is gonna Crash when the Space is zero, i will have to shutdown B&R soon if the Problem doesnt gets fixed.

Greetings

[nmdange]

See if these help at all
https://blog.workinghardinit.work/2015/ ... l-machine/
https://blog.workinghardinit.work/2014/ ... eckpoints/

[danielalpha]

Thanks for the Tipp, but as I mentioned above, there are no snapshots or similar in hyper V created.

[Mike Resseler]

Hey Daniel,

I'm afraid there is another problem which (at least I think) has nothing to do with Veeam. (PS: When we backup, we do the same thing, we take "snapshots"). Your RCT and MRT file are too large so I do think there is an issue there and you might need to contact Microsoft support for this. It seems that the change block tracking is going a bit crazy

[danielalpha]

Hello,
Sadly Microsoft isn't giving support to students with a home lab.
I'm gonna move the VM today to another very slow storage and will try to figure out the problem.

Greetings

[danielalpha]

Is it a try worth, to make a Backup with Veeam Agent, then delete the old VHDx which is buggy and create a new vhdx and then restore the Backup to that ?

[Mike Resseler]

That would be sure a possibility. You would threat it as a "physical box". Don't forget to create the recovery media ISO then so you can start the recovery from that ISO, will make your life much easier

[danielalpha]

Good Morning,
i moved the VHDx now to a another Storage. As magic, Hyper V did start automatically the Merge Progress after the VHDx was moved.

At now the System is in a normal State, but i am now testing with Veeam B&R if the Problem will occur again.

As you said Veeam creates Snapshots like the Checkpoints from Hyper V, in my opinion it could be possible that the Mechanism that automatically merges this checkpoints could be buggy in my system ?

Thanks for your Help so far.

Greetings

[danielalpha]

Hello,
anyone got an idea? One of the 4 TB Drives is getting the Files again, i will have to migrate the Storage, but i dont have another 4 TB here
So it would be nice, if i could fix that problem without migrating the Store.
The System Disk seems to dont overflow again, the avhdx files there from veeam are deletet after backup successfully.

Greetings

[Mike Resseler]

So If I understand it correctly. You have migrated the VM, you take backups with Veeam B&R and the AVHDX files are deleted after backup. At that point in time, how big are the MRT and RCT file? They should be smaller again

[danielalpha]

Good Morning, thanks for your Reply.

So i try to explain it well:

I had the Problem VM-Wide, with the System VHDx (about 30 GB) and with the Storage VHDx (About 2 TB yet). In Hyper V were no Checkpoints as i could see. i Tried to merge with Powershell, but there was not even an Output, i think because Windows has no Checkpoints registered and so the Action was successfull quitted.

Then i had to move (Just the one) the System VHDx File to another Storage because the Flash Storage was almost full and then all the VMs and the Host would crash.

After Hyper V completet the move Action of the System VHDx successfully, it autmatically shows me that it began Merging. It took about 12 Hours to Merge all the SystemVHDx Files and the Files of the 2 TB VHDx. At that Point, everything was in a normal state again.

But when Veeam takes new backups, it creates new Files again, as it is normal, and it deletes now old Files in the SystemVHDx, but still not on the great Drive, with 2 TB.

I would now sayed i just move that Storage one time too, as Workaround for that problem, but i have no Storage at home where i could migrate it, and the Filer is always Busy too and has to be High Avaible. I will Check today if the Problem is surely Fixed with migrating the Storage one time, and if it is, i will buy a external Drive to migrate the Storage. Pretty much Work, but by the Way, Veeam B&R is awesome in Home Lab especially for Testing etc pp..

Sorry for my bad English.

Greetings

[danielalpha]

Is that normal ?

https://ibb.co/VM8pqw7

[Mike Resseler]

Hey Daniel,

The picture makes it a bit more clear (a picture says more than a 1000 words ). What I can see is that you two disks in that directory. (Exchange.VHDX and ExchangeBackEnd.VHDX). Both those drives have a MRT and RCT file and the sizes are normal sizes, nothing wrong with that. As I said, those files are the mechanism to understand what blocks are changed in between backups. Then I see again two disks (Exchange_XXXX.AVHDX and ExchangeBackEnd_XXXX.AVHDX). Those are both snapshots. They also have a MRT and RCT file, which is again normal, and size of those files is also normal. What seems to be not normal is the size of those snapshots. It means that the VM is running on those snapshots already for quite some time. Hence the storage issues you are seeing. From the timestamps, I can see you are running on a snapshot disk for 1 month, 1 day and about 12 hours which is:
a) Way too long
b) Not good for the performance of your Exchange server

So that snapshot needs to get merged. If you can't get it merged, then you would loose a LOT of data. 1 Month+ worth of data actually. Most likely, that snapshot isn't Veeam's, but comes from someone taking a checkpoint (that is how it is called in Hyper-V) with the Hyper-V UI. So open Hyper-V manager, and see if it has running checkpoints. You should see it in there.

Now here is the thing. If you are going to merge that snapshot, you are in for a serious performance degradation (unless you are running on 2k16 or higher ReFS). At this point, to be on the safe side, I really would ask you to create a support call. Our engineers will be able to help you with this and get this fixed so you can start clean again and take backups with Veeam

[FE_BHD]

I experienced the same problem and it does seem to be a Microsoft problem.
Situation:
- a lot of avhdx, mrt and rct files (not every VM, but in my case only one)
- virtual hard drive pointing to one of those avhdx files


What didn't work:
- Creating a manual checkpoint and deleting it
- Manually merging old Checkpoints (ruined the chain)


What did work:
- Move some old checkpoints to gain space
- shut down VM
- create backup with veeam
- move ALL the old checkpoints
- restore everything from veeam to original location

--> virtual hard drive now pointing to the vdhx file
No new avhdx files are created, I moved the old ones (will delete them in a month or so)

Hope this helps. No use for high availability of course.

[justkaka]

Veeam Community version 10 -
I have a lot of avhdx, mrt and rct files
Virtual Disks but I have 30 files in my Virtual Hard Disks folder (vhdx and mrt)

I have NOT created any checkpoints - I assume these are are created by
Backups complete fine. Yesterday I patched both HOST and problem VM (no other problems just these files)

I managed to Backup & Restore these files to another Veeam Server
In the restored version there are only the 2 VHD's
everything looks uptodate!
I checked data and Sample text files I created before Backup
Other files created / updated recently all look OK

I this the solution??
Create a Full Backup > Delete the orginal VM on the source Server > Restore to original location?

Any gotccha to be aware of ??

I just can't afford to loose any data!
Apprecaite any comments

[PetrM]

Hello,

I would say this is a workaround but you should investigate the root cause with support to find the permanent solution. Most likely Microsoft support needs to be involved as well. Not clear what kind of data loss do you try to avoid: you restore a VM to the state that it has in backup.

Thanks!

[justkaka]

Looks like everything is OK with the restored VM

I was just being cautious !

Looks like Veeam is able to figure out and combile the Checkpoints/Mrt files etc - or it a Windows thing..
Full VHD - no checkpoint or mrt's in restore
All my data seems ok

[bas60]

Hi Justkaka / Petrm

Did you manage to check everthing is OK??
any lingering issues

Windows Servr 2019 Server, Windows Server 219 VM (Hyper-V config V 9) - standalone server NOT domain linked.

I have 2 VM's - one has these MRT, RTC & AVHD files - the VM vm with issue has 3 disks so 3 lots of these files
I have 4 checkpoints that I didn't create - multiple mrt & avhdx files
Starting April there is 1 a month - different days BUT same time 6:30am
My Backup finishes about 5:45am
There is Plenty of disk space so I don't think that is a issue

Backup seems to be Complete and I can Restore the VM to a new Installation of Windows and Veeam 11
Like Justkaka I can restore the Backup on this other Server and the stored Disk are just 2 VHDX files - there are NO AVHDX, RCT or MRT files
And restored data contain my OS & data - boots fine
Sample file I just placed before backup are restored

My Application and its data all seem Ok - has the latest data from the backup was made and before/after the avhdx etc

If all OK I was thinking of moveing to a new Physical Server which has Veeam 11 and reformatting the old Physical Server

Any input / comments appreciated

[PetrM]

Hello,

Not sure that I've got the problem that you're trying to solve. What exactly would you like to migrate to a new server and how AVHDX, MRT and RCT files are related to the migration process?

Thanks!

[bas60]

Guess what I'm asking is :

if I run Veeam Backup today and Restore it
The Restored version only has 3 VHDX files AND my data / Server seems OK (no multiple AVHDX, MRT & RCT files)
YET on the Source Server there alre still the AVHDX, MRT & RCT files

The manual merging process I've googled ie. merging Child to correct Parent AND in the right order seems complicated and perhaps may result in error

is it safe to assme that Backup has somehow created a Merged VHDX for each of the snapshot?

if so is Backup / Restore my solution.
If so I can just restore the Backup on the New Phyiscal Server and ditch the old server with the AVHDX, MRT & RCT files ?

[PetrM]

Hello,

Yes, you can follow backup-restore workaround suggested in this thread. It's up to you whether to remove the original VM or not, but I recommend to make sure that the restored workload works fine and has actual data before you remove the original VM.

Thanks!