Disk space drops on Veeam server

[pshute]

I've noticed over the 2 years we've been running Veeam, that twice there have been big drops in disk space on C: the drive where Veeam is installed, and the disk space has never recovered.

I've discovered two files matching the size and dates of these drops:
veeamflr-000043e300001b02-0000.flat 132GB 16-12-2016 11:53:23 am
veeamflr-00007c94000057c0-0000.flat 89GB 23-02-2018 10:33:39 am

No jobs are currently running. Is it safe to delete these files? If so, why are they there? Is this the result of us restarting services while jobs are running?

[pshute]

Probably related to the question asked here, but not answered:
https://forums.veeam.com/vmware-vsphere ... =c#p259658

[Vitaliy S.]

If there are no active restore sessions/active instant VM recovery operation, then these files are safe to delete.

[pshute]

Thanks. Can I suggest that Veaam should check for these abandoned files and clean them up? These two files ate 220GB of a 300GB in just over a year before we realised. The next restore attempt would probably have failed for lack of space.

[pshute]

One of the files was locked (the 83GB one), and couldn't be deleted, despite no jobs running at the time. I stopped every Veeam service and task, and waited 10 minutes as per https://www.veeam.com/kb1727, still locked.

I ended up rebooting the server, and that fixed it.

[Vitaliy S.]

Can I suggest that Veaam should check for these abandoned files and clean them up?

Yes, I agree, but this situation should not happen in the first place, so I wounder how to reproduce it and share it with our QA team.

One of the files was locked (the 83GB one), and couldn't be deleted, despite no jobs running at the time. I stopped every Veeam service and task, and waited 10 minutes as per https://www.veeam.com/kb1727, still locked.

I ended up rebooting the server, and that fixed it.

Next time I would also suggest reaching out to our support team, so that they could capture all the logs, verify what process is locking it and then submit an issue to the R&D team. Glad that you've fixed that anyway!

[pshute]

I have had to delete one of these files again. This is how I think the file ended up there:
- a user logged into the backup server as himself, and began an Exchange mailbox item restore
- they left the Explorer for MS Exchange open and went home
- I later noticed there was insufficient space for the night's backup, and could see there was an Exchange Item Restore job running, and wanted to end it so the file space would increase
- I couldn't see the Explorer window because I was logged in under a different username.
- I double clicked the job, and clicked Cancel Restore Task.
- It never finished cancelling it, I assume because the Explorer was open.
- I rebooted the server, and deleted the *.flat file manually.

[foggy]

Thanks for sharing, Peter. I will pass it on to QA for confirmation and then we'll think of ways to address this behavior.

[pshute]

I think the ideal solution would be to allow another user to force the other user's Explorer for MS Exchange session closed. Having to reboot servers or restart services, and then clean up orphaned files isn't good, even if they get cleaned up automatically.

[Mike Resseler]

Peter,

I just passed it to the DEV team to look at it. I believe we already have a timeout but can't remember for sure. Asked for more information and will let you know once I have it

[Mike Resseler]

Peter,

This seems to be related to the mount server. Our devs will look further into it

Mike

[mrholm]

Hi
I know this is an old thread but we ran into some silimiar today. A large file server needed restore, the server have 22 disks (I know!!) and restore points are stored on StaoreOnce.
When starting the mount from the backup server, 22x .flat files are cretaed under c:\windows\temp\"veeamtemporarycatalogname" and start taking up space.
The largest one is taking up 60 GB, the next 50 GB and so on until 162 GB is taken so that C: drive is now filled.
Even though no process is running we can't remove the files they're open in "System"
I will restart the server and remove the files but I have two questions:

1. Is there any way to change the path to where Veeam store the .flat files?
2. Is this based on the fact that the restore point is on StoreOnce? Since we don't experiencing this with restore Points stored on Windows Repos.

[valerio_vanni]

It just happened to me too (B&R community edition on a Windows 2019 STD server).

I wasn't actively restoring anything, only browsing backup images (files and AD objects). I found my disk filled within some minute.
I don't know what exactly triggered this, other times I've browsed images without filling C: disk.

The files are
C:\temp\$manyrandomdirs\$randomfile.flat (size from zero to 73 GB). All locked by SYSTEM, so I will not be able to retry until next reboot (I hope I'll be then...).

I stopped all veeam* services. Now they have stopped growing, but they are undeletable.
Also, event viewer is filling with warning NTFS 140, saying "device not ready" for "C:\VeeamFLR\$nameofbackupimage\$Volume".
diskpart with "list volume" shows only physical disks, I cannot find any place to stop this.

[Gostev]

Do you possibly have some antivirus on that server? This is what can be acting up.

[valerio_vanni]

No. There is the integrated Defender, but it has real time protection disabled.

After a reboot I could delete those files. Last evening I tried to browse some image and I didn't notice anything.

But I didn't look too well, at that moment: today I see again a veeamflr****.flat file of 40 GB in C:\Windows\Temp, with a last evening time and date.
Only one, now C: isn't full, but it's the same issue. I looked with process explorer, and it's locked by SYSTEM process so i will not be able to delete it until next reboot.

[veremin]

Then, it's time to reach our support team, as this is not expected and closer investigation is required here. Thanks!

[zhawkins]

I am also having this issue. We updated to Veeam 10 and C:\Windows\Temp\ started having locked Veeam flat files filling up the c:\ drive. I've opened a case #: 04106208

[veremin]

Thanks, Zachary, for the ticket number provided (it helps us and future readers). For now, kindly, keep working with the support team and ask for additional assistance (if any is needed). Thanks!

[ml35]

Then, it's time to reach our support team, as this is not expected and closer investigation is required here. Thanks!

wouldn't it be better to simply check for those files and automatically delete them, and automatically notify user to contact support? why let those file linger there useless? I just found one accidentally on free agent / windows 2004

[foggy]

This is an upexpected situation, those files should be automatically deleted. If that doesn't happen, we encourage to open a case with support. Thanks!

[pshute]

I wasn't actively restoring anything, only browsing backup images (files and AD objects). I found my disk filled within some minute.

If you read my posting above from 1 Aug 2018, I was able to reproduce the problem by just browsing. In my case I was using the Explorer for MS Exchange. It looks like this creates a file for the browser to display all the information in the backup.

That works fine if you have enough room for the file. It deletes the file when you close the Explorer.

You would have trouble if you didn't have enough room for the file. In our case we had room, but the Explorer was crashed instead of closed, so the file stayed. The reason it was crashed instead of closed was because one user left it open while logged into the server with his own account, and other users were unable to close it other than by killing his session or rebooting. This happened several times over several months until we ran out of space.

If you have several people logged in at once, all simultaneoulsy browsing restore points (or just left the Explorer open), you could quickly run out of room.