Comprehensive data protection for all workloads
Post Reply
pshute
Veteran
Posts: 254
Liked: 14 times
Joined: Nov 23, 2015 10:56 pm
Full Name: Peter Shute
Contact:

Disk space drops on Veeam server

Post by pshute »

I've noticed over the 2 years we've been running Veeam, that twice there have been big drops in disk space on C: the drive where Veeam is installed, and the disk space has never recovered.

I've discovered two files matching the size and dates of these drops:
veeamflr-000043e300001b02-0000.flat 132GB 16-12-2016 11:53:23 am
veeamflr-00007c94000057c0-0000.flat 89GB 23-02-2018 10:33:39 am

No jobs are currently running. Is it safe to delete these files? If so, why are they there? Is this the result of us restarting services while jobs are running?
pshute
Veteran
Posts: 254
Liked: 14 times
Joined: Nov 23, 2015 10:56 pm
Full Name: Peter Shute
Contact:

Re: Disk space drops on Veeam server

Post by pshute »

Probably related to the question asked here, but not answered:
https://forums.veeam.com/vmware-vsphere ... =c#p259658
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Disk space drops on Veeam server

Post by Vitaliy S. »

If there are no active restore sessions/active instant VM recovery operation, then these files are safe to delete.
pshute
Veteran
Posts: 254
Liked: 14 times
Joined: Nov 23, 2015 10:56 pm
Full Name: Peter Shute
Contact:

Re: Disk space drops on Veeam server

Post by pshute »

Thanks. Can I suggest that Veaam should check for these abandoned files and clean them up? These two files ate 220GB of a 300GB in just over a year before we realised. The next restore attempt would probably have failed for lack of space.
pshute
Veteran
Posts: 254
Liked: 14 times
Joined: Nov 23, 2015 10:56 pm
Full Name: Peter Shute
Contact:

Re: Disk space drops on Veeam server

Post by pshute »

One of the files was locked (the 83GB one), and couldn't be deleted, despite no jobs running at the time. I stopped every Veeam service and task, and waited 10 minutes as per https://www.veeam.com/kb1727, still locked.

I ended up rebooting the server, and that fixed it.
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Disk space drops on Veeam server

Post by Vitaliy S. »

pshute wrote:Can I suggest that Veaam should check for these abandoned files and clean them up?
Yes, I agree, but this situation should not happen in the first place, so I wounder how to reproduce it and share it with our QA team.
pshute wrote:One of the files was locked (the 83GB one), and couldn't be deleted, despite no jobs running at the time. I stopped every Veeam service and task, and waited 10 minutes as per https://www.veeam.com/kb1727, still locked.

I ended up rebooting the server, and that fixed it.
Next time I would also suggest reaching out to our support team, so that they could capture all the logs, verify what process is locking it and then submit an issue to the R&D team. Glad that you've fixed that anyway!
pshute
Veteran
Posts: 254
Liked: 14 times
Joined: Nov 23, 2015 10:56 pm
Full Name: Peter Shute
Contact:

Re: Disk space drops on Veeam server

Post by pshute »

I have had to delete one of these files again. This is how I think the file ended up there:
- a user logged into the backup server as himself, and began an Exchange mailbox item restore
- they left the Explorer for MS Exchange open and went home
- I later noticed there was insufficient space for the night's backup, and could see there was an Exchange Item Restore job running, and wanted to end it so the file space would increase
- I couldn't see the Explorer window because I was logged in under a different username.
- I double clicked the job, and clicked Cancel Restore Task.
- It never finished cancelling it, I assume because the Explorer was open.
- I rebooted the server, and deleted the *.flat file manually.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Disk space drops on Veeam server

Post by foggy »

Thanks for sharing, Peter. I will pass it on to QA for confirmation and then we'll think of ways to address this behavior.
pshute
Veteran
Posts: 254
Liked: 14 times
Joined: Nov 23, 2015 10:56 pm
Full Name: Peter Shute
Contact:

Re: Disk space drops on Veeam server

Post by pshute »

I think the ideal solution would be to allow another user to force the other user's Explorer for MS Exchange session closed. Having to reboot servers or restart services, and then clean up orphaned files isn't good, even if they get cleaned up automatically.
Mike Resseler
Product Manager
Posts: 8044
Liked: 1263 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: Disk space drops on Veeam server

Post by Mike Resseler »

Peter,

I just passed it to the DEV team to look at it. I believe we already have a timeout but can't remember for sure. Asked for more information and will let you know once I have it
Mike Resseler
Product Manager
Posts: 8044
Liked: 1263 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: Disk space drops on Veeam server

Post by Mike Resseler »

Peter,

This seems to be related to the mount server. Our devs will look further into it

Mike
mrholm
Expert
Posts: 170
Liked: 15 times
Joined: Apr 20, 2018 8:12 am
Full Name: Mats Holm
Contact:

Re: Disk space drops on Veeam server

Post by mrholm »

Hi
I know this is an old thread but we ran into some silimiar today. A large file server needed restore, the server have 22 disks (I know!!) and restore points are stored on StaoreOnce.
When starting the mount from the backup server, 22x .flat files are cretaed under c:\windows\temp\"veeamtemporarycatalogname" and start taking up space.
The largest one is taking up 60 GB, the next 50 GB and so on until 162 GB is taken so that C: drive is now filled.
Even though no process is running we can't remove the files they're open in "System"
I will restart the server and remove the files but I have two questions:

1. Is there any way to change the path to where Veeam store the .flat files?
2. Is this based on the fact that the restore point is on StoreOnce? Since we don't experiencing this with restore Points stored on Windows Repos.
valerio_vanni
Lurker
Posts: 2
Liked: never
Joined: Mar 04, 2020 2:48 pm
Full Name: Valerio Vanni
Contact:

Re: Disk space drops on Veeam server

Post by valerio_vanni »

It just happened to me too (B&R community edition on a Windows 2019 STD server).

I wasn't actively restoring anything, only browsing backup images (files and AD objects). I found my disk filled within some minute.
I don't know what exactly triggered this, other times I've browsed images without filling C: disk.

The files are
C:\temp\$manyrandomdirs\$randomfile.flat (size from zero to 73 GB). All locked by SYSTEM, so I will not be able to retry until next reboot (I hope I'll be then...).

I stopped all veeam* services. Now they have stopped growing, but they are undeletable.
Also, event viewer is filling with warning NTFS 140, saying "device not ready" for "C:\VeeamFLR\$nameofbackupimage\$Volume".
diskpart with "list volume" shows only physical disks, I cannot find any place to stop this.
Gostev
Chief Product Officer
Posts: 31455
Liked: 6646 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Disk space drops on Veeam server

Post by Gostev »

Do you possibly have some antivirus on that server? This is what can be acting up.
valerio_vanni
Lurker
Posts: 2
Liked: never
Joined: Mar 04, 2020 2:48 pm
Full Name: Valerio Vanni
Contact:

Re: Disk space drops on Veeam server

Post by valerio_vanni »

No. There is the integrated Defender, but it has real time protection disabled.

After a reboot I could delete those files. Last evening I tried to browse some image and I didn't notice anything.

But I didn't look too well, at that moment: today I see again a veeamflr****.flat file of 40 GB in C:\Windows\Temp, with a last evening time and date.
Only one, now C: isn't full, but it's the same issue. I looked with process explorer, and it's locked by SYSTEM process so i will not be able to delete it until next reboot.
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Disk space drops on Veeam server

Post by veremin »

Then, it's time to reach our support team, as this is not expected and closer investigation is required here. Thanks!
zhawkins
Lurker
Posts: 1
Liked: 1 time
Joined: Jan 04, 2019 7:27 pm
Full Name: Zachary Hawkins
Contact:

Re: Disk space drops on Veeam server

Post by zhawkins » 1 person likes this post

I am also having this issue. We updated to Veeam 10 and C:\Windows\Temp\ started having locked Veeam flat files filling up the c:\ drive. I've opened a case #: 04106208
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Disk space drops on Veeam server

Post by veremin »

Thanks, Zachary, for the ticket number provided (it helps us and future readers). For now, kindly, keep working with the support team and ask for additional assistance (if any is needed). Thanks!
ml35
Enthusiast
Posts: 42
Liked: 11 times
Joined: May 30, 2019 8:42 pm
Contact:

Re: Disk space drops on Veeam server

Post by ml35 »

veremin wrote: Mar 05, 2020 4:02 pm Then, it's time to reach our support team, as this is not expected and closer investigation is required here. Thanks!
wouldn't it be better to simply check for those files and automatically delete them, and automatically notify user to contact support? why let those file linger there useless? I just found one accidentally on free agent / windows 2004
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Disk space drops on Veeam server

Post by foggy »

This is an upexpected situation, those files should be automatically deleted. If that doesn't happen, we encourage to open a case with support. Thanks!
pshute
Veteran
Posts: 254
Liked: 14 times
Joined: Nov 23, 2015 10:56 pm
Full Name: Peter Shute
Contact:

Re: Disk space drops on Veeam server

Post by pshute »

valerio_vanni wrote: Mar 04, 2020 3:17 pm I wasn't actively restoring anything, only browsing backup images (files and AD objects). I found my disk filled within some minute.
If you read my posting above from 1 Aug 2018, I was able to reproduce the problem by just browsing. In my case I was using the Explorer for MS Exchange. It looks like this creates a file for the browser to display all the information in the backup.

That works fine if you have enough room for the file. It deletes the file when you close the Explorer.

You would have trouble if you didn't have enough room for the file. In our case we had room, but the Explorer was crashed instead of closed, so the file stayed. The reason it was crashed instead of closed was because one user left it open while logged into the server with his own account, and other users were unable to close it other than by killing his session or rebooting. This happened several times over several months until we ran out of space.

If you have several people logged in at once, all simultaneoulsy browsing restore points (or just left the Explorer open), you could quickly run out of room.
Post Reply

Who is online

Users browsing this forum: Bing [Bot], Ivan239, Mircea Dragomir, nikita.kozlenko and 178 guests