Host-based backup of Microsoft Hyper-V VMs.
Post Reply
drubs
Novice
Posts: 4
Liked: never
Joined: Jan 12, 2016 4:51 am
Contact:

Production Drive is getting low on free space - HyperV - CSV

Post by drubs »

Hi all,

So setup is as follows;
HyperV
-3 Node Hyper-V Cluster (Server 2012 R2)
-VMs stored over 2 CSVs
-Underlying FibreChannel Storage (no access to Hardware VSS, so using SoftwareVSS)

Veeam Backup and Replication v9
-Physical Server
-Direct Attached Storage (SAS to JBOD)
-Storage repository provided by Storage Spaces (6PhysicalDisk, 1PhysicalDiskHotspare, 2wayMirror, 3Column)

Network
-Everything is talking over 10GbE

Backup Jobs and Scheduling
-50VMs between multiple tenants

And error message I'm receiving;
"Production drive 'C:\ClusterStorage\Volume1' is getting low on free space (2.9 GB left), and may run out of free disk space completely due to open snapshots."

Only occuring on some a few days during the last week. First time it happened, I came into work with a large number of VMs(All on C:\ClusterStorage\Volume1), in a paused state due to insufficient storage. Not fun.

C:\ClusterStorage\Volume1 normally has at least 1.2TB of free space. But during backups only seems to have 20GB freespace. Finding it hard to believe that snapshots are using that much space.

There are 35 VMs that live on CSV1 and that are backed up, and they total roughly 3.3TB storage used(both dynamic disks and fixed disks). These 35 VMs are seperated into I think around 5 or 6 jobs

Questions
Am I underestimating the size that the Snapshot will be?
Or is there a way to tell the size and anticipate how much is required?
Is there a chance that snapshots are being left open permanently?

Sorry i know there a lot of information, but just any help in understanding the issue would be awesome!

Cheers
Chris
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Production Drive is getting low on free space - HyperV -

Post by foggy »

Snapshot size depends on how active VM is and the time it runs off the snapshot. I'd check for existing snapshots (both VM-level and volume-level). For volume-level you can use vssadmin list shadowstorage and vssadmin list shadows (or DISKSHADOW> list shadows all) commands.
drubs
Novice
Posts: 4
Liked: never
Joined: Jan 12, 2016 4:51 am
Contact:

Re: Production Drive is getting low on free space - HyperV -

Post by drubs »

Thanks for the suggestions Foggy, that put me in the right direction to clearing up the issue.

Tried the above VSSadmin commands, but nothing showed up. Researched a bit more as to why not and was led to a suggestion in another forum, to move the CSV ownership to another node. Once i did this the expected free storage was accurate again (1.2TB).

Not really a fix, and I'm sure there is still some underlying behavior that I don't quite understand. But for now I will keep a closer eye on it, and if I find anything post back here.

Cheers
Chris
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Production Drive is getting low on free space - HyperV -

Post by veremin »

As it's rather hard to find out the root cause of the experienced behaviour through forum correspondence, it might stand to reason to open a support ticket and let them investigate it directly, making a definite statements after logs analysis, etc. Thanks.
nmdange
Veteran
Posts: 527
Liked: 142 times
Joined: Aug 20, 2015 9:30 pm
Contact:

Re: Production Drive is getting low on free space - HyperV -

Post by nmdange » 2 people like this post

If changing the CSV owner resulted in space getting freed up, then it is likely due to the System Volume Information folder growing unexpectedly. Make sure you have the updates I mentioned in this post: microsoft-hyper-v-f25/csv-system-volume ... 34478.html
pantos21
Novice
Posts: 3
Liked: never
Joined: Jun 26, 2014 9:51 am
Full Name: pantos
Contact:

Re: Production Drive is getting low on free space - HyperV -

Post by pantos21 »

Hi all,

I have the exact same issue and when i change the csv owner the space i freed up. But i have windows 2012 and not 2012 R2 that the recomened pathces works. I canot find these patches for windows 2012

regards
pantos
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Production Drive is getting low on free space - HyperV -

Post by foggy »

You can open a case with Microsoft to check if similar patches exist for earlier version.
javier.martinez.c
Lurker
Posts: 1
Liked: never
Joined: Feb 14, 2020 1:12 pm
Full Name: Javier Martinez
Contact:

Re: Production Drive is getting low on free space - HyperV - CSV

Post by javier.martinez.c »

Try this instesd. Using cmd as elevated:
DISKSHADOW
LIST SHADOWS ALL
then you will see the shadows of CSV volumes. You can delete an specific shadow copy using the ID:
DELETE SHADOWS ID {XXX-XXX-XX-XX-XXX}
Example:
DISKSHADOW> delete shadows ID {1ce495ca-dbb7-45a4-b7dd-df2011c881b0}
Deleting shadow copy {1ce495ca-dbb7-45a4-b7dd-df2011c881b0}...
1 shadow copy deleted.
DISKSHADOW>

Using this, I was able to free about 500GB on a CSV volume. This was caused by orphaned shadow copies on a failed backup.

best regards,
Javier
Post Reply

Who is online

Users browsing this forum: No registered users and 30 guests