Production Drive is getting low on free space - HyperV - CSV

[drubs]

Hi all,

So setup is as follows;
HyperV
-3 Node Hyper-V Cluster (Server 2012 R2)
-VMs stored over 2 CSVs
-Underlying FibreChannel Storage (no access to Hardware VSS, so using SoftwareVSS)

Veeam Backup and Replication v9
-Physical Server
-Direct Attached Storage (SAS to JBOD)
-Storage repository provided by Storage Spaces (6PhysicalDisk, 1PhysicalDiskHotspare, 2wayMirror, 3Column)

Network
-Everything is talking over 10GbE

Backup Jobs and Scheduling
-50VMs between multiple tenants

And error message I'm receiving;
"Production drive 'C:\ClusterStorage\Volume1' is getting low on free space (2.9 GB left), and may run out of free disk space completely due to open snapshots."

Only occuring on some a few days during the last week. First time it happened, I came into work with a large number of VMs(All on C:\ClusterStorage\Volume1), in a paused state due to insufficient storage. Not fun.

C:\ClusterStorage\Volume1 normally has at least 1.2TB of free space. But during backups only seems to have 20GB freespace. Finding it hard to believe that snapshots are using that much space.

There are 35 VMs that live on CSV1 and that are backed up, and they total roughly 3.3TB storage used(both dynamic disks and fixed disks). These 35 VMs are seperated into I think around 5 or 6 jobs

Questions
Am I underestimating the size that the Snapshot will be?
Or is there a way to tell the size and anticipate how much is required?
Is there a chance that snapshots are being left open permanently?

Sorry i know there a lot of information, but just any help in understanding the issue would be awesome!

Cheers
Chris

[foggy]

Snapshot size depends on how active VM is and the time it runs off the snapshot. I'd check for existing snapshots (both VM-level and volume-level). For volume-level you can use vssadmin list shadowstorage and vssadmin list shadows (or DISKSHADOW> list shadows all) commands.

[drubs]

Thanks for the suggestions Foggy, that put me in the right direction to clearing up the issue.

Tried the above VSSadmin commands, but nothing showed up. Researched a bit more as to why not and was led to a suggestion in another forum, to move the CSV ownership to another node. Once i did this the expected free storage was accurate again (1.2TB).

Not really a fix, and I'm sure there is still some underlying behavior that I don't quite understand. But for now I will keep a closer eye on it, and if I find anything post back here.

Cheers
Chris

[veremin]

As it's rather hard to find out the root cause of the experienced behaviour through forum correspondence, it might stand to reason to open a support ticket and let them investigate it directly, making a definite statements after logs analysis, etc. Thanks.

[nmdange]

If changing the CSV owner resulted in space getting freed up, then it is likely due to the System Volume Information folder growing unexpectedly. Make sure you have the updates I mentioned in this post: microsoft-hyper-v-f25/csv-system-volume ... 34478.html

[pantos21]

Hi all,

I have the exact same issue and when i change the csv owner the space i freed up. But i have windows 2012 and not 2012 R2 that the recomened pathces works. I canot find these patches for windows 2012

regards
pantos

[foggy]

You can open a case with Microsoft to check if similar patches exist for earlier version.

[javier.martinez.c]

Try this instesd. Using cmd as elevated:
DISKSHADOW
LIST SHADOWS ALL
then you will see the shadows of CSV volumes. You can delete an specific shadow copy using the ID:
DELETE SHADOWS ID {XXX-XXX-XX-XX-XXX}
Example:
DISKSHADOW> delete shadows ID {1ce495ca-dbb7-45a4-b7dd-df2011c881b0}
Deleting shadow copy {1ce495ca-dbb7-45a4-b7dd-df2011c881b0}...
1 shadow copy deleted.
DISKSHADOW>

Using this, I was able to free about 500GB on a CSV volume. This was caused by orphaned shadow copies on a failed backup.

best regards,
Javier