No space for snapshot

[Peejay62]

Hi,

I started rolling out VAL on numerous servers. Some backups succeed, some of them fail with "no space for snapshot". I have already been reading some threads concerning these snapshot issues and before putting in a call can anybody enlighten me on where the snapshots are actually stored during backup? Also, the various settings in Veeam.ini concerning snapshots suggest there can be some adjustment for snapshot processing, the intended use isn't that clear though. For example, there is a value "location" suggesting the place where to store your snapshot data, if this is the case then can an nfs mounted filesystem be used for that purpose. As the systems where snapshots fail are different from eachother I suspect there is a per machine adjustment for snapshotting required?
Any insight in this will be much appreciated. I have a lot of Linux servers to deploy with VAL.

Thanks, Peter

[PTide]

Hi,

There are two snapshot types that VAL can use: 'stretch' (default) and 'common'.

The difference between those two is that 'stretch' snapshot can grow, while 'common' snapshot is allocated in a single large chunk when a backup job starts.

Failover to 'common' occurs if there is no filesystem that supports 'stretch' present (EXT4, XFS).

Here is a brief idea how VAL build a list of candidate devices where to place stretch snapshot file:

1. Apprx.required storage space is calculated (% of total space occupied)

2. Loop devices, snapshot block devices, removable devices are excluded.

3. Devices with no filesystem present are excluded (that is, things like devices that constitute RAID

4. If the devices satisfies all the previous criteria but is not mounted it is skipped too.

5. Check if the filesystem is supported for snapshot placement.

6. If the device resides on another device that is in backup scope (dm devices, for example) - skip

7. If the amount of free space cannot be obtained - skip.

8 If sector size is larger that snapshot block size (16K) - skip (this one should not occur normally unless someone has changed the default value for snapshot blocksize)

If after all those checks there are more than 1 device left, VAL sorts the list:

Primary criteria: EXT4 or XFS are preferred (these support fallocate)
Secondary criteria: free space

Then VAL checks if the device is in the backup scope. If yes, then it skips it (it can work, however the performance is not very good).

If the device is not in the backup scope, then VAL calculates how much of free space can be actually used for snapshot - if [free space x limitfreepercent] is greater than the approximate value from bullet #1.
If it is not greater, then we skip this one too.

If no more devices left, VAL picks the first one from sorted.

That's a brief explanation just to give you an idea of how it works. You can explicitly set snapshot file location via location parameter.

Snapshot requires block-level access (that is, NFS cannot be used)

We aim to publish a detailed article on that subject soon.

Thanks!

[Peejay62]

thanks for this brief yet detailed info!
So, if I get it right, per default stretch is used and I can force snapshot to a location? syntax would be as an example : location=/var/snapshotdirectory ?
And what is the minimal size allocation needed for stretch (cannot see the default) and can I define this with portionsize then?
I know, lot of questions but they can help me in understanding and getting this up and running in a good way. As said, I now have some systems running fine and some of them failing due out of space for snapshot.

thanks,

[PTide]

So, if I get it right, per default stretch is used and I can force snapshot to a location?

Right, by default the agent will attempt to use 'stretch'. Yes, you can force it to place snapshot file in a specific location, and the syntax is correct.

And what is the minimal size allocation needed for stretch (cannot see the default) and can I define this with portionsize then?

That depends on data change rate on a particular system. By default VAL assumes that a certain percent of total free space on the volumes in backup should be enough. That is, if you include sda1 and sdb1 into backup scope and there is 100GB of free space in total on both partitions, VAL will request the resulting snapshot size will be 100G x %, but not less than portionsize. If an overflow occurs, then we attempt to auto-tuned the value on retries.

And remember - it is recommended to contact support so they can adjust the values in case auto-tune does not work.

Thanks!

[Peejay62]

Ok, thanks again.
To see if I got this right: I could force it to use common snapshot (type=common), restrict minimum and maximum size for snapshot (minsize/maxsize) and force it to write to my desired location.
I have some machines around 200GB with not that much space (1,5 Gb) available but they also don't change a lot at all. I expect a backup will take not that long and during that period it seems almost impossible that snapshot space will be consumed. (hardly any change-rate). PS. I am assuming that snapshot space is used for storing changes during backup. I think the stretch alorithm decides there is insufficient snapshot space available so the job fails. By using common and restricting snapshotsize to to the free space I suspect I can get the job working?

thanks,

[PTide]

I could force it to use common snapshot (type=common), restrict minimum and maximum size for snapshot (minsize/maxsize) and force it to write to my desired location.

Right, you can do that. Also please note that limitFreePercent defines the upper limit for stretch too. That is, if you define location=/snapshot_dir, and limitFreePercent is set to '50', then the snapshot will take no more than 50% of free space available on the device where snapshot_dir resides at. Just make sure that portionSize is not larger than the [limitFreePercent x freeSpace].

I am assuming that snapshot space is used for storing changes during backup.

VAL uses COW (copy-on-write) snapshot, so it works as follows: when the system has to change a block during the backup, the original value of the block is first copied to the snapshot store, and after that the old value is overwritten with new data on the disk.

By using common and restricting snapshotsize to to the free space I suspect I can get the job working?

If you know for sure that the amount of changes won't be larger than 'X', then yes, that approach will work too.

Thanks!