Newbie question: Snapshots and quiet time

[Dru Albright]

Hello, all.

This is probably a pretty basic question, but our Veeam expert is on vacation and I'm mostly trying to understand part of the process. We had a VM fail over the weekend partly due to backups (though Veeam wasn't really at fault, our clients were). But I'd like to make sure I understand WHY the failure occurred.

We have a SQL server running on an ESX 5.1 host, which is backed up using Veeam (we inherited the config, not our first choice). Without alerting us, our clients decided to do a series of massive data crunches in some huge databases. This triggered some large snapshotting, as you might imagine. This led to the server VM going down, due to a lack of disk space on the datastore where one of the VMs drives was stored. This server had two datastores dedicated to it that no other machines shared. The clients made the huge changes and continued making them when the nightly backup kicked off. This led to an error where vSphere reported that it could not reset the VM, then the esx host reported there was no more space for the virtual disk, referencing the snapshot file for one of the drives. Of course, since the snapshots had consumed all the room, I couldn't do a proper consolodation or snapshot deletion, because those tasks require at least a little room to work. VMWare and Veeam both pointed me to moving those drives to another datastore, creating a new snapshot and then Delete All, which cleared everything out and got the machine back. The client then did it again last night, but since it was on a new larger datastore, it only messed up the backup, it didn't drop the VM.

So my question is basically this: how exactly does the snapshot work during the backup while changes are made? I glanced at a few of the articles, but it wasn't quite clear to me.
Is this what happens?

1. Veeam Backup Starts
a. Veeam creates snapshot of each drive and locks the original disk file
b. Veeam records any changes (the delta) in the snapshot file

2. Veeam backs up the VMDK files to the specified location

3. Veeam Backup Finishes
a. Veeam combines the snapshot delta and the original disk file
b. Veeam deletes the now unneeded snapshot file

Is that correct?
Am I correct in thinking that if changes happen during that third step that a NEW snapshot file is produced to cover that interim period (i.e. a 00002 file)? And that if that happens quickly and the snapshots are large then it can potentially overwhelm the process, consuming all the drive space before the backup is completed (as I think is happening here)?

This is obviously a process problem, not a technical one, per se. I'm just trying to make sure I understand how the backup is working.

Thanks!

[veremin]

It's not Veeam that handles the snapshot creation and removal process. Veeam plays only "requestor" role, instructing vSphere via certain APIs to take and consolidate a snapshot.

Your understanding is mostly correct; the full description of snapshot technology can be found here.

Thanks.

[Dru Albright]

Perfect. That article is very helpful. Thanks.

[veremin]

You're welcome. Feel free to ask, should other clarification be needed.