Veeam killed a VM?

[AJ83]

Hi there, i was wondering if anyone else ever expirenced this issue:
Situation:
Datastore 1000GB (100GB free space) with VMs
- Random small VMs [550GB]
- SQL1 [150GB]
- SQL2 [200GB]

1. Veeam runs backup job in which both SQL1 and SQL2 are being backup up.
2. SQL1 backup takes a long time because of large delta.
3. Scheduled job on SQL1 generates alot of IO. Normally the veeam backup would`ve been ready for 2 hours, but now the server is still in snapshot mode so snapshot grows very fast.
4. SQL1 snapshot fills up 100GB free space and SQL1 ends up in paused IO mode because of failure of being able to extend snapshot.
5. Veeam backup of SQL1 completes and starts backup of the next server, SQL2.
6. Veeam tries to bring SQL2 in snapshot mode, SQL2 server powers off because it is unable to create snapshot file!

How is this possible, doesn`t Veeam check for free space on relevant datastores before trying to bring a VM in snapshot mode? I don`t have to explain i don`t like explaining to our CEO that the act of trying to backup a server brings it down. I couldn`t really deny his opinion that the backup software is at fault here.

In my opinion, the fact that SQL1 ended up in pause mode is not a problem with Veeam, it can (and probably will) happen once in a while. The power off of SQL2 however, shouldn`t have, veeam could have detected there was no free space left for a snapshot to be created.

What do you think?

[dellock6]

Can you first confirm from vCenter logs that the datastore was completely filled because of snapshots on those VMs?

Luca.

[AJ83]

It has been some time (it happened on 20th of june)
Only thing i could find was in de vmkwarning log:

Code: Select all

2013-06-20T06:15:44.436Z cpu19:12097737)WARNING: Swap: vm 12093644: 5501: Failed to extend swap file type=regular from 0 KB to 4194304 KB. status=No space left on device
2013-06-20T06:15:44.436Z cpu19:12097737)WARNING: MemSched: vm 12093644: 5877: Insufficient swap: need=4194304KB
2013-06-20T06:31:57.170Z cpu21:12094587)WARNING: Swap: 5495: Failed to extend swap file from 0 KB to 4194304 KB.
2013-06-20T06:31:57.170Z cpu21:12094587)WARNING: Swap: vm 12090492: 5501: Failed to extend swap file type=regular from 0 KB to 4194304 KB. status=No space left on device
2013-06-20T06:31:57.170Z cpu21:12094587)WARNING: MemSched: vm 12090492: 5877: Insufficient swap: need=4194304KB
2013-06-20T06:37:13.362Z cpu13:12090810)WARNING: Swap: 5495: Failed to extend swap file from 0 KB to 4194304 KB.
2013-06-20T06:37:13.362Z cpu13:12090810)WARNING: Swap: vm 12099003: 5501: Failed to extend swap file type=regular from 0 KB to 4194304 KB. status=No space left on device
2013-06-20T06:37:13.362Z cpu13:12090810)WARNING: MemSched: vm 12099003: 5877: Insufficient swap: need=4194304KB

I don`t think i can confirm through logs that it was the snapshot filling up the datastore, but i have confirmed it myself at the time. I had to sVmotion some VM`s of the datastore to free up some space, resume the SQL1 and commit the large snapshot.

[Gostev]

How is this possible, doesn`t Veeam check for free space on relevant datastores before trying to bring a VM in snapshot mode?

Yes, Veeam does perform such checks, and there are a couple of different safe nets around this situation:
1. Starting 10 GB free disk space on the production datastore (the value is configurable in the user interface), Veeam starts warning you about disk space issues on production datastore with the corresponding warnings in the job.
2. If there is less than 2 GB free disk space on the production datastore (the value is configurable through the registry), Veeam will refuse to create the VM snapshot, and will simply fail out of the corresponding VM's processing.

[AJ83]

Yes, Veeam does perform such checks, and there are a couple of different safe nets around this situation:
1. Starting 10 GB free disk space on the production datastore (the value is configurable in the user interface), Veeam starts warning you about disk space issues on production datastore with the corresponding warnings in the job.
2. If there is less than 2 GB free disk space on the production datastore (the value is configurable through the registry), Veeam will refuse to create the VM snapshot, and will simply fail out of the corresponding VM's processing.

Thanks Gostev!

Option 2 is what i wanted to know, apparently, the implementation isn`t 100% failsafe. 2GB can be gone in a couple of seconds if you have fast storage, i would like to change that to 25GB, how is this done?

We are using build 6.5.0.144
The option 'Warn me when free disk space is below:"10GB" ' is enabled. I`m changing it to 50GB

[Gostev]

No problems! I am out of office on vacation right now, but someone from my team will talk to R&D to find out the registry key name, and post it here.

[foggy]

2. If there is less than 2 GB free disk space on the production datastore (the value is configurable through the registry), Veeam will refuse to create the VM snapshot, and will simply fail out of the corresponding VM's processing.

The key is BlockSnapshotThreshold in the standard Veeam hive and the value is in GBs (DWORD).

[AJ83]

The key is BlockSnapshotThreshold in the standard Veeam hive and the value is in GBs (DWORD).

The standard veeam hive should be HKEY_LOCAL_MACHINE\SOFTWARE\VeeaM\Veeam Backup and Replication\ ?

The key you mentioned isn`t there, should it already exist? I can create the key, but how can i check if it`s picked up?

[Vitaliy S.]

The standard veeam hive should be HKEY_LOCAL_MACHINE\SOFTWARE\VeeaM\Veeam Backup and Replication\ ?

Correct.

The key you mentioned isn`t there, should it already exist? I can create the key, but how can i check if it`s picked up?

You should create this key manually and it will be automatically picked up by Veeam backup server.

[AJ83]

Correct. You should create this key manually and it will be automatically picked up by Veeam backup server.

Thanks, i added the key. I reckon veeam services need to be restarted. Can i check in logs if it is active or do i just have to test it to make sure it works?

[foggy]

I think it's easier to test - that way you will be at least sure it actually works rather than just know that the value is read from the registry.