Multiple VSS Consistency?

[Poweruser]

Hi!
If i backup n hyper-v VMs which include: AD PDC, exchange, SQL server, etc.
How will veeam create the VSS/Snapshot?

are all VMs created in the same "second" of job execution or step by step?
exchange vm needs to be the same as AD PDC was in the same moment. is that always true?

and what about multiple VSS inside one machine? is this consistent too?
time(filesystem VSS) == time(sql VSS)?

[Mike Resseler]

Hi PowerUser.

Just to be clear, you have different VMs running, one AD server, one Exchange, one SQL server and they are all in the same job. Correct?

The way it exactly works is described here: https://helpcenter.veeam.com/docs/backu ... tml?ver=95

Multiple VSS inside one machine is always at the same time. That is not something that Veeam does, this is specific for the VSS subsystem by Windows. The moment we tell the VM to use VSS (we are the requestor) the freeze happens with all the VSS providers doing their job.

For different machines, the time in between can be different, but it should not be a problem at all. You can recover an AD and an Exchange server together even if there is a slight different in time of the VSS freeze. Is there a specific reason why they should be exactly the same?

[Mike Resseler]

Hi PowerUser.

Just to be clear, you have different VMs running, one AD server, one Exchange, one SQL server and they are all in the same job. Correct?

The way it exactly works is described here: https://helpcenter.veeam.com/docs/backu ... tml?ver=95

Multiple VSS inside one machine is always at the same time. That is not something that Veeam does, this is specific for the VSS subsystem by Windows. The moment we tell the VM to use VSS (we are the requestor) the freeze happens with all the VSS providers doing their job.

For different machines, the time in between can be different, but it should not be a problem at all. You can recover an AD and an Exchange server together even if there is a slight different in time of the VSS freeze. Is there a specific reason why they should be exactly the same?

[Poweruser]

a) Okay, so its not consistent over multiple VMs. For Exchange/AD i think it can cause troubles in some cases, but for a slight difference it may work most times.

b) What about 2 VSS executions on the same VM? will it be sync oder slightlty async?
Reason is: Software useses Filesystem + SQL and these two need to be always consistent, means changes have to be tracked at the same moment. does VSS obey this? i fear not

c) The same as b), but inside one VSS execution on MSSQL. Will it freeze alle Databases at the same moment, then backup, then unfreeze all? If i check the logs, there are some seconds between each db freeze inside one instance. is it consistent or just per database only?

[Mike Resseler]

a) Actually, it should never be an issue at all. What might be an issue is that when you restore AD (let's say with a VSS snapshot of 22.00 hour) and an exchange (let's say with a VSS snapshot of 23.00 hour). Imagine that between 22.00 and 23.00 you created a new user in AD and gave him a mailbox. When you restore AD, obviously that user is not known, but it is known in Exchange. But even then, that mailbox will become 'orphaned' and when you recreate the user in AD, it will get itself fixed in Exchange.

b) No. Inside the VM the Microsoft subsystem takes care of the "job" and uses an internal mechanism so that every freeze is at the exact same moment

c) Yes, it will be frozen all at the same moment. I know that you can see differences in the log, but VSS does replay to make sure that the actual data presented in the VSS snapshot is at the exact same time. To explain it (hopefully, you can tell me if I failed ) simply: The VSS requestor (Veeam in this case) requests to the VSS system of that VM to create a snapshot. The VSS system tells all the providers on that VM (SQL/ AD/ File/ Exchange/ ...) to prepare for backup. You will indeed see that they all prepare and that (logical actually) one provider might be done faster than the other. At that point in time, the VSS system uses its magic to get everything "in sync". Then it tells the requester (Veeam) that everything is ready and we grab that VSS snapshot. When we tell the VSS system that we are done, the system tells every provider to unfreeze (which again, will not be done at the exact same moment since some will be faster than others) Makes sense?

[Poweruser]

Okay Mike, thanks for your detailed explaination.
As a big software vendor says: backup software needs to be able to do a "prepareToFreeze" command.
then this is what you described in c.

so if i use windows ntfs, shares, sql, ad, exchange, etc it should be all finally in sync per VM basis but not over all vms.
could it be possible that you add a sync over all vms? keep it frozen until everything is ready?
would be nice!

[Mike Resseler]

The problem with that is that I don't see it necessary (Don't forget that keeping that freeze state is very resource intensive so you want to keep it as limited as possible...)

[Poweruser]

Resources and time is no matter for my backup jobs.
i got plenty of both :-)
iam thinking about adding a second job in the night for an offline backup.
shutting all vms down in the right order, do the backup, and restart them in the right order again.
i think that would be the safest and best way to make a backup.
additionally the online backup should run and clean up damn exchange logs. ;-)

anyway i think it is nice if the state is equal over all vms.
its the safest way because everything belongs together as one "instance" in my installation.

[Mike Resseler]

Obviously that is a safe way but as said, it is not really necessary. But being for 20 years in the backup world, I understand perfectly your concerns

PS: Make sure you test surebackup to see if different VM's can start in a quarantined environment and if they work well. You will love this functionality, especially since you have enough resources

[Poweruser]

Yes i think i will love this functionality, too. But i dont know how to start and will have to check it soon, when i find time. Seems to be more than a 5 minute job to set it up the first time.
The big software vendor which uses sql+filedatabases says you SHOULD have offline backup in every case next to online backups because it is safe in this case only.
They fear about differences between smb share and sql state because they both are linked together and the snapshot needs to be at identical time for filesystem+sql.

[Mike Resseler]

I am not going to talk about that software vendor or try to understand why they state that. What I assume is that the box is using SQL server and the data of the databases are stored on that SMB share. So I still stand with the fact that it doesn't need to be the identical time but if you make huge changes to the config on the server in between the backup of the data and server, you might run into issues

[Poweruser]

they do it a bit tricky.
first all mssql databases are saved onto the shared. i dont know why.
second they have an own database which is something like sqlite or more uglier. its sits there and contains metadata, usersettings in ini files and thousands other legacy things like logs, configs, etc they have developed in the last centuries from dos to windows.
means: the main stuff is sql but all the older stuff and the management stuff goes onto smb.

edit: nevertheless i tried to to an offline backup as suggested with prefreeze jobs.
but it did not work, because veeam has an fixed script timeout.
their offline backup prejob script is taking much time.

1. database check
2. online backup to folder
3. shutdown mssql db
4. do a loop ping for 15 minutes to wait that sql is safe down.
lets guess after 20-30 minutes or later veeam will get its script done.
but veeam does not want to wait as long as desired :-(

[Mike Resseler]

I probably don't want to know what the name of that solution is... How long before a database can shutdown?

I believe you can change the timeout of the pre-script by looking for this key in the registry:

HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication" PreJobScriptTimeoutSec. If I am not mistaken the default is 900 seconds. So you can change that to a bigger number.

[Poweruser]

as in other threads i changed these values and the more specific vss timeout.
but veam timeouts after 10 minutes.

[Mike Resseler]

Did you restart the service?

[Poweruser]

yes, the main backup service.
but the kb article says default is another value i think 15m instead of 10.
maybe the article is outdated for vss timeout.
btw there is another similiar topic in this board about this.

[Mike Resseler]

In that case I would advice you to open a support call. To the best of my knowledge, this setting should work and should timeout only after the time in the setting

[Poweruser]

about the timeout part: it was resolved in thread: (i hope)
https://forums.veeam.com/veeam-backup-r ... ml#p290113