is it really important to use AAIP?

[Luca82]

I was wondering how important the application consistency and the use of AAIP really are. Veeam promotes this feature a lot but I would like to know if any of you asked the same question: if I do a crash consistent backup (or replication), what is the real risk? Doesn't the database restart during the restore? Out of 100 backups / replicas what is the percentage chance that the database is actually corrupt? If I think of all hardware replication systems (storage replicas orchestrated by SRM) they are all crash consistent and not application consistent! And in case it should be corrupted, I suppose that with its transaction logs it can be rebuilt and restarted without too much difficulty ... so forgive the somewhat tendentious question, but how important is it really to integrate with AAIP and VSS?

[Andreas Neufert]

Hello Luka, very good question.

There are multiple areas that we should talk about.

Consistency:
Nearly all Databases (and their application) can survive a "power loss" event of a server, which would be the same as a failover or restore of a crash consistent backup/replication. With any of these events (power loss) you could run into some trouble with the software applications that lead to a needed repair process. Even the filesystem sometimes need a repair (checkdisk on windows for example). Now the question is if you detect it right away at restore and as well how long it would take for this repair. A autorecover of a big database can take hours. As well do you have the needed skills and know how the databases work and do you have the knowledge on how to repair those. So my suggestion is to use guest processing whenever you can to avoid this. When there is a real disaster and you really run into some issues then you maybe wished to would have spend the time upfront for using the guest processing.

Application restore to a former point:
Many applications replicate themself accross system or depend on different servers data state being "in sync". Veeam sets before backup specific flags for the applications that allows them after restore to detect that they are restored and the applications automatically take actions to update their communication/data.
A good example is, that when you restore an active directory server that was backed up with AAIP, we will set the active directory server automatically in non-authorative restore mode to not harm the active directoy. This is the way Microsoft recommends it. Modern versions of Active Directoy have a self detection for these scenarios in there but older ones depend on these actions. Without AAIP you would need to know this and bring the active directory system at boot after restore manually in this stage. Other example is that we bring the Exchange Databases autoamtically in recovery mode after restore and you do not have to do this manually. In the end specifically if you want to avoid these manual tasks or do not know the details of the applications, enabling AAIP will help you a long way.

LogBackups/Log Truncation:
Databases like Microsoft SQL and Exchange have transaction logs that need to be backed up and commited (deleted) to avoid that the disks run full. With guest processing we do this for many applications as well. You would have to use another database backup method to perform this task if you run crash consistent backups only.

Please allow me to say that all crash consistent backup methods are the same in case of application stability. Doing crash consisten backups on VMware has the advantage that all disk IO end at the same time which is an advantage over storage snapshots which have much more problems because of async states between disks.

So overall Veeam Crash Consisten backups can be used the same way as with the other tools. AAIP just offers you so many advantages and I would always use them whenever possible. Little time investment for so much benefits.

[Luca82]

thank you very much for the very accurate answer. In principle I could summarize by saying: if there is the possibility of using AAIP, that is we work with Windows machines, then why not? It has no disadvantages .... it only has small or big advantages. Things could change for linux because we don't have the ability to leverage VSS so you have to "stop" the database activities before making backups or snapshots ... and this could create inconvenience or slowdowns in production.

[Andreas Neufert]

Correct on the Windows side.

For Linux the AAIP should only be used if you have Oracle Databases in Archive Mode (we provide consistency) or if you want to run Pre- and Post- backup scripts.

[Luca82]

I'm sorry I take this opportunity to clarify another doubt. It is correct to say that the backup with AAIP is not consistent in absolute terms, but with AAIP during the restore Veeam will make the Database consistent. So the consistency is guaranteed thanks to the restore process: I know it's a small thing but I need it to understand if I have well understood the functioning of the AAIP

[Andreas Neufert]

No, on Windows before backup everything is brough into consistency mode with AAIP that support VSS and as well Oracle is brought even in consistency mode when VSS was not configured for it. On Linux we bring Oracle in consistency mode. Veeam Agent for Linux support some additional consistency for Linux Databases.

The recovery settings are just an additional feature, so that Admins have to worry less about the special treatment of applications at restore.

[JaySt]

Little time investment for so much benefits.

this little bit really depends on the environment. In bigger enterprise environments, this can go the other way in regards of network security policies, account permissions etc....

[Origin 2000]

Only if AAIP is enabled FOR BACKUP veeam will restore a Windows DC automatically within the non-authorizied mode which can be very important.

Regards,
Joerg

[fsr]

One option in Linux, if you cannot interrupt services to take a snapshot, is to make a local backup with the native tools, which most likely allow to take a backup without any downtime. Then, if you ever need to restore the VM, you have the option to restore the database from the local backup.

[AlexHeylin]

In some cases it's ABSOLUTELY ESSENTIAL to use either AAIP, or at least native invocation of Windows VSS.

By design Windows Domain Controllers will BSOD (kernel panic and halt) if they detect that they were restored from a backup which was taken online and not taken in the correct application aware way. You will only find this out when you try to boot up the restored DC. Until then everything looks fine and all backups and backup checks etc will pass.

[Andreas Neufert]

I think this is not the case really. Back in the old days, restoring a DC without restore mode led to the point that Active Directory was not syncing any entry correctly and you had a split brain situation.
Microsoft addressed this I think in Windows 2012R2 and added a "snapshot restore" protection to active directory. But anyway it can cause some specific issues later if the non-authorative restore mode was not used.

Never heard about the BSOD related to this. Can you please give any reference to it?

[Andreas Neufert]

Googled and I found it in our own forums ;.)
veeam-agent-for-windows-f33/bmr-of-serv ... 63157.html

https://social.technet.microsoft.com/Fo ... essentials

Done so many AD restores for Demo sover the years and never ran into this issue with the BSOD.

[mkaec]

I try to use AAIP as little as possible because I don't like giving Veeam B&R administrative access to the VMs. I think we are long past the time when it's acceptable to make the backup application a Domain Admin.

[Herbert Sz]

I think we are long past the time when it's acceptable to make the backup application a Domain Admin.

Soon you'll be able to use gmsa's instead.

[Andreas Neufert]

No Domain Admin membership needed. Local Admin is enough. As Herbert said GMSAs will be available soon.

The reason for local admin rights is our agent less approach where we install temporary a VSS requestor and this is the only way then. As well for restore of files and specific applications you need admin rights as well to perform all needed operations for it.

[johannesk]

Correct on the Windows side.

For Linux the AAIP should only be used if you have Oracle Databases in Archive Mode (we provide consistency) or if you want to run Pre- and Post- backup scripts.

Hi Andreas,

Just to make this crystal clear - is it unnecessary or worse to use AAIP for linux servers that are not database servers?

And am I right that non AAIP backup is the same as a snapshot done with vcenter?

Regards,
Jóhannes

[Andreas Neufert]

There is no side effect in using AAIP with Linux Systems that do not have any Oracle Database.
We will just connect, run a detection logic if there is an Oracle database and if there is non, just continue with regular processing. You could say that you are loosing some seconds in processing speed.
On the other side, having the authentication in place will help you to streamling the file level recovery.

Beside the oracle processing, you can use pre and post script processing with Veeam AAIP.

Guest processing does not change anything related to the VMware Snapshot processing and VM snapshots are used as usual. Veeam AAIP does NOT use the vSphere Guest Quiescense.