Now this is a story all about how my life got flipped-turned upside down. I'd like to take a minute, just sit right there, I'll tell you how Veeam became our invaluable availability platform.
A few years ago I was a week into leave after my first child was born. My colleagues in the office were in the process of a scheduled migration of our AD domain controllers, migrating all DCs from server 2008R2 to server 2012R2 and later upgrade the functional level to 2012R2. Pre migration tasks on each DC were AD backup, Veeam backup and VMware snapshot. Somewhere during the migration process replication across the domain became dysfunctional when a DC became only partially removed from the domain and some DCs were recognizing incorrect DCs with FISMO role(s) they didn't hold. Long story short AD was completely FUBAR, perhaps aided by user error. Our AD backups taken just prior to the migration didn't work when they tried to restore. I received a phone call outlining the above points and we figured we’d try using the VMware snapshots. Reverting from snapshots wouldn't fix the issues because of the tombstone lifetime age, each DC was in a state where the other DCs tombstone lifetimes had expired and could not be synchronized to. This is something we had not taken into account or event thought of beforehand would be an issue. AD was new to us and while not yet deployed to the desktop it was used widely across the organization and as a backend authentication source, the impact would be felt should AD not be back online the following business day. AD repair/recovery couldn't fix the issues. We quickly came to the determination that a complete Veeam restore of AD domain controllers was necessary.
I remoted into my workstation and began thinking out the restore process. While coming up with a restore plan I came to the realization that all of the backup infrastructure was AD authenticated, which obviously presented a rather large problem, no access to backup data. This prompted a complete reconfiguration of the backup infrastructure to use "workgroup" configurations on repos and other backup assets (something to keep in mind in case ship happens, no fault of Veeam). We had actually moved away from this model several months before in favor of centralized management using AD credentials, never thinking AD would be the single point of failure. Once reconfigured I was able to restore the domain controllers one by one. The secret to this success was Veeam’s ability to run through the Active Directory repair/recovery tools automatically thanks to its application-aware processing feature. One by one they came online and all was well, like nothing ever happened, successfully avoiding the weeks long process of rebuilding our AD infrastructure manually. I was then able to go back to enjoying life with my new little one
. Thanks, Veeam!
I can't imagine using our old backup product in this situation with any modicum of success, file restores would frequently be unreliable, much less an entire server or VMs. I have to say that the Veeam Availability platform is the most reliable and consistently performing backup, replication and monitor system I've ever used. My inbox is always green instead of red and I can find and fix issues before they become problems.
TLDR; AD domain became FUBAR and Veeam saved the day with a full domain restore. Go Veeam
or go home.