Inconsistent behavior of backup jobs and best practices?

[patrik.wyde]

Hi!

I am currently running Veeam Backup & Replication 7 to evaluate the purchase of Enterprise Plus license. Veeam B&R has been implemented in a small VMware lab environment for testing purposes.

So far I have been testing how the product perform backup jobs and I am seeing inconsistent behaviors that worry me. Prior to posting this thread I've Googled and also searched this forum to find answers. I am however not able to find any satisfying answers to the questions I have. Hopefully this thread will clear some of them and for that I thank in advanced.

VMware Setup

• 1 x ESXi 5.5 (physical host)
  1 x vCenter Server Appliance v5.5

Virtual Machines

• 1 x vCenter Server Appliance v5.5
  2 x Domain Controllers (Windows Server 2012 R2)
  3 x Member Servers (Windows Server 2012 R2)
  1 x Unix Server (OmniOS)

Storage (fileserver/SAN/NAS) Setup
One VM acting as fileserver/SAN/NAS running OmniOS (Solaris).

Veeam B&R Implementation
One Veeam B&R Server running on a domain joined Windows Server 2012 R2 installation as a VM. The Server is also Veeam B&R proxy.

Backup Repository has been setup on a SMB share on the Unix VM running OmniOS (Solaris).

A domain account has been created to perform Veeam B&R operations. Account is named vbackup_svc and is a member of the local administrators security group on the Veeam B&R server which is named vbackup.lab.local. All Veeam services on vbackup.lab.local are run with this account. The account is also member of the administrator group in vSpehere.

Domain Setup
One Active Directory domain for testing. All the above servers are domain joined, including vCenter Server and the Unix serer running OmniOS.

GPOs applied in the domain are security compliance baselines (http://social.technet.microsoft.com/wik ... notes.aspx) generated from Microsoft Security Compliance Manager (http://technet.microsoft.com/en-us/libr ... 77002.aspx).

Permissions are controlled through security groups as usual.

Tests and questions...
I created a backup job of one of the two virtual Domain Controllers (dc1.lab.local). The credentials used to perform the backup job is set to the vbackup_svc domain account. Please note that this account is only member of the local administrators group on the Veeam B&R server (vbackup.lab.local). The backup job finishes successfully. According to documentation the vbackup_svc need to be member of the local administrators group on the server that backups is being performed on. Even though vbackup_svc account is not member of administrators the job finishes successfully. I find this a bit strange as it contradicts documentation. Still I think the result of the test is positive from a security perspective as I do not want to grant vbackup_svc account too much privileges in the domain.

I then create a backup job of the second virtual Domain Controller (dc2.lab.local). The results are the same, meaning the backup job finishes with success.

Now I create a backup job that include both virtual Domain Controllers (dc1.lab.local and dc2.lab.local). This time the backup job finishes with warnings(?). All three jobs have enabled "application-aware image processing" and "guest file indexing". How is it possible that the first two jobs finish successfully and not the third? The warnings received in the report are:

Code: Select all

Name	Status	Start time	End time	Size	Read	Transferred	Duration	Details
dc1.lab.local	Warning	09:18:25	09:23:02	40,0 GB	12,3 GB	6,7 GB	0:04:36	Failed to index guest file system. VSSControl: Index failed
dc2.lab.local	Warning	09:18:27	09:23:09	40,0 GB	19,4 GB	6,7 GB	0:04:41	Unable to truncate transaction logs. Details: Cannot keep snapshot. No job found.

Now I proceed with trying to grant the vbackup_svc account local administrative privileges on the Domain Controllers. Since these are DCs, there are no local accounts as there is on member servers. The vbackup_svc account must therefore become a member of the Administrators security group in the domain (BUILTIN\Administrators). After joining the group, the backup job of both virtual Domain Controllers are finished successfully.

How is it possible that the two separate backup jobs of the two Domain Controllers in the domain can be backed up, but not the third job that include both DCs when the vbackup_svc account is only granted local administrator privileges on the Veeam B&R server?

This brings up my questions regarding best practices of permissions in a Windows Active Directory domain. As an administrator of a Windows Active Directory domain I want to minimize accounts that are members of administrator groups. Especially the "BUILTIN\Administrators" group. What is the best practice regarding this. is there a way fine grain the permissions. How is this solved in your environments?

Any comments and thoughts are greatly appreciated.

[Vitaliy S.]

Hello,

I created a backup job of one of the two virtual Domain Controllers (dc1.lab.local). The credentials used to perform the backup job is set to the vbackup_svc domain account. Please note that this account is only member of the local administrators group on the Veeam B&R server (vbackup.lab.local).

Account that is used to perform Guest OS VSS freeze and indexing does not need to be the member of local admin group on the backup server. Actually, this account might be just a local account on the processed VM.

According to documentation the vbackup_svc need to be member of the local administrators group on the server that backups is being performed on.

Yes, that's correct. Is it a domain admin account? The one you use?

Still I think the result of the test is positive from a security perspective as I do not want to grant vbackup_svc account too much privileges in the domain.

No need to do that, you can use local admin accounts to perform VSS freeze and indexing.

How is it possible that the two separate backup jobs of the two Domain Controllers in the domain can be backed up, but not the third job that include both DCs when the vbackup_svc account is only granted local administrator privileges on the Veeam B&R server?

I'm not sure that the problem here is in difference of the accounts used. Based on the error message, there might be some issues with the freeze operation. Full logs review will definitely shed some light on the job failure. You can ask our support team for assistance with this.

Let me know if that helps!

[patrik.wyde]

Hi!

Thank you for your reply.

Account that is used to perform Guest OS VSS freeze and indexing does not need to be the member of local admin group on the backup server. Actually, this account might be just a local account on the processed VM.

Yes I understand that.

Yes, that's correct. Is it a domain admin account? The one you use?

Yes it is (LAB\vbackup_svc).

No need to do that, you can use local admin accounts to perform VSS freeze and indexing.

In a Windows domain environment you really do not want to use local accounts. As it contradicts centralized management of user accounts. The problem I am describing is on Domain Controllers. They do not have any local accounts. meaning I cannot open "Computer Management > Users and Groups", because there is none. To grant local administrative privileges on a Domain Controller, the account needs to be a member of the "BUILTIN\Administrators" security group in the domain and can be found in "Active Directory Users and Computers" MMC snap-in. Making an account a member of this group might be against account policies in a organization. Hence my question and problem.

I'm not sure that the problem here is in difference of the accounts used. Based on the error message, there might be some issues with the freeze operation. Full logs review will definitely shed some light on the job failure. You can ask our support team for assistance with this.

As I am no Veeam B&R technician, the logs are very hard to read. I can gladly post the logs but since I do not own a Veeam B&R licence, I am not sure if the support team will even look at them. I am currently testing the product to see if I should invest in a license that includes support. Meaning I am running a trial version of Veeam B&R.

Speaking of logs. Which log should I be looking in to identify the problem? There are so many in C:\ProgramData\Veeam\Backup\...

[Vitaliy S.]

To grant local administrative privileges on a Domain Controller, the account needs to be a member of the "BUILTIN\Administrators" security group in the domain and can be found in "Active Directory Users and Computers" MMC snap-in. Making an account a member of this group might be against account policies in a organization. Hence my question and problem.

Yes, you do not have "local" accounts on the Domain Controller, but in order to perform all required steps for backing it up in application-consistent state you need to be using built-in domain admin. This is especially required for VMs like DCs, as during backup job we do not just use VSS to freeze the OS, we run extra steps that would prepare DC for the backup.

As I am no Veeam B&R technician, the logs are very hard to read. I can gladly post the logs but since I do not own a Veeam B&R licence, I am not sure if the support team will even look at them. I am currently testing the product to see if I should invest in a license that includes support. Meaning I am running a trial version of Veeam B&R.

Yes, I agree that logs might be hard to read, you can ask assistance from our support team with this task. We do provide support during evaluation period as well.

Speaking of logs. Which log should I be looking in to identify the problem? There are so many in C:\ProgramData\Veeam\Backup\...

I would recommend checking the job log itself (it should be named after your job). Take a look a the latest session where you had this failure and compare it to the successful job run of the standalone DC.

[patrik.wyde]

Yes, you do not have "local" accounts on the Domain Controller, but in order to perform all required steps for backing it up in application-consistent state you need to be using built-in domain admin. This is especially required for VMs like DCs, as during backup job we do not just use VSS to freeze the OS, we run extra steps that would prepare DC for the backup.

Ok, I understand. this is what confuses me. If I create a backup job of dc1 (only dc1, no other DC and/or Windows member server) and use the domain account LAB\vbackup_svc in the backup job as credentials. The job finishes with success. Please note that the domain account LAB\vbackup_svc is not a member of either "Domain Admins" or "BUILTIN\Administrators" security group.

If I create a backup job that include both DCs, meaning dc1 and dc2, the job finishes with the warnings in my first post.

The two backup jobs have identical configuration. The only difference is that the job that finishes with success contains only one (1) Domain Controller and the job that finishes with warnings contain both Domain Controllers in the Windows domain. I just don't understand this. Both jobs (regardless of how many DCs the jobs contain) should finish with success and no warnings.

Yes, I agree that logs might be hard to read, you can ask assistance from our support team with this task. We do provide support during evaluation period as well.

Ok, I did not know that. I'll have to investigate how I register a ticket with the support team.

I would recommend checking the job log itself (it should be named after your job). Take a look a the latest session where you had this failure and compare it to the successful job run of the standalone DC.

Ok, so it should be located in the following directory "C:\ProgramData\Veeam\Backup\Domain_Controllers\" and reside in the file named "Job.Domain_Controllers.Backup.log"?

[Vitaliy S.]

1. Is it just a regular user? I'm also confused by this fact, so comparing two job logs should definitely shed some light on this.
2. You can give them a reference to this post in order to save time by not explaining the same thing twice. Let me know your Case ID, so I could also track it.
3. Correct.