I have made a 2008 r2 server into a domain controller and want to turn on application-aware processing,
I enter the username and password for the domain and click TEST but it does not say if it passed or failed just shows a yellow triangle.
It should work via VIX as we do not have IP connection to the VM.
Please confirm if this is working as expected with VIX
Yes, it is working as expected. If your VM is not accessible via network, there there will be a failover to network-less processing mode (VIX API). If you want to use application-aware image backups for this DC VM, you can leave it as it is and tick a checkbox in the job settings which will require AAIP process to complete with a successful state.
You can also reverse the sequence of connection attempts using the InverseVssProtocolOrder (DWORD) registry value to save some time during the backup. Thanks.