OK. So here are my observations so far.
First of all it's necessary to explain our setup a bit. Actually we have 3 mailbox servers running. Server 1 and Server 2 are holding the active databases. Server 3 is a complete passive node meaning that he holds only passive database copies of Server1 and Server2. And this is the one we are backing up every day (as far as i remember it's best practice by Microsoft to backup only the passive database copies).
In the case where you perform the backup of passive databases the MS Exchange Replica
VSS Writer is involved in all the backup tasks performed on this node. So in case of a successfull backup run MS Exchange Replica
VSS Writer notifies Server 1 and Server 2 with the active databases about the successfull backup and issues the log truncation command to them. Log truncation happens on the active node then and these changes are replicated back to the passive copies. Log truncation doensn't happen on the passive node itself!
Now after i've restarted COM+ Event System service every first backup run by Veeam was successfull. Always! But we ran into another issue with this. Log truncation didn't happen and i was also missing the corresponding event on the passive node that should occure when the disks were backed up and Veeam says "Truncating Transaction Logs". Event that should appear at that moment in your Application Log should be from MSExchangeRepl with ID 2046:
Code: Select all
The Microsoft Exchange Replication service VSS Writer instance 17aaf2a1-d81c-4e1f-ab2e-cccdf26066db has successfully completed the backup of database 'xxx'.
Database log truncation has been requested for this database. Log truncation will occur on the active copy after the next log generation is created. Log truncation will occur automatically on the passive copies after that log file is copied.
But after restarting COM+ Event System Service and Veeam backup was successfull i couldnt see any of these events. And also on the active nodes no events occured for the successfull backup run. Taking a look into the properties of the databases just showed "Last full backup" with the wrong time stamp. So it seemed that Exchange wasn't aware that a successfull backup was made and log should have been truncated.
After further investigation i've found out why. After restarting Server 3 i've opened a command prompt and issued the command "vssadmin list writers". A bunch of
VSS writers showed up. It looks like this:
Code: Select all
vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
(C) Copyright 2001-2005 Microsoft Corp.
Writer name: 'Microsoft Exchange Replica Writer'
Writer Id: {76fe1ac4-15f7-4bcd-987e-8e1acb462fb7}
Writer Instance Id: {ce816b93-4819-4491-b01f-219143971883}
State: [1] Stable
Last error: No error
Writer name: 'Microsoft Exchange Writer'
Writer Id: {76fe1ac4-15f7-4bcd-987e-8e1acb462fb7}
Writer Instance Id: {5978e961-78bf-4cb8-8022-91f8d553027b}
State: [1] Stable
Last error: No error
...and many more VSS writers installed on the system
If i now restart the COM+ Event System service and issue the above command again the vssadmin list writers shows......nothing! The list was empty in my case. To get the writers back i've restarted Volume Shadow Copy service. But not every writer was back again. I was still missing the Exchange writers (especially the Replica
VSS writer). So i've restarted the Information Store too (as it is a passive node i could do this without any affect on my production environment. Never do this on an active node while working hours
) and the Exchange writers showed up again when doing a vssadmin list writers. And guess what. First backup run worked like a charm and log truncation was performed without any issue.
Looks like we have found a workaround for our environment and maybe it's helpfull for someone else of you guys out there.
Will still watch this behaviour like every day. If i find out anything else on this case i will keep you guys in this forum informed. So stay tuned.
Ah and before i forget it a big thanks to habibalby and Amerloc for pointing me into the right direction