Backing up Exchange 2010 - How to truncate logs?

[Dave-Departed]

Hi All,

I have a backup job that is backing up a VM hosting Exchange 2010, which is a member of our DAG. Currently, the live mailbox database copies are on a physical server, which is backed up to tape still, but soon, our Exchange environment will be fully virtualised.

As a test, I paused the Exchange tape backup for 1 night, and just let Veeam backup the VM DAG member hosting the passive mailbox database copies. While the VM was backed up successfully, the Exchange log files were not truncated. Can anyone tell me if there's anything special I should be doing on the Veeam job properties to enable this, please?

Thanks in advance.

Dave

[Vitaliy S.]

Hi Dave,

Please check if you have "truncate logs" option selected in the advanced settings of the application-aware image processing step of the backup job wizard. See our User Guide (page 125) for further details.

Thanks!

[Dave-Departed]

Hi Vitaly,

Yes, the option is set to "Truncate logs on successful backup only". And last night's backup job completed successfully, yet no log truncation?

Thanks

Dave

[Vitaliy S.]

Could this be the case?

[Dave-Departed]

I suppose it could be, but why wouldn't my Exchange logs be replayed on both DAG nodes? All db copies are healthy, and all have a Replay Queue Length of 0.

Should I raise a support case?

[Dave-Departed]

Not sure how relevant it is, but I just ran vssadmin list writers on the VM DAG node, and it shows the 'Microsoft Exchange Replica Writer' as State: [1] Stable, but Last Error: Retryable error.

[Dave-Departed]

And in the event logs, I also see:

ID: 2026
Level: Error
Source: MSExchangeRepl
Machine: server.company.com
Message: The Microsoft Exchange Replication service VSS Writer (Instance 3c14e9cd-532a-4064-b14c-0f2890625d75) failed with error C7FF07D7 when preparing for snapshot.

The plot thickens...

[Vitaliy S.]

Yes, looks like you've found it, you need to make sure all Exchange writers have no errors in their latest error state and there should be no errors in the Event Logs either.

[Dave-Departed]

Indeed, however... Reading further into the logs, I see event ID 2046 from Exchange VSS Writer for each of our mailbox databases, which states:

The Microsoft Exchange Replication service VSS Writer instance 3c14e9cd-532a-4064-b14c-0f2890625d75 has successfully completed the backup of database 'database name'.

Database log truncation has been requested for this database. Log truncation will occur on the active copy after the next log generation is created. Log truncation will occur automatically on the passive copies after that log file is copied.

Yet, obviously, that is not happening.

So maybe I just need to reboot the passive DAG member or restart the Exchange services to get rid of the 'retryable error'?

[Vitaliy S.]

Ok, now since we see that the log truncation call has been triggered, you need to wait till the next log generation is created on the active node. I have performed a quick search and found a similar discussion on the TechNet forums, please take a look: the logs are not truncated

[Dave-Departed]

Thanks, Vitaly, I checked the link.

Our log files have grown an extra 1GB since I checked this morning, so new logs must have been generated on the active node, yet it would appear that the logs have not been truncated. We have logfiles going back to 3 days ago in there.

So you think that, even though the VSS error was present, the fact that the 2046 events for all 4 databases precede it means that they were backed up successfully and the log files *should* truncate?

Thanks

Dave

[Vitaliy S.]

Correct, Veeam backup job has sent the API call to VSS writer to truncate the log files, now we need to investigate why this call didn't go further.

[Cokovic]

Just a simple question regarding your events you can see. Where do you look at? This are the events from the passive node right?

As the log truncation still happens on the active node please take a look into the events on your active node if you can find any error. Additionally please start you Exchange Management Console, open Organization Configuration --> Mailbox and then right click one of your databases and select properties. Whats the "Last full backup" time? Does it correspond with your last Veeam backup?

I had it once that log truncation didn't happened for about 4 days (over the weekend). Backup in Veeam was always successfull but we had a failed and suspended database which seems to have prevented log truncation. In the "Last full backup" time in the database properties i could see that the timestamp didn't get updated even Veeam was claiming everything went fine with the backup.

I can't tell you what exactly fixed this log truncation thing. As we had made several changes to our VM config (removed some unused drives to improve snapshot creation and commit times) and fixed the failed and suspended databases i've also performed a new active full. Maybe it's worth to give a new active full a try.

[Dave-Departed]

Yeah, these were events from the passive node. I will check the active node shortly!

As for the Last Successful Backup timestamp, it does NOT match the Veeam backup time, infact it's a day behind (timestamp says LSB was 01 Nov @ 1:56am, last Veeam backup ran between 00:00-04:00am 02 Nov)!

[Dave-Departed]

The only event of any relevance that I can see in the App log of the active server is this:

Event ID: 1007
Source: MSExchange Mailbox Replication

Details:

Code: Select all

The Mailbox Replication service was unable to determine the set of active mailbox databases on a mailbox server.
Mailbox server: Server.domain.com
Error: MapiExceptionNetworkError: Unable to make admin interface connection to server. (hr=0x80040115, ec=-2147221227)
Diagnostic context:
    ......
    Lid: 15000   dwParam: 0x6BA      Msg: EEInfo: prm[1]: Pointer val: 0x0000000000000000
    Lid: 15000   dwParam: 0x6BA      Msg: EEInfo: prm[2]: Pointer val: 0x1E66A8C000000000
    Lid: 16280   dwParam: 0x6BA      Msg: EEInfo: ComputerName: n/a
    Lid: 8600    dwParam: 0x6BA      Msg: EEInfo: ProcessID: 8580
    Lid: 12696   dwParam: 0x6BA      Msg: EEInfo: Generation Time: 2012-11-02 02:48:36:627
    Lid: 10648   dwParam: 0x6BA      Msg: EEInfo: Generating component: 18
    Lid: 14744   dwParam: 0x6BA      Msg: EEInfo: Status: 10060
    Lid: 9624    dwParam: 0x6BA      Msg: EEInfo: Detection location: 318
    Lid: 13720   dwParam: 0x6BA      Msg: EEInfo: Flags: 0
    Lid: 11672   dwParam: 0x6BA      Msg: EEInfo: NumberOfParameters: 0
    Lid: 24060   StoreEc: 0x80040115
    Lid: 23746  
    Lid: 31938   StoreEc: 0x80040115
    Lid: 19650  
    Lid: 27842   StoreEc: 0x80040115
    Lid: 20866  
    Lid: 29058   StoreEc: 0x80040115

[Cokovic]

Do you see informational events from ESE BACKUP ID 913 and from ESE ID 213 on your active node?

Also take a look into your Veeam log and see when log truncation happened. At this time there should be also at least informational events in your app log on your active node (or maybe warnings. For example from ESE, ESE BACKUP or MSExchangeRepl).

[Cokovic]

And in the event logs, I also see:

ID: 2026
Level: Error
Source: MSExchangeRepl
Machine: server.company.com
Message: The Microsoft Exchange Replication service VSS Writer (Instance 3c14e9cd-532a-4064-b14c-0f2890625d75) failed with error C7FF07D7 when preparing for snapshot.

The plot thickens...

Regarding this error if found this one here on the web. Possibly a network or communication issue? Could you check this too?

MS Exchange Replication Service needs to be able to communicate with the Information Store of the active node over RPC as log truncation will happen there and not on your passive node.

Maybe you could also give a new active full a try? As i mentioned before i had this once in our environment and after fixing the failed and suspended database copy next incremental run didn't changed anything. Backup successfull but no log truncation. So i've started a new active full in Veeam and then everything went fine. Dont know if it helps in your case.

Btw whats the patchlevel of your Exchange DAG ebvironment?

[Dave-Departed]

Where do I find the Veeam log, please? In the backup job html report, I see no details other than basic ones.

As for the Event IDs you listed below, the latest I see them is the PREVIOUS night, which is when it was still being backed up by tape! Nothing is listed for ESE or ESE BACKUP during last night's backup window on the active node, except for a couple relating to online defrag.

So what's happening here? Veeam says the databases were backed up correctly, and event logs on the passive node say "truncate the logs please", yet the logs remain (and are still growing), and there is no notification on the active node that any databases were backed up!!

I'm confused!

[Dave-Departed]

Regarding this error if found this one here on the web. Possibly a network or communication issue? Could you check this too?

MS Exchange Replication Service needs to be able to communicate with the Information Store of the active node over RPC as log truncation will happen there and not on your passive node.

Maybe you could also give a new active full a try? As i mentioned before i had this once in our environment and after fixing the failed and suspended database copy next incremental run didn't changed anything. Backup successfull but no log truncation. So i've started a new active full in Veeam and then everything went fine. Dont know if it helps in your case.

Btw whats the patchlevel of your Exchange DAG ebvironment?

Forgive my ignorance, I'm fairly new to Veeam... What's an active full, exactly? Our jobs are set to reverse incremental, so obviously the most current backup is essentially a full.

As for the patch level, the entire Exchange environment is 2010 SP2.

[Dave-Departed]

Ahh, I already came across that link myself too, checked it out... I do see quite a few errors where it complains that the DC can't be contacted temporarily, and then it comes back (FYI, our stupid setup, which was in place before I joined the company, was that we originally had 2 x Exchange 2010 boxes, both multi-role with MBX/CAS/HUB installed, and set in the unsupported configuration to piggyback the CAS array onto the DAG cluster... And to top it all off, one of the Exchange boxes was also made into a DC!!).

I'm in the process of adding a new DC, and getting rid of the DC role from the aforementioned Exchange box (which, incidentally, is the one that the other DAG member complains that it can't contact intermittently), so that can only help matters.

[veremin]

Where do I find the Veeam log, please? In the backup job html report, I see no details other than basic ones.

It all depends on which type of OS you’re using for your Veeam B&R server. So, a little help for those who might have problems finding an information:

• Windows Server 2003: %allusersprofile%\Application Data\Veeam\Backup
• Windows 7, Windows Server 2008 and R2: %allusersprofile%\Veeam\Backup
• Linux servers: /var/log/VeeamBackup/

Furthermore, there is a way to collect each node’s log files from one place. Under the menu Help, you can use “Support information” option, to collect the individual log files all around the backup environment.

Thanks.

[Cokovic]

Aaah ok. Is your DC he can't contact your DNS server too?

Which Updaterollup do you use? Is it Update Rollup 4 v2 (latest one)?

Btw. there are some recommended additional hotfixes available for Microsoft DAG clusters. Here is a link for you:

http://blogs.technet.com/b/rmilne/archi ... pdate.aspx

And yeah or you take a look into Veeam logs to find out when log truncation should have happend or you look into the statistics window of your job.

An new Active Full within Veeam can be run if you right click your job and select "Active Full".

[Cokovic]

...

Furthermore, there is a way to collect each node’s log files from one place. Under the menu Help, you can use “Support information” option, to collect the individual log files all around the backup environment.

Thanks.

Thats true. But in case you have a lot of jobs running and depending on your job history this can throw out a really big file.

[Dave-Departed]

The DC IS a DNS server too, yes, but we have another DC (which also holds all the FSMO roles, as well as DNS too), which Exchange seems to ignore (well, except for when there is an event every so often in the logs, where Exchange 'finds' both DCs again).

Not UR4 yet, no. I will get that applied!

We are trying to move away from the stupid setup, into a fully-supported setup (2 x MBX servers in a DAG, with 2 x CAS/HT servers using NLB).

[Dave-Departed]

An new Active Full within Veeam can be run if you right click your job and select "Active Full".

What will this do, exactly?

[veremin]

Thats true. But in case you have a lot of jobs running and depending on your job history this can throw out a really big file.

Yep, there's no denying it.

Still, the only purpose was to provide complete information.

[Cokovic]

What will this do, exactly?

Just creating a new full backup file. Dont really know if it's related to the log truncation error. But after i've created a new active full my logs were truncated again. Could be also due to the fact we have fixed the error with the failed and suspended database copies. It's just a suggestion and if you have the space left on your backup repository maybe worth to give it a try.

[Dave-Departed]

Ahh ok, I see.

But won't this just happen tonight anyway, when the backup runs as scheduled?

[Cokovic]

Running in reverse incremental mode changes will be injected directly into existing full backup so that you've got a "new" full backup file every day. So some kind of a daily "synthetic full" every day (excuse my terminology and correct me if i'm wrong )

May i ask how many mailboxes you've got in your environment?

[Dave-Departed]

Only around 60 mailboxes.

So is new Active Full possible when running in Reverse Incremental mode?